Program, device and method for creating survey questions

Abstract

To provide a program etc. for acquiring evaluable expectations and satisfaction levels by asking a variety of questions to users who are the subject of a survey. [Solution] In a program that causes a computer to function to collect questionnaires from users based on evaluation items, the program functions as a question creation means that sends prompts to a generation AI server for each evaluation item, instructing the creation of questions about the future and questions about the present, and receives the questions about the future and questions about the present from the generation AI server, and an answer collection means that sends the questions about the future and questions about the present to each user terminal, respectively, receives the answers to each question from the terminal, and stores the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels for each user, in an answer database.

Description

[Technical field]

[0001] The present invention relates to a questionnaire technique for asking questions to a plurality of users. [Background technology]

[0002] Conventionally, there is a technology for an organizational development support system that supports the strengthening and improvement of an organization (for example, see Patent Document 1). According to this technology, scores for expectation and satisfaction are obtained from the answers of a plurality of members to a plurality of survey questions, which are questions for which expectation scores and satisfaction scores are defined, and a score indicating the overall motivation of the members is calculated using the scores for the plurality of expectation and satisfaction as parameters. [Prior art documents] [Non-patent literature]

[0003] [Patent Document 1] Patent No. 6298855 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the expectation and satisfaction level are asked to members using predefined questions as in Patent Document 1, it is not possible to ask a variety of questions. In other words, there is a problem in that the expectation and satisfaction level must be obtained from fixed questions. In response to this, the inventors of the present application first considered whether it might be possible to obtain evaluable expectations and satisfaction levels by asking a variety of questions to users who are the subjects of a survey. Secondly, we wondered whether it would be possible to obtain an appropriate evaluation index from users' response values ​​for each evaluation item. Thirdly, we wondered whether it would be possible to obtain an emotional evaluation index from the text of qualitative responses from users. Fourthly, we considered whether it would be possible to identify nouns that have strong emotional impact from the qualitative responses from users.

[0005] Therefore, a first object of the present invention is to provide a program, device, and method that can obtain evaluable expectations and satisfaction levels by asking a variety of questions to users who are the subject of a survey. [Means for solving the problem]

[0006] <First aspect of the present invention> According to a first aspect of the present invention, there is provided a program for causing a computer to function so as to collect a questionnaire from a user based on an evaluation item, the program comprising: The evaluation items are questionnaire items for multiple users belonging to an organization or a service, a question creation means for sending a prompt for instructing the creation of a question about the future and a question about the present for each evaluation item to a generation AI (Artificial Intelligence) server, and receiving the question about the future and the question about the present from the generation AI server; an answer collection means for transmitting questions about the future and questions about the present to each user terminal, receiving answers to each question from the terminal, and storing the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels, respectively, for each user, in an answer database; and make it work 、 The question creation means includes attribute information of the organization or service in the prompt to create a question based on the organization or service. like It is characterized by making a computer function.

[0008] According to another embodiment of the program of the first invention, The question creating means includes, in the prompt, either or a combination of user attribute information and / or past answer information of the user in order to create each question for each user. It is also preferable to function the computer in such a manner.

[0009] According to another embodiment of the program of the first invention, The question creation means sends a prompt to the generation AI server requesting that the question to be created be a quantitative answer. It is also preferable to function the computer in such a manner.

[0010] According to another embodiment of the program of the first invention, The method for creating questions is sending prompts to the generation AI server instructing it to create a forward question about the future, a paradoxical question about the future, a forward question about the present, and a paradoxical question about the present; receiving from the generation AI server a forward question asking about the future, a paradoxical question asking about the future, a forward question asking about the present, and a paradoxical question asking about the present; The means of collecting responses were: Quantitative answers to forward questions about the future were taken as positive expectations, and quantitative answers to reverse questions about the future were taken as negative expectations. Quantitative answers to forward questions about the present are treated as positive satisfaction, and quantitative answers to reverse questions about the present are treated as negative satisfaction, and are stored in the answer database. It is also preferable to function the computer in such a manner.

[0011] According to another embodiment of the program of the first invention, The question creation means sends a prompt to the generation AI server requesting that the question to be created be a qualitative answer; The answer collection means receives, from each user terminal, a qualitative answer to a question asking about the future and a qualitative answer to a question asking about the present; a qualitative answer estimation means for transmitting to a generation AI server a prompt requesting a quantitative answer to the question asking about the future and a quantitative answer to the question asking about the present from the qualitative answer to the question asking about the future and the qualitative answer to the question asking about the present, and receiving from the generation AI server the quantitative answer to the question asking about the future and the quantitative answer to the question asking about the present; It is also preferable to make the computer function as follows.

[0012] According to another embodiment of the program of the first invention, Qualitative answers are user statements, The qualitative answer estimation means transmits prompts for a quantitative answer to the question about the future and a quantitative answer to the question about the present to the generation AI server based on a sentiment analysis of the sentence. It is also preferable to function the computer in such a manner.

[0013] According to another embodiment of the program of the first invention, an evaluation comment acquisition means for transmitting a prompt to the generation AI server for each evaluation item or each user, instructing the creation of an evaluation comment based on the expectation and satisfaction of the response database, and further receiving the evaluation comment for the evaluation item or the user from the generation AI server, and displaying the evaluation comment on an administrator terminal; It is also preferable to make the computer function as follows.

[0014] According to another embodiment of the program of the first invention, an evaluation index acquisition means for transmitting a prompt to the generation AI server for each evaluation item or each user, instructing the derivation of an evaluation index from the expectation and satisfaction of the response database, receiving an evaluation index for the evaluation item or the user from the generation AI server, and displaying the evaluation index on an administrator terminal; It is also preferable to make the computer function as follows.

[0015] According to a first aspect of the present invention, in a management device that collects a questionnaire from users based on evaluation items, The evaluation items are questionnaire items for multiple users belonging to an organization or a service, a question creation means for sending a prompt for instructing the creation of a question about the future and a question about the present for each evaluation item to a generation AI (Artificial Intelligence) server, and receiving the question about the future and the question about the present from the generation AI server; an answer collection means for transmitting questions about the future and questions about the present to each user terminal, receiving answers to each question from the terminal, and storing the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels, respectively, for each user, in an answer database; Have death, The question creation means includes attribute information of the organization or service in the prompt to create a question based on the organization or service. It is characterized by:

[0016] According to a first aspect of the present invention, there is provided a method for managing a device that collects questionnaires from users based on evaluation items, comprising the steps of: The evaluation items are questionnaire items for multiple users belonging to an organization or a service, The device is A first step of sending a prompt to a generation AI (Artificial Intelligence) server for each evaluation item, the prompt instructing the generation of a question asking about the future and a question asking about the present, and receiving the question asking about the future and the question asking about the present from the generation AI server; a second step of transmitting questions about the future and questions about the present to each user terminal, receiving answers to each question from the terminal, and storing the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels, respectively, for each user, in an answer database; Run The first step is to include attribute information of the organization or service in the prompt to create a question based on the organization or service. It is characterized by:

[0017] <The second invention> According to the second aspect of the present invention, there is provided a program for causing a computer to function so as to derive an evaluation index from a predetermined range of response values ​​from a user to a questionnaire based on an evaluation item, the program comprising: A training list is created by configuring a row in which the same answer value is listed for all evaluation items with multiple rows in which different answer values ​​are listed, and setting an evaluation index for the answer value for each row. In the training stage, the answer values ​​of all evaluation items are associated with the evaluation index for each row of the training list, and training is performed. In the estimation stage, the answer values ​​of all evaluation items that are the target data are input, and a machine learning model that estimates the evaluation index is created. The present invention is characterized in that it causes a computer to function.

[0018] According to another embodiment of the program of the second invention, The training list is For the center row, the median is assigned to all the answer values ​​of the evaluation items, and the median is associated with the evaluation index. For the highest-ranked row, the highest value is assigned to all answer values ​​of the evaluation items, and the highest value is assigned to the evaluation index. For the lowest-ranked row, the lowest value is assigned to all answer values ​​of the evaluation items, and the lowest value is associated with the evaluation index. From the center row to the highest row, increment all answer values ​​of the evaluation items and increment the evaluation index to match them. From the center row to the lowest row, all answer values ​​of the evaluation items are decremented and the evaluation index is decremented to correspond. It is also preferable to function the computer in such a manner.

[0019] According to another embodiment of the program of the second invention, The answer value is a pair of expectation and satisfaction based on the answer received from each user, The median answer value is the value where expectation and satisfaction level match. The highest response value is the value that results in the lowest expectation and the highest satisfaction. The lowest response value is the value that results in the highest expectation and the lowest satisfaction. It is also preferable to function the computer in such a manner.

[0020] According to another embodiment of the program of the second invention, The answer value is the satisfaction level based on the answers received from each user divided by the expectation level, The median is 1.0, The evaluation index is a normalized value that falls within a specified range above and below the median value. It is also preferable to function the computer in such a manner.

[0021] According to another embodiment of the program of the second invention, The training list is For the center row, the median is assigned to all the answer values ​​of the evaluation items, and the median is associated with the evaluation index. For the highest-ranked row, the highest value is assigned to all answer values ​​of the evaluation items, and the lowest value is assigned to the evaluation index. For the lowest-ranked row, the lowest value is assigned to all answer values ​​of the evaluation items, and the highest value is assigned to the evaluation index. From the center row to the highest row, increment all answer values ​​of the evaluation items and decrement the evaluation index to match them. From the center row to the lowest row, all answer values ​​of the evaluation items are decremented and the evaluation index is incremented to correspond. It is also preferable to function the computer in such a manner.

[0022] According to another embodiment of the program of the second invention, The answer value is a pair of expectation and satisfaction based on the answer received from each user, The median answer value is the value where expectation and satisfaction level match. The highest response value is the value that results in the highest expectation and the lowest satisfaction. The lowest response value is the value that results in the lowest expectation and the highest satisfaction. It is also preferable to function the computer in such a manner.

[0023] According to another embodiment of the program of the second invention, The answer value is the expectation based on the answer received from each user divided by the satisfaction level, The median is 1.0, The evaluation index is a normalized value that falls within a specified range above and below the median value. It is also preferable to function the computer in such a manner.

[0024] According to another embodiment of the program of the second invention, A machine learning model is a library or generative AI server that applies decision tree algorithms. It is also preferable to have the computer function as:

[0025] <The third aspect of the present invention> According to a third aspect of the present invention, there is provided a program for causing a computer to function that accumulates qualitative answers to questions in a questionnaire for users, the program comprising: The questions are a mixture of future and present questions, a qualitative answer estimation means for estimating an emotion score for a sentence using a sentiment analysis model, estimating a higher evaluation index as the sentiment score is more positive, and estimating a lower evaluation index as the sentiment score is more negative; The present invention is characterized in that it causes a computer to function.

[0026] <The fourth invention> According to a fourth aspect of the present invention, there is provided a program for causing a computer to function that accumulates qualitative answers to questions in a questionnaire for users, the program comprising: The questions are a mixture of future and present questions, A positive / negative classification means for classifying text as positive or negative using a sentiment analysis model; a noun extraction means for extracting nouns from a sentence using a morphological analysis model; a positive / negative counting means for counting, for each noun, the number of sentences including the noun classified as positive and the number of sentences including the noun classified as negative; a noun specific display means for visually highlighting and displaying a noun having a large number of negative numbers relative to a predetermined condition or more among the plurality of nouns; The present invention is characterized in that it causes a computer to function. Effect of the Invention

[0027] According to the program, device, and method of the present invention, firstly, by asking a variety of questions to users who are the subject of a survey, it is possible to obtain evaluable expectations and satisfaction levels. [Brief description of the drawings]

[0028] [Figure 1] FIG. 2 is a functional configuration diagram of a management device according to the first aspect of the present invention. [Diagram 2] FIG. 4 is an explanatory diagram of a user attribute database. [Diagram 3] FIG. 13 is an explanatory diagram of a response database. [Figure 4] FIG. 11 is a first explanatory diagram of a question generating unit. [Diagram 5] These are questions for each user that are created by the question creation unit. [Figure 6] FIG. 2 is a second explanatory diagram of the question creation unit. [Figure 7] FIG. 13 is an explanatory diagram of a qualitative answer estimation unit. [Figure 8] FIG. 11 is a diagram showing an optional functional configuration of a management device according to the first aspect of the present invention. [Figure 9] 4 is an explanatory diagram of a target user extraction unit, an item-specific answer calculation unit, and a statistical database. FIG. [Figure 10] FIG. 13 is an explanatory diagram of an evaluation comment acquisition unit. [Figure 11] FIG. 13 is an image showing intermediate items as evaluation items. [Figure 12] FIG. 13 is an explanatory diagram of an evaluation index acquisition unit in the first training stage. [Figure 13] FIG. 11 is a first explanatory diagram of an evaluation index acquisition unit in the first estimation stage. [Figure 14] FIG. 11 is a second explanatory diagram of the evaluation index acquisition unit in the first estimation stage. [Figure 15] FIG. 13 is an explanatory diagram of an evaluation index acquisition unit in the second training stage. [Figure 16] FIG. 13 is an explanatory diagram of an evaluation index acquisition unit in the second estimation stage. [Figure 17] FIG. 13 is a functional configuration diagram of a management device according to the third aspect of the present invention. [Figure 18] FIG. 13 is a functional configuration diagram of a management device according to the fourth aspect of the present invention. [Figure 19] 13 is an explanatory diagram of a positive / negative counting section and a noun identification display section. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0030] FIG. 1 is a functional configuration diagram of a management device according to the first aspect of the present invention.

[0031] 1, the management device 1 collects questionnaires from users based on evaluation items, and displays the answers to the questionnaires from a plurality of users to an administrator. In recent years, emphasis has been placed on "engagement," which means "trust" and "goodwill" toward organizations and services. If the organization is, for example, a company, the manager corresponds to the business owner, and the user corresponds to the employee. If the service is, for example, a financial service, the manager corresponds to the customer manager, and the user corresponds to the customer user. For example, a manager can recognize areas in which the company should improve from the results of a survey of employees. Also, for example, a customer manager can recognize areas in which the service should be improved from the results of a survey of customer users. It is desirable for managers and customer managers to operate their organizations and services in such a way that the members of those organizations or services have a high level of engagement.

[0032] Let us consider as an example a system in which corporate management evaluates the engagement of its employees. Managers are required to maintain high levels of engagement so that employees can work together as a unit. However, even if managers try to improve the "work environment" for employees, in reality, employees may place more importance on "evaluation and compensation." In this case, improving the work environment is not an important issue for employees. This also varies depending on employees' organizational attributes (department, position, work location, years of service) and personal attributes (gender, age, educational background, family structure), and is also an issue that cannot be measured directly.

[0033] In response to this, there is a system that collects answers to survey questions based on members' "expectations" and "satisfaction." In this system, members are asked to answer their expectations and satisfaction with values ​​ranging from 1 to 5, for example. From the expectations and satisfaction based on the members' engagement, it is possible to identify areas for improvement in the organization or service. In particular, the management device 1 of the present invention can collect the expectations and satisfaction levels of users by asking a variety of questions to the users who are the subjects of the survey.

[0034] The management device 1 has a Web interface as a website server. The management device 1 also communicates with a generation AI server 3 that creates questions, and a user terminal 4 that transmits the questions to users to have them answer them. The administrator terminal 2 accesses the management device 1 from a browser or application that is pre-installed. The user terminal 4 also accesses the management device 1 through a browser or application that is pre-installed.

[0035] <First aspect of the present invention> 1, the management device 1 of the present invention includes a user attribute database 101, an answer database 102, a question creation unit 11, an answer collection unit 12, and a qualitative answer estimation unit 13. These functional components may be realized by executing a program that causes a computer installed in the device to function. The process flow of these functional components can also be understood as a question creation method.

[0036] [Generation AI Server 3] The generative AI (generative artificial intelligence) server 3 understands the context of a question and generates a sentence in natural language. Examples include ChatGPT (registered trademark) from OpenAI, Claude (registered trademark) from Anthropic, Gemini (registered trademark) from Google, and Llama (registered trademark) from Meta. These are pre-trained with general language knowledge from a large-scale text corpus. In addition, the generative AI server 3 can improve the accuracy of the sentence by repeating fine tuning.

[0037] [User Attribute Database 101] FIG. 2 is an explanatory diagram of the user attribute database.

[0038] The user attribute database 101 accumulates attribute information for each user. For example, in the case of an organization such as a company, there are organizational attributes and / or personal attributes. According to Fig. 6, for each user ID (identifier), organizational attributes such as "department," "job title," "work location," and "years of employment" are recorded, while personal attributes such as "gender," "age," "educational background," and "family structure" are recorded. Departments include, for example, the sales department, development department, and general affairs department. Job titles may include, for example, department manager, section manager, group leader, no job title, personnel evaluation rank, and so on.

[0039] [Question Creation Section 11] The question creation unit 11 sends a prompt to the generation AI server 3 to instruct it to create a "question asking about the future" and a "question asking about the present" for each evaluation item. In response, it receives the "question asking about the future" and the "question asking about the present" from the generation AI server 3 and stores them in the answer database 102.

[0040] FIG. 3 is an explanatory diagram of the response database.

[0041] 3, the response database 102 classifies, for example, employee engagement in a company into a hierarchical structure of major items, medium items, and minor items. These items are hierarchically organized survey items for multiple users belonging to an organization or service. According to the embodiment of the present invention, an example will be described in which the "evaluation items" are medium items, but of course they may be major items or minor items.

[0042] According to Figure 3, assuming an organization such as a company, the major categories are divided into three categories: "Work content and environment," "Human relationships," and "Organizational policy and culture." The medium categories are further divided into four for a total of 12 categories. The minor categories are further divided into five for each medium category.

[0043] The administrator can freely edit the item categories and item names from the administrator terminal 2. In other words, the administrator can freely decide what viewpoints to focus on in order to improve the evaluation items. The content of the questions can also be freely changed in accordance with changes to the content and number of major, medium, and minor items. FIG. 3 shows an example of items assumed to be company employees. Of course, it may be an example of items assumed to be user members of financial services. The administrator can edit the item categories and item names as desired.

[0044] In the response database 102, the expectation and satisfaction level for each evaluation item (for example, sub-item) are recorded for each user. For example, multiple questions are presented to the user about sub-items based on major and medium-sized items, and the answers to those questions are collected. Then, from the collected answers, the expectation level and the satisfaction level are derived for each evaluation item and / or each subitem and recorded in the answer database 102. The expectation level and the satisfaction level may be expressed as a numerical value from 1 to 5, for example. For example, multiple answers may be obtained from multiple questions regarding the major item "job content and environment," the medium item "job characteristics," and the small item "clarity of roles and responsibilities," and expectations and satisfaction levels are derived from those answers.

[0045] FIG. 4 is a first explanatory diagram of the question generating unit.

[0046] The question creating unit 11 creates a prompt for instructing the creation of a "question asking about the future" and a "question asking about the present" for each combination of a major item, an evaluation item (medium item), and a minor item in the answer database 102.

[0047] At this time, the question creation unit 11 includes the <attribute information> of the organization or service in the prompt in order to create a question based on the organization or service. This allows a question to be created according to the organization or service. The attribute information may be, for example, a company profile or IR (Investor Relations) information.

[0048] Furthermore, in order to create questions for each user, the question creation unit 11 may include in the prompt any one or a combination of the user attribute information {user_profile} of the user, the user's past answers {result}, and the user's past expectations and satisfaction {result}. This allows the creation of questions appropriate for each user.

[0049] <Quantitative answer> The question creation unit 11 may send a prompt to the generation AI server 3 requesting that the question to be created be a "quantitative answer." The quantitative answer may be one that prompts the user to select one of multiple options (e.g., 1, 2, 3, 4, 5) in response to the question, or a numerical value from a predetermined range (e.g., 1.0 to 5.0). The numerical value may be input by the user entering the numerical value itself, or may be specified by shifting a slide bar.

[0050] According to FIG. 4, for example, the following prompt is sent to the generation AI server 3: -------------------------------------------------------------------------------- {company_profile} Based on {user_profile} Create "questions about the future" and "questions about the present" for employees based on the major category "job content and environment," the medium category "job characteristics," and the small category "clarity of roles and responsibilities." Each question should have a quantitative answer (1-5). -------------------------------------------------------------------------------- {company_profile}: Organization profile information {user_profile}: User attribute information In response, the question creation unit 11 receives a “question asking about the future” and a “question asking about the present” from the generation AI server 3.

[0051] According to FIG. 4, the following questions are created by the generation AI server 3: -------------------------------------------------------------------------------- (Question about the future) “Would you like your roles and responsibilities to be more clearly defined? Please answer from 1: Don't want to ~ 5: Want to." (Question about the present) “Do you feel your roles and responsibilities are clear? Please answer from 1: I don't feel it to 5: I feel it." --------------------------------------------------------------------------------

[0052] FIG. 5 shows questions for each user created by the question creating unit.

[0053] 5, a "question asking about the future" and a "question asking about the present" corresponding to each sub-item are created for each user. In order to obtain detailed answers, a plurality of "questions asking about the future" and a plurality of "questions asking about the present" may be mixed.

[0054] <Qualitative answer> FIG. 6 is a second explanatory diagram of the question generating unit.

[0055] The question creation unit 11 sends a prompt to the generation AI server 3 requesting that the question to be created be a "qualitative answer." A qualitative answer is, for example, a sentence input by a user. According to FIG. 6, for example, the following prompt is sent to the generation AI server 3: -------------------------------------------------------------------------------- {company_profile} Based on {user_profile} Create "questions about the future" and "questions about the present" for employees based on the major category "job content and environment," the medium category "job characteristics," and the small category "clarity of roles and responsibilities." Please ensure your questions are qualitative answers (sentences). -------------------------------------------------------------------------------- In response to this, the question creation unit 11 receives a question that will be a qualitative answer from the generation AI server 3.

[0056] According to FIG. 6, the following questions are created by the generation AI server 3: -------------------------------------------------------------------------------- (qualitative question) Please feel free to describe below your future roles and responsibilities, and your current roles and responsibilities. --------------------------------------------------------------------------------

[0057] The question creation unit 11 outputs the questions for each sub-item and each user received from the generation AI server 3 to the answer collection unit 12.

[0058] [Response Collection Section 12] The answer collection unit 12 transmits the "question asking about the future" and the "question asking about the present" to each user terminal 4, and receives the answer for each question from the user terminal 4. The questions may be ones that result in quantitative answers or ones that result in qualitative answers. The answer collection unit 12 stores, for each user, answers to questions about the future as "expectations" and answers to questions about the present as "satisfaction" in the answer database 102. Here, when the answer collection unit 12 receives a qualitative answer to a question asking about the future and a qualitative answer to a question asking about the present from each user terminal 4, it outputs those qualitative answers to the qualitative answer estimation unit 13. The sentence serving as the qualitative answer may be a text input by the user himself / herself to the user terminal 4, or may be a text obtained by speech recognition based on speech input from a microphone.

[0059] [Qualitative answer estimation part 13] FIG. 7 is an explanatory diagram of the qualitative answer estimation unit.

[0060] The qualitative answer estimation unit 13 sends a prompt to the generation AI server 3 requesting a "quantitative answer to the question asking about the future" and a "quantitative answer to the question asking about the present" from the "qualitative answer to the question asking about the future" and the "qualitative answer to the question asking about the present". The prompts may then ask for a quantitative answer to the question about the future and a quantitative answer to the question about the present based on a sentiment analysis of the qualitative answer sentence.

[0061] Here, the generation AI server 3 processes the sentences using text mining, divides them into words and phrases, and analyzes their frequency of appearance, correlation of co-appearance, appearance trends, time series, etc. This allows a sentiment analysis to be performed on the "description in response to a question about the future" and the "description in response to a question about the present", and quantitative answers for each are derived.

[0062] According to FIG. 5, for example, the following prompt is sent to the generation AI server 3: -------------------------------------------------------------------------------- Based on {company_profile}, Please estimate your expectations and satisfaction levels based on a sentiment analysis of the following qualitative responses (sentences) for the major category "Job content and environment," the medium category "Job characteristics," and the small category "Clarity of roles and responsibilities." <Qualitative answer> --------------------------------------------------------------------------------

[0063] In response to this, the qualitative answer estimation unit 13 receives from the generation AI server 3 a quantitative answer (for example, any one of 1 to 5) to the question asking about the future, and a quantitative answer to the question asking about the present.

[0064] <Collecting answers to normal and paradoxical questions> (S1) The question creation unit 11 sends a prompt to the generation AI server instructing it to create a forward question asking about the future, a paradoxical question asking about the future, a forward question asking about the present, and a paradoxical question asking about the present. (S2) Receive from the generation AI server 3 a forward question asking about the future, a paradoxical question asking about the future, a forward question asking about the present, and a paradoxical question asking about the present. (S3) The answer collection unit 12 stores in the answer database quantitative answers to forward questions asking about the future as positive expectations, quantitative answers to paradoxical questions asking about the future as negative expectations, quantitative answers to forward questions asking about the present as positive satisfaction levels, and quantitative answers to paradoxical questions asking about the present as negative satisfaction levels.

[0065] <Relationship between questions and answers regarding expectations and satisfaction> According to the embodiment of the present invention, the questions with defined expectations or with defined satisfaction are not directly asked to the user, i.e., they are not based on answers directly obtained by questions such as: Expectations: "Do you have high expectations for...?" (answer 1 to 5) Satisfaction: "Are you satisfied with ~?" (answer 1 to 5) In contrast, according to an embodiment of the present invention, a question is created by sending a prompt to the generation AI server 3 to create a question for the user (member). Furthermore, from the answer to that question, expectations and satisfaction levels are obtained using a machine learning model and the generation AI server 3. This makes it possible to derive expectations and satisfaction levels even if diverse answers are obtained from diverse questions.

[0066] FIG. 8 is a diagram showing an optional functional configuration of the management device in the first aspect of the present invention.

[0067] According to Fig. 8, in addition to the functional components of Fig. 1, it further includes a target user extraction unit 14, an item-specific answer calculation unit 15, a statistical database 103, an evaluation index acquisition unit 17, a machine learning model, and an evaluation comment acquisition unit 16. These functional components may be realized by executing a program that causes a computer mounted on the device to function. The process flow of these functional components can also be understood as an evaluation acquisition method.

[0068] FIG. 9 is an explanatory diagram of the target user extraction unit, the item-specific answer calculation unit, and the statistical database.

[0069] [Target user extraction part 14] The target user extraction unit 14 refers to the user attribute database 101 to extract only the expectation and satisfaction levels derived from only the target users having the "attribute information" specified by the administrator. For example, the target users may be "all employees." This allows you to select improvement items for all employees in the company. Also, for example, it is possible to extract "only employees with three years or less of service" as target users, or "only those with the title of section manager" as target users. The extracted target users (for example, IDs) are output to the item-specific answer calculation unit 15.

[0070] [Item-specific answer calculation section 15] The item-specific answer calculation unit 15 calculates the degree of expectation and the degree of satisfaction for each item from the degrees of expectation and the degree of satisfaction of the multiple target users extracted by the target user extraction unit 14. 9, a data group is created for only the target user to be stored in the statistical database 103. This makes it possible to derive only the expectation and satisfaction levels for a plurality of evaluation items in a hierarchy specified by the administrator.

[0071] [Statistical Database 103] The statistical database 103 is divided into a hierarchy of major items, medium items, and minor items, similar to the response database 102. The expectation and satisfaction level derived by the item-specific response calculation unit 15 are recorded for each item.

[0072] 9, the expectations and satisfaction levels in the statistical database 103 are based on answers to questions based on organizations or services. That is, the expectations and satisfaction levels are statistical values ​​calculated from answers collected from multiple users. Specifically, for example, the expectation and satisfaction of each small item are statistical values ​​such as the average or median of the expectation and satisfaction collected from a plurality of users. Furthermore, the expectation and satisfaction of a medium item may be expressed as a statistical value such as the average or median of the expectation and satisfaction of all the small items belonging to that medium item. Similarly, the expectation and satisfaction of a major category may be expressed as statistical values ​​such as the average or median of the expectation and satisfaction of all the sub-categories belonging to that major category.

[0073] [Evaluation comment acquisition section 16] FIG. 10 is an explanatory diagram of the evaluation comment acquisition unit.

[0074] The evaluation comment acquisition unit 16 sends a prompt to the generation AI server 3 for each evaluation item or for each user, instructing the generation AI server 3 to create an "evaluation comment" based on the expectation and satisfaction level in the statistical database 103. Then, the evaluation comment acquisition unit 16 further receives the "evaluation comment" for that evaluation item or that user from the generation AI server 3, and displays that evaluation comment on the administrator terminal 2.

[0075] According to FIG. 10, for example, for the entire company, the following prompt is sent to the generation AI server 3: -------------------------------------------------------------------------------- Based on {company_profile}, Main item: "Work content and environment" Sub-item "Characteristics of the Work" Clarity of roles and responsibilities:<Expectations><Satisfaction> Meaning of work:<Expectations><Satisfaction> Skills Utilization:<Expectations><Satisfaction> .................................................................................................................................... Medium item “Work environment” Comfort of the work environment: <Expectations><Satisfaction> Work-life balance:<Expectations><Satisfaction> .................................................................................................................................... .................................................................................................................................... Major item: "Organizational policy and culture" Subsection "Vision and Mission" .................................................................................................................................... Please create your evaluation comment from here. --------------------------------------------------------------------------------

[0076] In response to this, the evaluation comment acquisition unit 16 receives evaluation comments giving advice about the organization of the company from the generation AI server 3. By instructing the prompt to point out "good points" and "bad points," it is possible to receive text and images describing "good points" and "bad points" about "job characteristics" in the evaluation comments. The evaluation comments are presented to the manager as AI insights for encouraging the current situation and improvement.

[0077] [Evaluation Index Acquisition Section 17] The evaluation index acquisition unit 17, like the evaluation comment acquisition unit 16, sends a prompt to the generation AI server 3 to instruct the creation of an "evaluation index" from the expectations and satisfaction levels in the statistical database 103 for each evaluation item or for each user. The evaluation index acquisition unit 17 then further receives the "evaluation index" for that evaluation item or that user from the generation AI server 3, and displays that evaluation index on the administrator terminal 2. The evaluation index is a value that comprehensively evaluates the expectation and satisfaction levels, and can be used to determine whether the organization or the entire service is in an ideal state for the user.

[0078] The evaluation index is associated with the following expectations and satisfaction levels: High expectations and high satisfaction: Users have high expectations of an organization or service and are actually highly satisfied (ideally good items) = the evaluation index is close to the median. High expectations / low satisfaction: Although users have high expectations of an organization or service, they are dissatisfied because their expectations are not met (the item most in need of improvement) = the evaluation index is high. Low expectations / high satisfaction: The user had no expectations for the organization or service, but was more satisfied than expected (unnecessarily good item) = the evaluation index is low. Low expectations / low satisfaction: Users have no expectations of the organization or service and are not satisfied (items that no one pays attention to) = Evaluation index is close to the median. According to these views, it is preferable for the plots of all endpoints to be as close to the median (e.g., 100%) as possible.

[0079] FIG. 11 is an image showing intermediate items as evaluation items.

[0080] According to FIG. 11, the following items are displayed in a hierarchical manner: Major item [Work content and environment] ->Sub-item [Work characteristics] -> Sub-item [Clarity of roles and responsibilities] [Meaning of work] [Skill Utilization] [Opportunities for growth] Here, when a medium-level item is selected by the administrator, multiple small items belonging to that medium-level item are displayed. At this time, "expectation level," "satisfaction level," and "evaluation index (value index)" are displayed for each medium-level item and each small item. The evaluation index shown to the administrator does not include the difference between expectation level and satisfaction level, but displays a normalized value within a specified range (for example, 20% to 180%). The lower the evaluation index, the more it can be recognized that the evaluation item needs improvement. On the other hand, the higher the evaluation index, the more it can be recognized that the evaluation item is unnecessarily fulfilling.

[0081] <The second invention> The evaluation index acquisition unit 17 can acquire an optimal evaluation index by training the machine learning model or the generation AI server 3. The evaluation index acquisition unit 17 derives an evaluation index from a "predetermined range of response values" from a user for a questionnaire based on an evaluation item.

[0082] FIG. 12 is an explanatory diagram of the evaluation index acquisition unit in the first training stage.

[0083] The evaluation index acquiring unit 17 may have a machine learning model. As the machine learning model, an external generation AI server 3 may be used. A machine learning model is a library or generation AI server that applies a decision tree algorithm. Specifically, for example, XGBoost (eXtreme Gradient Boosting) can be used. This is an implementation of the Gradient Boosting Decision Tree algorithm. It works by creating decision trees one by one, weighting incorrect data based on the majority vote of the previous decision trees, and proceeding with training to cover the mistakes in the next decision tree, thus avoiding overlearning.

[0084] FIG. 12 shows an example of deriving an evaluation index for a medium item from response values ​​for multiple small items. The "answer value" is a "pair of expectation and satisfaction level" based on the answer received from each user for each minor item. First, create a <training list> in advance. The training list is composed of multiple rows in which the same answer value is listed for all evaluation items, and an evaluation index is set for the answer value for each row.

[0085] And the machine learning model works as follows: (Training stage) For each row of the training list, training is performed by associating answer values ​​of all evaluation items with evaluation indices. (Estimation stage) The response values ​​of all evaluation items that are the target data are input, and the evaluation index is estimated.

[0086] 12, rows are arranged for all combinations of expectation levels 5 to 1 and satisfaction levels 1 to 5. Here, the expectation and satisfaction response values ​​are classified into the following two patterns according to the configuration. <First pattern> The median response value is the value where expectation and satisfaction levels match. The highest response value is the value that results in the "lowest expectation" (e.g., expectation level 1) and the "highest satisfaction" (e.g., satisfaction level 5). The minimum response value is the value that results in the "highest expectation" (e.g., expectation level 5) and the "lowest satisfaction" (e.g., satisfaction level 1). - "Evaluation index = Satisfaction / Expectations". The evaluation index is a normalized value with a median of 1.0, falling within a specified range above and below the median.

[0087] The training list is set up as follows: For the center row, the median (expectation = satisfaction) is assigned to all response values ​​for the evaluation items, and the median (1,00) is assigned to the evaluation index. For the highest-ranked row, the highest value (Expectation 1, Satisfaction 5) is assigned to all answer values ​​for the evaluation items, and the evaluation index is assigned the "highest value (Satisfaction / Expectation = 5.00 -> 180%)". For the lowest-ranked row, the minimum value (Expectation 5, Satisfaction 1) is assigned to all response values ​​for the evaluation items, and the evaluation index is assigned the "minimum value (Satisfaction / Expectation = 0.20->20%)". From the center row to the highest row, increment all answer values ​​of the evaluation items and "increment" the evaluation index to match. From the center row to the lowest row, all response values ​​for the evaluation items are decremented and the evaluation index is "decremented" to correspond.

[0088] According to FIG. 12, for all small items, expectation level 1 and satisfaction level 2 are associated with an evaluation index of 120% based on the evaluation index (satisfaction level / expectation level=2.00). Furthermore, for all minor items, an evaluation index of 80% based on the evaluation index (satisfaction / expectation=0.80) is associated with expectation level 5 and satisfaction level 4.

[0089] <Second pattern> The median response value is the value where expectation and satisfaction levels match. The highest response value is the value that results in the "highest expectation" (e.g., expectation level 5) and the "lowest satisfaction" (e.g., satisfaction level 1). The minimum response value is the value that results in the "lowest expectation" (e.g., expectation level 1) and the "highest satisfaction" (e.g., satisfaction level 5). - "Evaluation index = expectation / satisfaction." The evaluation index is a normalized value with a median of 1.0, falling within a specified range above and below the median.

[0090] The training list is set up as follows: For the center row, the median (expectation = satisfaction) is assigned to all response values ​​for the evaluation items, and the median (1,00) is assigned to the evaluation index. For the highest-ranked row, the maximum value (expectation 5, satisfaction 1) is assigned to all answer values ​​for the evaluation items, and the evaluation index is assigned the minimum value (expectation / satisfaction = 5.00->20%). For the lowest-ranked row, the minimum value (expectation 1, satisfaction 5) is assigned to all response values ​​for the evaluation items, and the maximum value (expectation / satisfaction = 0.20->180%) is assigned to the evaluation index. From the center row to the highest row, all answer values ​​for the evaluation items are incremented and the evaluation index is "decremented" to correspond. From the center row to the lowest row, all response values ​​for the evaluation items are decremented and the evaluation index is "incremented" to correspond.

[0091] According to FIG. 12, an evaluation index of 120% based on the evaluation index (expectation level / satisfaction level=0.50) is associated with expectation level 1 and satisfaction level 2 for all small items. Furthermore, for all minor items, an expectation level of 5 and a satisfaction level of 4 are associated with an evaluation index of 80% based on the evaluation index (expectation level / satisfaction level=1.25).

[0092] FIG. 13 is a first explanatory diagram of the evaluation index acquisition unit in the first estimation stage. According to FIG. 13, response values ​​(expectations and satisfaction levels) of a plurality of subitems based on the evaluation items of the medium item "work characteristics" that is the target data are input, and the evaluation index of the medium item "work characteristics" is estimated.

[0093] FIG. 14 is a second explanatory diagram of the evaluation index acquisition unit in the first estimation stage. According to FIG. 14, response values ​​(expectations and satisfaction levels) for multiple medium items based on the target data for the entire company are input, and a comprehensive evaluation index for the entire company is estimated.

[0094] FIG. 15 is an explanatory diagram of the evaluation index acquisition unit in the second training stage.

[0095] According to Figure 15, rows for all combinations of answer values ​​from 5.0 to 0.2 are arranged. The training list is set as follows: <First pattern> Answer value = Satisfaction / Expectation For the center row, the median (1.0) is assigned to all response values ​​for the evaluation items, and the median (100%) is associated with the evaluation index. For the highest-ranked row, the highest value (for example, satisfaction level 5 / expectation level 1 = 5.00) is assigned to all response values ​​for the evaluation items, and the evaluation index is assigned the "highest value (180%)." For the lowest-ranked row, the minimum value (for example, satisfaction level 1 / expectation level 5 = 0.20) is assigned to all response values ​​for the evaluation items, and the "minimum value (20%)" is assigned to the evaluation index. From the center row to the highest row, increment all answer values ​​of the evaluation items and "increment" the evaluation index to match. From the center row to the lowest row, all response values ​​for the evaluation items are decremented and the evaluation index is "decremented" to correspond.

[0096] <Second pattern> Answer value = Expectation / Satisfaction For the center row, the median (1.0) is assigned to all response values ​​for the evaluation items, and the median (100%) is associated with the evaluation index. For the highest-ranked row, the minimum value (for example, expectation 1 / satisfaction 5 = 0.20) is assigned to all response values ​​of the evaluation items, and the "maximum value (180%)" is assigned to the evaluation index. For the lowest-ranked row, the maximum value (for example, expectation 5 / satisfaction 1 = 5.00) is assigned to all answer values ​​of the evaluation items, and the "minimum value (20%)" is assigned to the evaluation index. From the center row to the highest row, all answer values ​​for the evaluation items are decremented and the evaluation index is "incremented" to correspond. From the center row to the lowest row, all answer values ​​for the evaluation items are incremented and the evaluation index is "decremented" to correspond.

[0097] FIG. 16 is an explanatory diagram of the evaluation index acquisition unit in the second estimation stage. According to FIG. 16, response values ​​(satisfaction / expectations: pattern 1) for multiple medium items based on the target data for the entire company are input, and a comprehensive evaluation index for the entire company is estimated.

[0098] That is, according to the embodiment of the present invention, an evaluation index is not calculated based on the difference between the expectation level and the satisfaction level, but rather an evaluation index that is highly evaluated when the difference between the expectation level and the satisfaction level is small for each evaluation item is not calculated.

[0099] <The third aspect of the present invention> FIG. 17 is a functional configuration diagram of a management device according to the third aspect of the present invention.

[0100] 17, the management device 1 has an answer collection unit 12 and a qualitative answer estimation unit 13. These functional components are realized by executing a program that causes a computer mounted on the management device 1 to function. The process flow of these functional components can also be understood as an estimation method.

[0101] The answer collection unit 13 transmits questions, including questions about the future and questions about the present, to the user terminal 4 as a questionnaire for the user. In response, the answer collection unit 13 receives a text of a qualitative answer to the question from the user terminal 4. The received text is output to the qualitative answer estimation unit 13.

[0102] The qualitative answer estimation unit 13 estimates an emotion score for a sentence using the emotion analysis model 5. Specifically, TextBlob or NTLK can be used. As for the emotion score that is output, a positive value (0 to +1) indicates a positive emotion, and a negative value (-1 to 0) indicates a negative emotion. The qualitative answer estimation unit 13 estimates a higher evaluation index the more positive the emotion score is, and estimates a lower evaluation index the more negative the emotion score is. Here, it is noted that the questions posed to the user are a mixture of questions about the future and questions about the present. If the answer to the question about the future is the expectation level and the answer to the question about the present is the satisfaction level, then a positive answer that is a mixture of these can be recognized as a high evaluation index because the expectation level and the satisfaction level are close in value. On the other hand, a negative answer that is a mixture of these can be recognized as a low evaluation index because the expectation level and the satisfaction level are far apart in value.

[0103] <The fourth invention> FIG. 18 is a functional configuration diagram of a management device in the fourth aspect of the present invention.

[0104] 18, the management device 1 has a response collection unit 12, a positive / negative classification unit 181, a noun extraction unit 182, a positive / negative counting unit 183, and a noun identification display unit 184. These functional components are realized by executing a program that causes a computer mounted in the management device 1 to function. The process flow of these functional components can also be understood as a display method.

[0105] 17 described above, the answer collection unit 13 transmits questions, including questions about the future and questions about the present, to the user terminal 4 as a questionnaire for the user. In response, the answer collection unit 13 receives a text of a qualitative answer to the question from the user terminal 4. The received text is output to the positive / negative classification unit 181.

[0106] The positive / negative classification unit 181 classifies text into positive or negative using the sentiment analysis model 5. Specifically, the above-mentioned TextBlob or NTLK can be used.

[0107] The noun extraction unit 182 uses the morphological analysis model 6 to extract nouns from the sentence.

[0108] FIG. 19 is an explanatory diagram of the positive / negative counting section and the noun identification display section.

[0109] Positive / negative counting unit 183 counts, for each noun, the number of positives for which sentences including the noun are classified as positives and the number of negatives for which sentences including the noun are classified as negatives.

[0110] The noun identification display unit 184 visually emphasizes and displays, among a plurality of nouns, nouns that have a large number of negatives relative to the number of positives that meet or exceed a predetermined condition. The predetermined condition may be, for example, a number in which the number of negatives is large relative to the number of positives, or a number in which the number of negatives is greater than or equal to a predetermined ratio relative to the number of positives.

[0111] 19, a plurality of nouns are displayed in association with each other on the manager terminal 2 by the noun identification display unit 184, and nouns with a negative number equal to or greater than a predetermined condition are highlighted. The manager can consider measures for improvement by visually checking the highlighted nouns together with the evaluation index for each evaluation item.

[0112] As described in detail above, according to the program, device, and method of the present invention, firstly, it is possible to obtain evaluable expectations and satisfaction levels by asking a variety of questions to users who are the subject of a survey. Secondly, an appropriate evaluation index can be obtained from the user's answer value for each evaluation item. Thirdly, an emotional evaluation index can be obtained from the sentences of the qualitative answers from users. Furthermore, fourthly, nouns that have strong emotional implications can be identified from the sentences in the qualitative responses from users.

[0113] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0114] 1 Management device 101 User Attribute Database 102 Answer Database 103 Statistical Database 11 Question Creation Department 12. Response Collection Department 13 Qualitative answer estimation section 14 Target user extraction section 15 Item-specific answer calculation section 16 Evaluation comment acquisition section 17 Evaluation Index Acquisition Department 2. Administrator terminal 3. Generative AI Server 4. User terminal 5. Sentiment Analysis Model 6 Morphological Analysis Model

Claims

1. A program for causing a computer to function so as to collect a questionnaire from a user based on an evaluation item, The evaluation items are questionnaire items for multiple users belonging to an organization or a service, a question creation means for sending a prompt for instructing the creation of a question about the future and a question about the present for each evaluation item to a generation AI (Artificial Intelligence) server, and receiving the question about the future and the question about the present from the generation AI server; an answer collection means for transmitting questions about the future and questions about the present to each user terminal, receiving answers to each question from the terminal, and storing the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels, respectively, for each user, in an answer database; and make it work. The question creation means includes attribute information of the organization or service in the prompt to create a question based on the organization or service. A program that causes a computer to function in such a manner.

2. The question creating means includes, in the prompt, either or a combination of user attribute information and / or past answer information of the user in order to create each question for each user.

2. The program according to claim 1, which causes a computer to function as described above.

3. The question generator sends a prompt to the generating AI server requesting that the question to be generated result in a quantitative answer.

2. The program according to claim 1, which causes a computer to function as described above.

4. The method for creating questions is sending prompts to the generation AI server instructing it to create a forward question asking about the future, a paradoxical question asking about the future, a forward question asking about the present, and a paradoxical question asking about the present; receiving from the generation AI server a forward question asking about the future, a paradoxical question asking about the future, a forward question asking about the present, and a paradoxical question asking about the present; The means of collecting responses were: Quantitative answers to forward questions about the future were taken as positive expectations, and quantitative answers to reverse questions about the future were taken as negative expectations. Quantitative answers to forward questions about the present are treated as positive satisfaction, and quantitative answers to reverse questions about the present are treated as negative satisfaction, and are stored in the answer database.

4. The program according to claim 3, which causes a computer to function as described above.

5. The question creation means sends a prompt to the generation AI server requesting that the question to be created be a qualitative answer; The answer collection means receives, from each user terminal, a qualitative answer to a question asking about the future and a qualitative answer to a question asking about the present; a qualitative answer estimation means for transmitting to a generation AI server a prompt for a quantitative answer to the question asking about the future and a quantitative answer to the question asking about the present from the qualitative answer to the question asking about the future and the qualitative answer to the question asking about the present, and receiving from the generation AI server the quantitative answer to the question asking about the future and the quantitative answer to the question asking about the present; 2. The program according to claim 1, further comprising:

6. Qualitative answers are user statements, The qualitative answer estimation means transmits prompts to the generation AI server for a quantitative answer to the question about the future and a quantitative answer to the question about the present based on a sentiment analysis of the sentence.

6. The program according to claim 5, which causes a computer to function as described above.

7. an evaluation comment acquisition means for transmitting a prompt to the generation AI server for each evaluation item or each user, the prompt instructing the creation of an evaluation comment based on the expectation and satisfaction level in the response database, and further receiving an evaluation comment for the evaluation item or the user from the generation AI server, and displaying the evaluation comment on an administrator terminal; 2. The program according to claim 1, further comprising:

8. an evaluation index acquisition means for transmitting a prompt to the generation AI server for each evaluation item or each user, instructing the derivation of an evaluation index from the expectation and satisfaction level in the response database, receiving an evaluation index for the evaluation item or the user from the generation AI server, and displaying the evaluation index on an administrator terminal; 2. The program according to claim 1, further comprising:

9. A management device that collects a questionnaire from a user based on an evaluation item, The evaluation items are questionnaire items for multiple users belonging to an organization or a service, a question creation means for sending a prompt for instructing the creation of a question about the future and a question about the present for each evaluation item to a generation AI (Artificial Intelligence) server, and receiving the question about the future and the question about the present from the generation AI server; an answer collection means for transmitting questions about the future and questions about the present to each user terminal, receiving answers to each question from the terminal, and storing the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels, respectively, for each user, in an answer database; having The question creation means includes attribute information of the organization or service in the prompt to create a question based on the organization or service. A management device comprising:

10. 1. A method for managing a device that collects a questionnaire from a user based on an evaluation item, comprising: The evaluation items are questionnaire items for multiple users belonging to an organization or a service, The device is A first step of sending a prompt to a generation AI (Artificial Intelligence) server for each evaluation item, the prompt instructing the generation of a question about the future and a question about the present, and receiving the question about the future and the question about the present from the generation AI server; a second step of transmitting questions about the future and questions about the present to each user terminal, receiving answers to each question from the terminal, and storing the answers to the questions about the future and the answers to the questions about the present as expectations and satisfaction levels, respectively, for each user, in an answer database; Run The first step is to include attribute information of the organization or service in the prompt to create a question based on the organization or service. A management method comprising:

Images