Determination device, determination method, and computer program

Abstract

To improve quality of a questionnaire with less labor.SOLUTION: A determination device comprises a control unit which, with respect to a questionnaire which is used for acquiring answers from a person being an examination object and includes one or more questions, determines the number of points of question and answer information including the question(s) and an answer form of the question(s) and included in the questionnaire, on the basis of a determination model for determining the number of points indicating a level of evaluation of the question and answer information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a determination device, a determination method, and a computer program. [Background technology]

[0002] When conducting a survey, a questionnaire is used. The higher the quality of the questionnaire, the more appropriate responses can be obtained, and the more effective the survey will be. Therefore, there is a demand for improving the quality of questionnaires used in surveys. For example, in companies that specialize in surveys, experienced staff visually check each questionnaire and correct the contents. Note that the work performed using questionnaires does not have to be limited to surveys, and questionnaires may also be used in surveys with a defined target.

[0003] On the other hand, there is also a demand for reducing the labor required to conduct a survey. For example, Patent Document 1 discloses a technology that uses a computer to detect inappropriate responses in a survey with high accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-28997 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in view of the above circumstances, and provides a technique that makes it possible to improve the quality of questionnaires with less effort. [Means for solving the problem]

[0006] One aspect of the present invention is a judgment device that includes a control unit that judges the score of question answer information included in a questionnaire that is used to obtain answers from a person being surveyed and includes one or more questions, based on a judgment model for determining a score that indicates the level of evaluation of the question answer information including the question and its answer format.

[0007] One aspect of the present invention is the above-mentioned judgment device, wherein the judgment model is a trained model obtained by executing a learning process using training data indicating specific examples of question answer information and evaluation scores for the specific examples.

[0008] In one aspect of the present invention, in the above-mentioned determination device, the control unit determines the score for each item indicated from a plurality of perspectives.

[0009] In one aspect of the present invention, in the determination device, the control unit further determines a response time required to answer the survey form.

[0010] One aspect of the present invention is the above-mentioned judgment device, wherein the judgment model is a trained model obtained by executing a learning process using training data indicating specific examples of question-answer attribute information indicating the attributes of questions and answer options in a questionnaire, and statistical values ​​of the response times required to answer the specific examples.

[0011] In one aspect of the present invention, in the determination device, the control unit further acquires information indicating corrections to the question and answer information included in the survey form.

[0012] One aspect of the present invention is the above-mentioned judgment device, wherein the control unit further acquires information indicating the correction content generated based on information including a specific example of a questionnaire and a specific example of correction content for the specific example.

[0013] One aspect of the present invention is a determination method that includes a determination step of determining a score for question answer information included in a questionnaire that is used to obtain answers from a person being surveyed and includes one or more questions, based on a determination model for determining a score that indicates the level of evaluation of the question answer information including the question and its answer format.

[0014] One aspect of the present invention is a computer program for causing a computer to function as a judgment device that includes a control unit that judges the score of question answer information included in a questionnaire that is used to obtain answers from survey subjects and includes one or more questions, based on a judgment model for determining a score that indicates the level of evaluation of the question answer information, including the question and its answer format. [Effects of the Invention]

[0015] The present invention makes it possible to improve the quality of questionnaires with less effort. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic block diagram showing a system configuration of a first embodiment of a determination system 100 of the present invention. [Figure 2] 2 is a schematic block diagram showing a specific example of the functional configuration of the terminal device 10. FIG. [Figure 3] 2 is a schematic block diagram showing a specific example of the functional configuration of the learning device 20. FIG. [Figure 4] FIG. 10 is a diagram showing a specific example of second teacher data. [Figure 5] 10 is a flowchart showing a specific example of processing by the learning device 20. [Figure 6] 2 is a schematic block diagram showing a specific example of the functional configuration of a determination device 30. FIG. [Figure 7] 10 is a flowchart showing a specific example of processing by the determination device 30. [Figure 8] FIG. 10 is a schematic block diagram showing the system configuration of a second embodiment of a determination system 100 according to the present invention. [Figure 9]FIG. 2 is a diagram illustrating an outline of an example of the hardware configuration of an information processing device 90 applied to the present embodiment. [Figure 10] FIG. 10 is a diagram showing a modified example of the determination device 30 configured in this manner. [Figure 11] FIG. 10 is a diagram showing a modified example of the determination device 30 configured in this manner. DETAILED DESCRIPTION OF THE INVENTION

[0017] [First embodiment] 1 is a schematic block diagram showing the system configuration of a first embodiment of a determination system 100 of the present invention. The determination system 100 is used to determine the quality of a questionnaire and corrections based on information about the questionnaire to be determined (hereinafter referred to as "questionnaire information").

[0018] The determination system 100 includes a terminal device 10, a learning device 20, and a determination device 30. The terminal device 10 and the determination device 30 are communicatively connected via a network 70. The learning device 20 and the determination device 30 may also be communicatively connected via the network 70. The network 70 may be a network using wireless communication or a network using wired communication. The network 70 may be configured using, for example, the Internet or a local area network (LAN). The network 70 may also be configured by combining multiple networks.

[0019] 2 is a schematic block diagram showing a specific example of the functional configuration of the terminal device 10. The terminal device 10 is configured using information equipment such as a smartphone, tablet, personal computer, dedicated device, etc. The terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, a data input unit 14, a storage unit 15, and a control unit 16.

[0020] The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 communicates data with other devices via the network 70 in accordance with the control of the control unit 16. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

[0021] The input unit 12 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), buttons, a touch panel, etc. The input unit 12 is operated by a user when inputting user instructions to the terminal device 10. The input unit 12 may be an interface for connecting the input device to the terminal device 10. In this case, the input unit 12 inputs an input signal generated in the input device in response to a user input to the terminal device 10. The input unit 12 may be configured using a microphone and a voice recognition device. In this case, the input unit 12 acquires an acoustic signal generated by the user's speech, performs voice recognition on the words spoken by the user, and inputs character string information of the recognition result to the terminal device 10. The voice recognition process may be performed by the control unit 16. The input unit 12 may be configured in any way as long as it is capable of inputting user instructions to the terminal device 10.

[0022] The output unit 13 outputs information in a form that can be recognized by the user. The output unit 13 may be, for example, an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The output unit 13 may be an interface for connecting an image display device to the terminal device 10. In this case, the output unit 13 generates a video signal for displaying image data and outputs the video signal to the image display device connected to the output unit 13. The output unit 13 may be a device for outputting sound, such as a speaker. The output unit 13 may be an interface for connecting an audio output device, such as a speaker or headphones, to the terminal device 10. In this case, the output unit 13 generates an audio signal for reproducing audio data and outputs the audio signal to the audio output device connected to the output unit 13. The output unit 13 may be configured as a touch panel integrated with the input unit 12.

[0023] The data input unit 14 accepts data input to the terminal device 10. The data input unit 14 particularly accepts input of survey form data to be judged. The survey form data may be data in any format as long as it is capable of representing the survey form. The survey form data may be image data showing an image of the survey form, document data including the character string of the survey form, or may be constructed in other ways. However, if the data is image data, it is desirable that character recognition processing be performed in the terminal device 10 or the judgment device 30 to obtain data indicating the character string included in the survey form.

[0024] The data input unit 14 may read out survey form data recorded on a recording medium such as a CD-ROM or a USB memory (Universal Serial Bus Memory). The data input unit 14 may receive survey form data from another information processing device via a network. The data input unit 14 may be configured in a different manner as long as it is capable of receiving survey form data. The survey form data input by the data input unit 14 may be stored in the memory unit 15. The survey form data may be generated in the terminal device 10 by operating the input unit 12. In this case, the terminal device 10 does not necessarily have to include the data input unit 14.

[0025] The memory unit 15 is configured using a memory device such as a magnetic hard disk drive or a semiconductor memory device. The memory unit 15 stores data used by the control unit 16. The memory unit 15 stores data required when the control unit 16 performs processing. The memory unit 15 stores, for example, data on the survey form to be judged.

[0026] The control unit 16 is configured using a processor such as a CPU (Central Processing Unit) and a memory (main storage device). The control unit 16 functions when the processor executes a program. Note that all or part of the functions of the control unit 16 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs: Solid State Drives), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The program may be transmitted via a telecommunications line.

[0027] The control unit 16 may execute, for example, an application installed on its own device (terminal device 10). A specific example of such an application is an application provided to the terminal device 10 as a dedicated application for the determination system 100. Another specific example of such an application is a web browser application. Such an application may be pre-installed on the terminal device 10, or may be downloaded each time a determination process is executed. For example, when implemented as a web browser application, the terminal device 10 may download and execute the application from a device specified by a specific web server (for example, the web server itself or another server) in response to the terminal device 10 connecting to the web server. The control unit 16 operates according to the program of the application being executed.

[0028] The control unit 16 controls the terminal device 10 in response to user operations and information received from the determination device 30. For example, the control unit 16 transmits information input by a target person or a user operating the input unit 12 to the determination device 30 by using the communication unit 11. For example, when the communication unit 11 receives information transmitted from the determination device 30 via the network 70, the control unit 16 generates screen data based on the received information and causes the output unit 13 to display the screen data. Such screen data includes images and text indicating the information transmitted from the determination device 30. For example, when the communication unit 11 receives information transmitted from the determination device 30 via the network 70, the control unit 16 generates voice data based on the received information and causes the output unit 13 to output the voice data.

[0029] 3 is a schematic block diagram showing a specific example of the functional configuration of learning device 20. Learning device 20 is configured using an information processing device such as a personal computer or a server device. Learning device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0030] The communication unit 21 is a communication device. The communication unit 21 may be configured as, for example, a network interface. The communication unit 21 communicates data with other devices via the network 70 in accordance with the control of the control unit 23. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

[0031] The storage unit 22 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 22 stores data used by the control unit 23. The storage unit 22 may function as, for example, a teacher data storage unit 221 and a trained model storage unit 222.

[0032] The teacher data storage unit 221 stores teacher data used in the learning process executed in the learning device 20. The teacher data stored in the teacher data storage unit 221 includes first teacher data and second teacher data.

[0033] The first teacher data includes question and answer information indicating the questions and answer options included in the questionnaire, and scores assigned to each piece of question and answer information. The question and answer information corresponds to an explanatory variable, and the scores correspond to a target variable. Scores may be assigned for multiple items. For example, scores may be assigned for each item from each perspective, including conciseness, understandability of Japanese, and logicality. For example, scores may be assigned for each item from five perspectives (conciseness, objectivity / neutrality, concreteness, accuracy, and politeness). The scores indicate a score evaluating the quality of the question and answer information using an index indicated by each item. For example, a higher score indicates that the quality of the question and answer information in that item can be evaluated higher. Note that if the answering method does not use options (e.g., free text), the question and answer information may be represented by information related to the question only.

[0034] The second teacher data includes question and answer attribute information that indicates the attributes of the questions and answer options included in the questionnaire, and the response time required to answer the questionnaire. The question and answer attribute information corresponds to the explanatory variable, and the response time corresponds to the objective variable. Figure 4 is a diagram showing a specific example of second teacher data. The second teacher data includes a questionnaire ID, response time, and question and answer attribute information. The questionnaire ID is identification information for each questionnaire used as second teacher data. The response time indicates the actual time required to answer each questionnaire. If the response time is obtained by multiple people, a statistical value (e.g., average, mode, median, etc.) may be used.

[0035] There is a correlation between question and answer attribute information and answer time. For example, the more questions there are, the longer the answering time required. Also, even if the number of questions is the same, questions with more option options require longer answering time. Also, even if the number of option options is the same, questions that require you to select all that apply require longer answering time than questions where you only need to select one. Furthermore, open-ended questions generally require longer answering time than questions with multiple options. For these reasons, it can be said that there is the above-mentioned correlation.

[0036] Question and answer attribute information indicates the number of questions and answers for each attribute included in the questionnaire. The question and answer attributes are called question and answer attributes. A question and answer attribute of single (2 categories) indicates that the question is in the format of selecting one answer from two options. A questionnaire with a single (2 categories) value of "m" indicates that it contains m questions (m is an integer greater than or equal to 0) in the format of single (2 categories). A question and answer attribute of single (n categories) indicates that the question is in the format of selecting one answer from n options (n ​​is an integer greater than or equal to 0).

[0037] The question-answer attribute Multiple (2 categories, all) indicates that the question is in the format of selecting all applicable answers from two options. A questionnaire with a Multiple (2 categories, all) value of "m" indicates that it contains m questions in the format of Single (2 categories). The question-answer attribute Multiple (n categories, all) indicates that the question is in the format of selecting all answers from n options.

[0038] The question-answering attribute Ranking (2 categories) indicates that the question is in a format where two options are arranged in order according to the question content and then answered. A questionnaire with a Ranking (2 categories) value of "m" indicates that it contains m questions in the Ranking (2 categories) format. The question-answering attribute Ranking (n categories) indicates that the question is in a format where n options are arranged in order according to the question content and then answered.

[0039] The question-answer attribute "free text" indicates that the question is in a format where the respondent can write their own answer without being given options. A questionnaire with a free text value of "m" indicates that it contains m free text questions.

[0040] The trained model memory unit 222 stores trained models obtained by a learning process using the teacher data stored in the teacher data memory unit 221. The trained model memory unit 222 stores, for example, a first trained model and a second trained model. The first trained model is a trained model obtained by a learning process using the first teacher data. The first trained model is a trained model that, when given question answer information on a questionnaire as an explanatory variable, outputs the score as the objective variable. The second trained model is a trained model obtained by a learning process using the second teacher data. The second trained model is a trained model that, when given question answer attribute information on a questionnaire as an explanatory variable, outputs the response time required to answer the questionnaire as the objective variable.

[0041] The control unit 23 is configured using a processor such as a CPU and a memory. The control unit 23 functions as an information control unit 231 and a learning control unit 232 by the processor executing a program. All or part of the functions of the control unit 23 may be realized using hardware such as an ASIC, PLD, or FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0042] The information control unit 231 controls the input and output of information. For example, the information control unit 231 acquires training data from other devices (information processing devices or storage media) and records the training data in the training data storage unit 221. For example, the information control unit 231 transmits the trained model stored in the trained model storage unit 222 to another device (for example, the determination device 30).

[0043] The learning control unit 232 executes a learning process using the training data stored in the training data storage unit 221. Specific examples of such learning processes include supervised learning for classification, such as support vector machines, random forests, and neural networks. The learning control unit 232 generates, for example, a first trained model for outputting a score for input answer to question information, based on the input answer to question information, by performing supervised learning. The learning control unit 232 generates, for example, a second trained model for outputting an estimated value of the time required to answer a questionnaire for the input answer to question attribute information, based on the input answer to question attribute information, by performing supervised learning.

[0044] The learning control unit 232 records the generated trained model in the trained model storage unit 222. The trained model obtained by the learning control unit 232 may be transmitted to the determination device 30 and recorded in the determination model storage unit 321 of the determination device 30.

[0045] 5 is a flowchart showing a specific example of processing by the learning device 20. First, the information control unit 231 acquires training data (step S101). The training data may be input by a user, acquired via communication from another information device, or acquired from a recording medium connected to the learning device 20, for example. The learning control unit 232 executes a learning process using the training data and records a trained model in the trained model storage unit 222 (step S102).

[0046] 6 is a schematic block diagram showing a specific example of the functional configuration of the determination device 30. The determination device 30 is configured using an information processing device such as a personal computer or a server device. The determination device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0047] The communication unit 31 is a communication device. The communication unit 31 may be configured as, for example, a network interface. The communication unit 31 communicates data with other devices via the network 70 in accordance with the control of the control unit 33. The communication unit 31 may be a device that performs wireless communication or a device that performs wired communication.

[0048] The storage unit 32 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 32 stores data used by the control unit 33. The storage unit 32 may function as a determination model storage unit 321, for example.

[0049] The determination model storage unit 321 stores a determination model used by the determination unit 333 when performing the determination process. The determination model may be configured using information of a trained model generated in advance by a learning process, for example. Such a learning process may be executed by another device (for example, the learning device 20) or by the device itself (the determination device 30). The determination model does not necessarily have to be generated by a learning process. The determination model may be configured using, for example, a lookup table that associates question and answer attribute information with answer times, or may be configured in another manner.

[0050] The control unit 33 is configured using a processor such as a CPU and a memory. The control unit 33 functions as an information control unit 331, a preprocessing control unit 332, and a determination unit 333 by the processor executing a program. All or part of the functions of the control unit 33 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0051] The information control unit 331 acquires the survey form information of the object to be judged from another device such as the terminal device 10. The acquired survey form information includes the data of the object to be judged itself. The information control unit 331 transmits information indicating the judgment result obtained by the judgment unit 333 to the other device such as the terminal device 10. Such exchange of information between the information control unit 331 and the other device may be performed by communication using the communication unit 31, for example.

[0052] The preprocessing control unit 332 acquires question and answer attribute information by performing predetermined preprocessing on the survey form acquired by the information control unit 331. That is, the preprocessing control unit 332 acquires the number of questions and answers for each attribute included in the survey form to be determined. Note that the question and answer attribute information of the survey form to be determined may not be acquired by preprocessing by the preprocessing control unit 332, but may be acquired in advance by another device or manually and input into the determination device 30 as part of the survey form information. In this case, the determination process by the preprocessing control unit 332 is unnecessary.

[0053] The determination unit 333 performs a determination process using the determination model stored in the determination model storage unit 321 and the questionnaire information to be determined. The determination unit 333 determines the score of each piece of question and answer information included in the questionnaire to be determined, for example, by a determination process using the first trained model. After obtaining scores for all of the multiple pieces of question and answer information included in one questionnaire, the determination unit 333 may further obtain statistical values ​​of the scores. For example, the total score, average score, mode, median, etc. for each item may be obtained. Furthermore, the total score, average score, mode, median, etc. for all items may be obtained. The determination unit 333 may output the obtained scores and their statistical values ​​as the evaluation result of the questionnaire to be determined.

[0054] The judgment unit 333 may obtain an estimated value of the response information required to answer the questionnaire to be judged, for example, by a judgment process using the second trained model.

[0055] 7 is a flowchart showing a specific example of the processing of the determination device 30. First, the information control unit 331 acquires survey form information of the survey form to be determined from the terminal device 10 (step S201). The preprocessing control unit 332 acquires question and answer attribute information by performing preprocessing on the survey form information to be determined (step S202). The determination unit 333 performs a determination process using at least the first trained model or the second trained model (step S203). The determination unit 333 transmits information indicating the determination result to the terminal device 10 (step S204).

[0056] The assessment system 100 configured in this manner makes it possible to improve the quality of questionnaires with less effort. Specifically, it is as follows. In the assessment system 100, the assessment device 30 obtains estimated scores and response times for the questionnaire to be assessed. This makes it possible to reduce the time required to evaluate the questionnaire to be assessed and estimate the response time. Then, by reviewing and correcting the contents of the questionnaire to be assessed based on the obtained estimated scores and response times, it becomes possible to easily improve the quality.

[0057] In particular, when scores are obtained for each item, it is possible to see which items have high scores and which items have low scores, making it easier to improve quality. For example, if the score for conciseness is low, reviewing the question and answer information from the perspective of improving conciseness will allow for more appropriate corrections and quality improvement than if the score were simply low. Furthermore, if the estimated response time is long, there is a risk that the rate of careless respondents will increase, so by modifying the questionnaire with a view to shortening the response time, appropriate corrections can be made. Note that a careless respondent is someone who gives an inappropriate answer, and an inappropriate answer is one given without carefully reading the contents of the questionnaire.

[0058] [Second embodiment] 8 is a schematic block diagram showing the system configuration of a second embodiment of the determination system 100 of the present invention. The same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted. The second embodiment will be described below, focusing on the components that are different from those in the first embodiment.

[0059] The determination system 100 in the second embodiment further includes a natural language generation device 40. In the second embodiment, the quality of the questionnaire and correction items are determined by processing of the determination device 30 and the natural language generation device 40 based on the questionnaire information to be determined.

[0060] The natural sentence generation device 40 is configured using a communication-enabled information processing device. The natural sentence generation device 40 is constructed using, for example, a Large Language Model (LLM). In response to input character string data input from another device (for example, the determination device 30), the natural sentence generation device 40 generates and outputs a natural sentence indicating a response to the natural sentence indicated by the input character string data.

[0061] The determination unit 333 in the second embodiment transmits natural sentences indicating an instruction to evaluate the questionnaire information to be determined to the natural sentence generation device 40. The determination unit 333 may generate and transmit a prompt indicating that the evaluation will be made, for example, based on the questionnaire information to be determined. The natural sentence generation device 40 generates natural sentences indicating an evaluation of the questionnaire information to be determined in response to the input instruction and transmits the natural sentences to the determination device 30. The determination unit 333 transmits the natural sentences obtained from the natural sentence generation device 40 to the terminal device 10. Note that the determination unit 333 may transmit the natural sentences obtained from the natural sentence generation device 40 to the terminal device 10 as they are, or may process the obtained natural sentences before transmitting them to the terminal device 10.

[0062] A specific example of the processing of the determination device 30 and the natural language generation device 40 will be described below. The determination unit 333 may generate a prompt indicating that the survey information to be evaluated will be evaluated without using the first trained model and the second trained model. In this case, the prompt may be generated by using, for example, a predetermined standard prompt phrase (template). The determination unit 333 may generate a prompt including the result of a determination process using the first trained model stored in the determination model storage unit 321. The determination unit 333 may generate a prompt including the result of a determination process using the second trained model stored in the determination model storage unit 321. The determination unit 333 may generate a prompt including both the result of a determination process using the first trained model stored in the determination model storage unit 321 and the result of a determination process using the second trained model. By generating such a prompt, a natural sentence including a determination result using the first trained model and the second trained model is obtained. Furthermore, by generating such a prompt, the natural sentence generation device 40 can further perform an evaluation based on the above-mentioned determination result and obtain the result. For example, a prompt may be generated to generate an evaluation result according to the distribution of scores for each item on the questionnaire to be evaluated, and a natural sentence indicating such an evaluation result may be obtained.

[0063] The natural language generation device 40 may generate natural language (hereinafter referred to as "evaluation result information") indicating the evaluation result using information related to the evaluation criteria of the questionnaire from the determination device 30 (hereinafter referred to as "evaluation criteria information"), for example, and transmit the evaluation result information to the determination device 30. The method of providing the evaluation criteria information to the natural language generation device 40 may be realized in any manner. For example, the evaluation criteria information may be provided by performing an additional learning process in advance using the evaluation criteria information in a large-scale language model of the natural language generation device 40. Such a method may be realized, for example, by performing fine tuning.

[0064] Alternatively, the evaluation criterion information may be provided by storing the evaluation criterion information in advance in the determination device 30 or another information processing device, and providing the result of information processing (e.g., information retrieval) based on the evaluation criterion information as an input to the large-scale language model. Such a method may be realized, for example, by using RAG (Retrieval Augmented Generation). Alternatively, the evaluation criterion information may be provided by transmitting the evaluation criterion information from the terminal device 10 every time the terminal device 10 requests an evaluation from the determination device 30, and providing the result of information processing (e.g., information retrieval) based on the evaluation criterion information by the determination device 30 as an input to the large-scale language model.

[0065] When performing processing using evaluation criteria information, the natural language generation device 40 may operate as follows. The natural language generation device 40 evaluates the survey form information to be evaluated according to the criteria indicated by the evaluation criteria. The natural language generation device 40 then generates natural language indicating the evaluation result and transmits the natural language to the determination device 30.

[0066] The evaluation criterion information may include, for example, a combination of a specific example of inappropriate answering information before correction and a specific example of answering information that has been corrected from the inappropriate state to an appropriate state. The evaluation criterion information may be configured to include the reason for such correction in addition to the combination of the two specific examples. By using such evaluation criterion information, when the questionnaire to be evaluated contains inappropriate answering information, the natural language generation device 40 can indicate how the answering information should be corrected, and can also provide a reason for the correction.

[0067] The evaluation criteria information may be configured as information that generates natural sentences that show specific examples of answering questions that have a high cognitive load and suggest corrections to answering questions that have a high cognitive load. By using such evaluation criteria information, it becomes possible to detect answering questions that have a high cognitive load in the questionnaire to be evaluated and suggest corrections to the answering questions.

[0068] The evaluation criteria information may be, for example, information listing rules that must be observed when creating answer options for Likert items. Specifically, the evaluation criteria information may be defined as information listing the following items:

[0069] A. Regarding the creation of Likert scale items [question items] item 1. Items that present specific amendments and the reasons for the amendments 1.1. Do not use katakana words, technical terms, or difficult words 1.2. Standardize your terminology 1.3. Avoid double negatives 1.4. Do not use uncommon kanji characters 1.5. Be careful of typos and spelling errors 1.6. Pay attention to the consistency between the question and the answer options 1.7. Pay attention to the consistency between the question and the items on the left 1.8. Write questions that fit the answer format 2. Items that point out what needs to be corrected and why 2.1. Avoid the overuse of ambiguous demonstrative pronouns 2.2. Sorting Similar Words 2.3. Be careful about the amount of kanji you use 2.4. Beware of Sensitive Content

[0070] B. Regarding the creation of Likert scale items [left items] item 1. Items that present specific amendments and the reasons for the amendments 1.1. Ensure completeness 1.2. Avoid parallel-point options 1.3. Avoid issue-dependent options 1.4. Standardize sentence endings 2. Items that point out what needs to be corrected and why 2.1. Aligning granularity

[0071] C. Creating Likert scale items [answer options] item 1. Items that present specific amendments and the reasons for the amendments 1.1. Ensure completeness 1.2. Avoid parallel-point options 1.3. Avoid issue-dependent options 1.4. Standardize sentence endings 2. Items that point out what needs to be corrected and why 2.1. Ensuring mutual exclusivity 2.2. Aligning granularity

[0072] D. Regarding the creation of Likert scale questions item 1. Items that present specific amendments and the reasons for the amendments 1.1. Avoid impolite expressions 1.2. Avoid ambiguous phrases 1.3. Focus your questions 1.4. Avoid sentences without a subject 1.5. Make the subject of the sentence specific 2. Items that point out what needs to be corrected and why 2.1. Avoid questions with high cognitive load 2.2. Beware of questions that are prone to recall bias 2.3. Highlight your answer 2.4. Emphasize answer instructions

[0073] Regarding the creation of questions for the E.SD method item 1. Items that present specific amendments and the reasons for the amendments 1.1. Matching left and right items

[0074] F. Other Questions item 1. Items that present specific amendments and the reasons for the amendments 1.1. Do not use katakana words, technical terms, or difficult words 1.2. Standardize your terminology 1.3. Avoid double negatives 1.4. Do not use uncommon kanji characters 1.5. Pay attention to the consistency between the question and the answer options 1.6. Be careful of typos and omissions 2. Items that point out what needs to be corrected and why 2.1. Avoid the overuse of ambiguous demonstrative pronouns 2.2. Sorting Similar Words 2.3. Be careful about the amount of kanji you use 2.4. Beware of Sensitive Content 2.5. Standardize the endings of questions and answer options (Why is the position different from C?)

[0075] G. Other items [Answer options] Creation item 1. Items that present specific amendments and the reasons for the amendments 1.1. Ensure mutual exclusivity 1.2. Align granularity 2. Items that point out what needs to be corrected and why 2.1. Ensure completeness 2.2. Avoid parallel-point options 2.3. Avoid issue-dependent options 2.4. Standardize sentence endings

[0076] H. Regarding the creation of [question sentences] for other items item 1. Items that present specific amendments and the reasons for the amendments 1.1. Avoid impolite expressions 1.2. Avoid ambiguous phrases 1.3. Don't ask parallel questions 1.4. Don't ask topic-driven questions 2. Items that point out what needs to be corrected and why 2.1. Focus your questions 2.2. Avoid questions with high cognitive load 2.3. Beware of questions that are prone to recall bias 2.4. Highlight your answer 2.5. Emphasize answer instructions 2.6. Avoid ambiguous questions

[0077] Furthermore, the evaluation criteria information may be defined as information that lists the following items: 1: Question: Does the question begin with "you"? 2: Question: Is there a double barrel question? 3: Question: Does it contain a sentence with a double negative? 4: Question: Does it contain socially desirable bias? 5: Question: Determine whether the question is cognitively demanding 6: Questions: Determining the level of politeness 7: Question: Does it contain an ambiguous demonstrative pronoun? 8: Question: Is it a memory question? 9: Questions: Suggest points to emphasize 10: Question text: Determine whether the question is sensitive, such as about crime or sexual behavior. 11: Answer options: Whether or not there is an option that the respondent would like to choose 12: Answer options: Whether the meaning of the option overlaps with other options 13: Answer options: Are the granularity (hierarchy) consistent so that the options you want to choose do not overlap? 14: Answer options: Whether or not it is a double barrel 15: Answer options: Are the endings of sentences consistent? 16: Entire questionnaire: Are there any katakana words, technical terms, or difficult terms that are difficult to understand? 17: Entire questionnaire: Are words with the same meaning consistent? 18: Overall questionnaire: Are there too many kanji characters? 19: Entire questionnaire: Whether or not uncommon kanji characters are used 20: Question: Is the wording appropriate from five perspectives (succinctness, objectivity / neutrality, specificity, accuracy, and politeness)? 21:Title: Whether or not it is possible to improve the motivation / enthusiasm of respondents

[0078] FIG. 9 is a diagram illustrating an example of the hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 includes a processor 91, a main memory device 92, a communication interface 93, an auxiliary memory device 94, an input / output interface 95, and an internal bus 96. The processor 91, the main memory device 92, the communication interface 93, the auxiliary memory device 94, and the input / output interface 95 are communicably connected to each other via the internal bus 96. The information processing device 90 may be applied to, for example, the learning device 20 and the determination device 30. In this case, for example, the communication unit 21 and the communication unit 31 may be configured using the communication interface 93. For example, the memory unit 22 and the memory unit 32 may be configured using the auxiliary memory device 94. Furthermore, the control unit 23 and the control unit 33 may be configured using the processor 91 and the main memory device 92. The information processing device 90 may also be applied to the natural language generation device 40 in the second embodiment.

[0079] (Variation) In this embodiment, the terminal device 10 and the determination device 30 are configured as separate devices, but they may also be configured as an integrated device. FIG. 10 is a diagram showing a modified example of the determination device 30 configured in this manner. The determination device 30 shown in FIG. 10 includes an input unit 34, an output unit 35, and a data input unit 36. The input unit 34, the output unit 35, and the data input unit 36 ​​of the determination device 30 shown in FIG. 10 function similarly to the input unit 12, the output unit 13, and the data input unit 14 of the terminal device 10, respectively. The control unit 33 operates, for example, in response to an operation on the input unit 34, performs a determination process using information input via the data input unit 36, and outputs the information using the output unit 35. The determination device 30 configured in this manner may be operated by a user. That is, in the first and second embodiments, the determination process is performed by a device (the determination device 30) different from the terminal device 10 operated by the user, but the determination process may also be performed by a device operated by the user. In this configuration, the determination device 30 operated by the user may operate as in the second embodiment by communicating with the natural language generation device 40.

[0080] In this embodiment, the learning device 20 and the determination device 30 are configured as separate devices, but they may also be configured as an integrated device. FIG. 11 is a diagram showing a modified example of the determination device 30 configured in this manner. The memory unit 32 of the determination device 30 shown in FIG. 11 also functions as a teacher data memory unit 322. The control unit 33 of the determination device 30 shown in FIG. 11 also functions as a learning control unit 334. The teacher data memory unit 322 functions in the same way as the teacher data memory unit 221 of the learning device 20. The learning control unit 334 functions in the same way as the learning control unit 232 of the learning device 20.

[0081] The learning device 20 may be implemented using a plurality of information processing devices. For example, the learning device 20 may be implemented using a device such as a cloud. For example, in the learning device 20, the memory unit 22 and the control unit 23 may each be implemented in different information processing devices. For example, the memory unit 22 of the learning device 20 may be distributed and implemented across a plurality of information processing devices. The determination device 30 may be implemented using a plurality of information processing devices. For example, the determination device 30 may be implemented using a device such as a cloud. For example, in the determination device 30, the memory unit 32 and the control unit 33 may each be implemented in different information processing devices. For example, the memory unit 32 of the determination device 30 may be distributed and implemented across a plurality of information processing devices.

[0082] In the first and second embodiments described above, the score for each piece of answering information to question is determined by the determination device 30 using the first trained model. In contrast, the score for each piece of answering information to question may be determined by the natural language generation device 40. For example, the natural language generation device 40 may determine whether or not the conventions in the answering information to question included in the evaluation criterion information are kept in each piece of answering information to determine the score based on the number of kept conventions and / or the number of broken conventions. In this case, the natural language generation device 40 may be provided with the evaluation criterion information and an instruction (prompt) to determine the score based on the number of kept conventions and / or the number of broken conventions.

[0083] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0084] 100...Determination system, 10...Terminal device, 11...Communication unit, 12...Input unit, 13...Output unit, 14...Data input unit, 15...Memory unit, 16...Control unit, 20...Learning device, 21...Communication unit, 22...Memory unit, 221...Teacher data memory unit, 222...Trained model memory unit, 23...Control unit, 231...Information control unit, 232...Learning control unit, 30...Determination device, 31...Communication unit, 32...Memory unit, 321...Determination model memory unit, 322...Teacher data memory unit, 33...Control unit, 331...Information control unit, 332...Preprocessing control unit, 333...Determination unit, 34...Input unit, 35...Output unit, 36...Data input unit, 40...Natural sentence generation device

Claims

1. a control unit that determines the score of each piece of question and answer information included in a questionnaire that is used to obtain answers from a person to be surveyed and includes one or more questions, based on a determination model for determining scores that indicate the level of evaluation of the question and answer information including the question and its answer options; the determination model includes a first trained model obtained by executing a learning process using a plurality of training data indicating specific examples of answer-to-question information and evaluation scores of the specific examples; The control unit is a determination device that inputs each piece of question and answer information included in the questionnaire into the first trained model and obtains the score of the input question and answer information as output, thereby determining the score for each piece of question and answer information.

2. The scores of the training data used in the first trained model are assigned to each item represented by a plurality of perspectives, The determination device according to claim 1 , wherein the control unit determines a score for each of the items by acquiring a score for the input question and answer information as an output of the first trained model for each of the items.

3. The control unit further determines the response time required to answer the questionnaire, the determination model includes a second trained model obtained by executing a learning process using multiple pieces of training data indicating specific examples of question and answer attribute information indicating attributes of questions and answer options in a questionnaire and statistical values ​​of response times required to answer the specific examples of the questionnaire, The control unit determines the response time required to answer the questionnaire by inputting each question and answer attribute information included in the questionnaire to the second trained model and obtaining as output a statistical value of the response time for the input question and answer attribute information.

4. A determination system comprising a determination device and a natural language generation device, the determination device includes a control unit that determines, for each piece of question and answer information included in a questionnaire that is used to obtain answers from a person to be surveyed and includes one or more questions, a score for the piece of question and answer information included in the questionnaire based on a determination model for determining a score indicating a high evaluation of the question and answer information including the question and its answer options; the determination model includes a first trained model obtained by executing a learning process using a plurality of training data indicating specific examples of answer-to-question information and evaluation scores of the specific examples; the control unit inputs each piece of question and answer information included in the questionnaire into the first trained model and obtains a score for each piece of input question and answer information as an output, thereby determining a score for each piece of question and answer information; the control unit transmits a natural sentence indicating an instruction to evaluate the questionnaire to be judged and information indicating the questionnaire to be judged to the natural sentence generation device; the natural language generation device includes a control unit that uses a large-scale language model to generate evaluation result information expressed in natural language, including information indicating corrections to the question and answer information included in the questionnaire to be judged, based on evaluation criteria information, which is information related to the evaluation criteria of the questionnaire, and transmits the generated evaluation result information to the judgment device; A judgment system in which the evaluation criteria information includes a combination of specific examples of question answer information in an inappropriate state before correction and specific examples of question answer information that has been corrected from the inappropriate state to an appropriate state.

5. The computer has a determination step for determining the score of each piece of question-answer information included in a questionnaire that is used to obtain answers from a person being surveyed and includes one or more questions, based on a determination model for determining a score indicating the degree of evaluation of the question-answer information including the question and its answer options, the determination model includes a first trained model obtained by executing a learning process using a plurality of training data indicating specific examples of answer-to-question information and evaluation scores of the specific examples; In the determination step, each piece of question and answer information included in the questionnaire is input to the first trained model, and the score of the input question and answer information is obtained as output, thereby determining the score for each piece of question and answer information.

6. a control unit that determines the score of each piece of question and answer information included in a questionnaire that is used to obtain answers from a person to be surveyed and includes one or more questions, based on a determination model for determining scores that indicate the level of evaluation of the question and answer information including the question and its answer options; the determination model includes a first trained model obtained by executing a learning process using a plurality of training data indicating specific examples of answer-to-question information and evaluation scores of the specific examples; The control unit is a computer program that causes a computer to function as a determination device that inputs each piece of question and answer information included in the questionnaire into the first trained model and obtains the score of the input question and answer information as output, thereby determining a score for each piece of question and answer information.

Images