Management device, management system, management program, and management method

Abstract

To provide a management device, management system, management program, and management method that improve the accuracy of analysis of oral responses. [Solution] A management device that outputs questions for analyzing the subject's speech patterns to an information terminal and acquires information related to the subject's voice in response to the questions as response data. The management device inputs prompt data, which includes generation instruction information, into a language model to generate an estimated answer to the question based on the materials and response data, and obtains an estimated answer corresponding to the question. Alternatively, the management device inputs prompt data, which includes generation instruction information, into a language model to rewrite a standard answer corresponding to the question based on the response data, and obtains an estimated answer corresponding to the question. The management device calculates the similarity between the response data relating to the subject's voice response to the question and the estimated answer corresponding to the question.

Description

Technical Field

[0001] The present invention relates to a management device, a management system, a management program, and a management method for managing the understanding status of a target person with respect to predetermined materials.

Background Art

[0002] With the evolution of generative AI, anyone can easily create sophisticated articles. In particular, by using an LLM (Large Language Model), highly accurate and natural articles can be generated depending on the ingenuity of the prompt. Under such circumstances, it is extremely difficult to determine whether the submitted article is the result of the author's own thinking.

[0003] For example, in a company's recruitment activities, the mainstream process is document screening based on a huge amount of historical data (resumes, work experience records, entry sheets, etc.) followed by multiple interviews, which requires a great deal of man-hours and costs. Nevertheless, if the applicant's own thinking is not reflected in the historical data, the significance of document screening is greatly impaired. In addition, there is also a concern that if the thinking process such as article composition and term selection continues to depend on an LLM, logical thinking ability and language expression ability will not be cultivated.

[0004] From such a background, the importance of oral Q&A, which can easily visualize the comprehension ability and thinking process that are difficult to grasp in writing, is increasing. For example, in the certification exams for highly specialized occupations such as engineers and real estate appraisers, in addition to knowledge evaluation by written exams, face-to-face reviews based on reports detailing practical achievements have been introduced. Also, in recent times when knowledge can be immediately supplemented by search etc., the opportunity to recognize one's own knowledge level is decreasing, and oral Q&A is also effective as a means to confirm the understanding status of specific matters in a short time.

[0005] With the recent spread of the internet, online oral interviews and other forms of interviews are being conducted, and the format of answering questions from AI agents is increasing, in addition to face-to-face interviews. Furthermore, with the spread of web interviews and recorded interviews, it is expected that hiring decisions will be made not only through real-time evaluation by interviewers, but also through post-interview review of recorded data. It is also anticipated that the need for oral question-and-answer sessions will expand to include the review of contract documents and the progress of daily work.

[0006] Systems for analyzing oral question-and-answer sessions have been known for some time (see, for example, Patent Documents 1 and 2). The system in Patent Document 1 is configured to have job seekers and recruiting companies conduct online interviews and to decide whether to hire a job seeker based on the frequency of occurrence of predetermined keywords from the interview record information. The system in Patent Document 2 is configured to generate questions based on a resume and to provide feedback on the analysis results of the answers to the generated questions. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2004-288107 [Patent Document 2] Japanese Patent Publication No. 2026-16250 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, conventional systems that uniformly evaluate oral responses are susceptible to variations in individual language use (vocabulary choice, phrasing habits, etc.), which affects the analysis results. This makes it difficult to improve the accuracy of the analysis, and in particular, when applying the system to the analysis of audio responses from multiple participants, it becomes difficult to ensure fairness.

[0009] The present invention was made to solve the above-mentioned problems and aims to provide a management device, management system, management program, and management method that improve the accuracy of analysis of oral responses. [Means for solving the problem]

[0010] A management device according to one aspect of the present invention includes: an output processing means that transmits analysis data to an information terminal to output questions for analyzing the speech patterns of a subject, and outputs the questions to the information terminal in at least one of voice and text; an acquisition processing means that, via the sound pickup unit of the information terminal, acquires information relating to the voice of the subject uttered in response to the questions as response data, and acquires information relating to the voice of the subject's answer to the questions as answer data; and an answer management means that determines the similarity between the answer data corresponding to the question and the estimated answer which is the estimated answer to the question. The response management means takes prompt data, which includes materials, response data, and generation instruction information that causes the language model to generate an estimated answer to a question based on the materials and response data, as input to a language model and obtains an estimated answer corresponding to the question from the language model. Alternatively, the response management means takes prompt data, which includes materials, response data, and generation instruction information that causes a standard response corresponding to the question to be rewritten based on the response data, as input to a language model and obtains an estimated response corresponding to the question from the language model.

[0011] A management system according to one aspect of the present invention comprises an information terminal equipped with a display unit, and a management device that transmits analysis data, which outputs questions for analyzing the speech patterns of a subject, and question data, which includes questions about predetermined materials, to the information terminal. The information terminal outputs questions in at least one form (voice and text) based on the analysis data, and outputs one or more questions in at least one form (voice and text) based on the question data. The management device acquires information relating to the voice of the subject uttered in response to the questions as response data via the sound pickup unit of the information terminal. The management device takes prompt data, which includes materials, response data, and generation instruction information that causes the system to generate an estimated answer to a question based on the materials and response data, as input to a language model. It then obtains an estimated answer, which is the estimated answer to the question, from the language model, obtains information related to the subject's voice response to the question as response data, and calculates the similarity between the response data corresponding to the question and the estimated answer corresponding to the question. Alternatively, the management device takes prompt data, which includes materials, response data, and generation instruction information that causes a standard answer corresponding to a question to be rewritten based on the response data, as input to a language model, obtains an estimated answer from the language model which is the answer that is estimated to be given to the question, obtains information related to the subject's voice response to the question as response data, and calculates the similarity between the response data corresponding to the question and the estimated answer corresponding to the question.

[0012] A management program according to one aspect of the present invention comprises one or more processors that function as: an output processing means that transmits analysis data to an information terminal to output questions for analyzing the speech patterns of a subject, and outputs the questions to the information terminal in at least one of voice and text; an acquisition processing means that transmits question data including questions about predetermined materials to an information terminal, and outputs one or more questions to the information terminal in at least one of voice and text; an acquisition processing means that, via the sound pickup unit of the information terminal, acquires information relating to the voice of the subject who responded to the questions as response data, and acquires information relating to the voice of the subject who answered the questions as answer data; and an answer management means that determines the similarity between the answer data corresponding to a question and the estimated answer which is the answer that is estimated to be given to the question. The response management means takes prompt data, which includes materials, response data, and generation instruction information that causes the language model to generate an estimated answer to a question based on the materials and response data, as input to a language model and obtains an estimated answer corresponding to the question from the language model. Alternatively, the response management means takes prompt data, which includes materials, response data, and generation instruction information that causes a standard response corresponding to the question to be rewritten based on the response data, as input to a language model and obtains an estimated response corresponding to the question from the language model.

[0013] A management method according to one aspect of the present invention is configured such that one or more processors perform the following steps: sending analysis data to an information terminal to output questions for analyzing the speech patterns of a subject, and having the questions output to the information terminal in at least one of voice and text; acquiring information relating to the voice of the subject who responded to the questions as response data via the sound pickup unit of the information terminal; taking prompt data including predetermined materials, response data, and generation instruction information that generates an estimated answer to a question about the materials based on the materials and response data as input to a language model, and acquiring an estimated answer which is the estimated answer to the question from the language model; sending question data including questions to an information terminal, and having one or more questions output to the information terminal in at least one of voice and text; acquiring information relating to the voice of the subject who responded to the questions as response data via the sound pickup unit of the information terminal; and determining the similarity between the response data corresponding to the question and the estimated answer corresponding to the question.

[0014] A management method according to one aspect of the present invention is configured such that one or more processors perform the following steps: sending analysis data to an information terminal to output questions for analyzing the language use of a subject, and having the questions output to the information terminal in at least one of voice and text; acquiring information relating to the voice of the subject who responded to the questions as response data via the sound pickup unit of the information terminal; taking prompt data including predetermined materials, response data, and generation instruction information that causes a standard answer corresponding to a question about the materials to be rewritten based on the response data as input to a language model, and acquiring an estimated answer from the language model which is an estimated answer to the question; sending question data including questions to an information terminal, and having one or more questions output to the information terminal in at least one of voice and text; acquiring information relating to the voice of the subject who responded to the questions as response data via the sound pickup unit of the information terminal; and determining the similarity between the response data corresponding to the question and the estimated answer corresponding to the question. [Effects of the Invention]

[0015] According to the present invention, since the response data related to the voice of the subject to the question is reflected in the estimated answer compared with the answer voice data, the analysis accuracy of the answer to the question can be improved.

Brief Description of the Drawings

[0016] [Figure 1] It is a configuration diagram illustrating a test management system according to an embodiment of the present invention. [Figure 2] It is a block diagram illustrating the functional configuration of the information terminal in FIG. 1. [Figure 3] It is a block diagram illustrating the functional configuration of the confirmation terminal in FIG. 1. [Figure 4] It is an explanatory diagram showing an example of a response acquisition screen displayed on the display unit of the information terminal in FIG. 1. [Figure 5] It is an explanatory diagram showing an example of an introduction screen displayed on the display unit of the information terminal in FIG. 1. [Figure 6] It is an explanatory diagram showing an example of a question screen displayed on the display unit of the information terminal in FIG. 1. [Figure 7] It is an explanatory diagram showing another example of a question screen displayed on the display unit of the information terminal in FIG. 1. [Figure 8] It is an explanatory diagram showing another example of a question screen displayed on the display unit of the information terminal in FIG. 1. [Figure 9] It is an explanatory diagram showing an example of a confirmation screen displayed on the display unit of the information terminal in FIG. 1. [Figure 10] It is an explanatory diagram illustrating a confirmation screen in a state where the text part is modifiable and displayed on the display unit of the information terminal in FIG. 1. [Figure 11] It is an explanatory diagram illustrating a confirmation screen after replenishment and displayed on the display unit of the information terminal in FIG. 1. [Figure 12] It is an explanatory diagram showing an example of a status confirmation screen displayed on the display unit of the confirmation terminal in FIG. 1. [Figure 13] It is an explanatory diagram showing another example of a status confirmation screen displayed on the display unit of the confirmation terminal in FIG. 1. [Figure 14]Figure 1 is an explanatory diagram showing an example of a certification screen displayed on the information terminal's display unit. [Figure 15] This is a sequence chart illustrating the outline of the process performed in cooperation between the management device and the information terminal shown in Figure 1, as part of the management method in an embodiment of the present invention. [Figure 16] This flowchart illustrates the processing steps taken after receiving question data from a management device, as part of an information terminal management method according to an embodiment of the present invention. [Figure 17] This flowchart illustrates the processing steps taken after transmitting question data to an information terminal, as part of a management method using a management device according to an embodiment of the present invention. [Figure 18] This is an explanatory diagram showing an example of a question screen (standby state) displayed on the information terminal's display unit in Figure 1. [Figure 19] This is an explanatory diagram showing an example of a question screen (recording status) displayed on the information terminal display unit in Figure 1. [Figure 20] This is an explanatory diagram showing an example of a confirmation screen (confirmation status question screen) displayed on the information terminal display unit in Figure 1. [Figure 21] This is an explanatory diagram showing another example of the confirmation screen (confirmation status question screen) displayed on the information terminal display unit in Figure 1. [Figure 22] This is an explanatory diagram showing other examples of the confirmation screen (confirmation status question screen) displayed on the information terminal display unit shown in Figure 1. [Modes for carrying out the invention]

[0017] Embodiment. Based on Figures 1 to 17, an example of the overall configuration, functional configuration, and operation of the management system 100 in an embodiment of the present invention will be described. The same reference numerals are used for components common to each figure, and their explanations will be omitted or simplified as appropriate. In each figure, some reference numerals will be omitted as appropriate to avoid making the drawings more complex. First, with reference to Figures 1 to 3, the overall configuration of the management system 100 and the functional configuration of its main devices will be described.

[0018] The management system 100 manages information related to predetermined documents and, as shown in Figure 1, includes at least a management device 10 and an information terminal 20. The management system 100 may also have a confirmation terminal 30. The management system 100 may also have an operation terminal 60 and a model server 80. The management device 10, information terminal 20, confirmation terminal 30, operation terminal 60, and model server 80 are able to communicate with each other via a network N such as the Internet, either by wire or wireless connection.

[0019] The types of materials that can be used vary depending on the intended use of this system, that is, the attributes of the people answering the questions. Specific examples of materials could include company websites, company brochures, company videos (concept videos, product and service introduction videos, recruitment videos, etc.), newspapers, magazines, contracts, and daily reports.

[0020] The operating terminal 60 is a PC (Personal Computer) or the like used by the operators of the system (hereinafter referred to as "this system") that manages questions and answers regarding materials, provided by the management device 10. PCs include desktop PCs, notebook PCs, tablet PCs, and smartphones. The operators on the operating side use the operating terminal 60 to add and edit information from the software and databases managed by the management device 10, and to perform maintenance on this system.

[0021] The model server 80 is a server that stores and manages language models 80m, for example, consisting of large language models (LLMs). In addition to the language models 80m, the model server 80 has a model control unit 82 that controls input and output from external devices to the language models 80m. The model server 80 is composed of a cloud server based on cloud computing, an on-premise physical server, or a server system that combines these.

[0022] The management device 10 manages questions and answers regarding the materials. More specifically, the management device 10 provides questions about the materials to the target person via the information terminal 20, and analyzes the target person's answers to those questions to digitize and manage the target person's understanding of the materials. Furthermore, the management device 10 visualizes this data via the information terminal 20 and the confirmation terminal 30. Understanding status is a concept that includes the degree of understanding, comprehension level, proficiency level, and awareness level. The management device 10 is composed of a cloud server based on cloud computing, an on-premise physical server, or a server system combining these.

[0023] As illustrated in Figure 1, the management device 10 includes a communication unit 11, a control unit 12, and a storage unit 13. The communication unit 11 is an interface for the control unit 12 to communicate with external devices such as an information terminal 20, a confirmation terminal 30, an operation terminal 60, and a model server 80 via wired or wireless connection.

[0024] The memory unit 13 stores various information, including the operation program of the control unit 12, which includes the management program 13p. The management program 13p is software that works in cooperation with computing devices such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit) to realize various functions. The management program 13p causes one or more processors to function as the information processing means 12a, output processing means 12b, acquisition processing means 12c, and response management means 12d, which will be described later. The memory unit 13 stores a template for prompt data (hereinafter simply referred to as "template"). Several pieces of information are pre-loaded into the template, and response data and other information are added afterward to complete the prompt data. Prompt data is input information that causes an information processing device such as a computer to generate a desired output.

[0025] The memory unit 13 stores information for displaying and operating various screens on external devices such as the information terminal 20 and the confirmation terminal 30. The memory unit 13 may have a subject database (subject DB) in which information about the subject is organized. The memory unit 13 stores information on questions (hereinafter simply referred to as "questions") for analyzing the subject's language use. The memory unit 13 may store information on multiple questions. The memory unit 13 may store one or more questions related to the materials. The memory unit 13 may store a question table in which identification numbers of multiple questions are associated with the content of each question. The question table may also have weight coefficients set for each question associated with it.

[0026] The memory unit 13 may store an evaluation table in which multiple numerical ranges divided according to the magnitude of similarity and multiple evaluation data assigned in stages to each numerical range are associated. The memory unit 13 may store a score calculation formula for determining a provisional score or an understanding score. The memory unit 13 may store a score correspondence table including multiple numerical ranges divided according to the magnitude of the provisional score and multiple understanding scores assigned in stages to these ranges. The memory unit 13 may store a standard answer database (DB) that associates multiple questions with multiple standard answers. The memory unit 13 is composed of RAM (Random Access Memory), ROM (Read Only Memory), flash memory, eMMC (embedded Multi Media Card), SSD (Solid State Drive), or HDD (Hard Disk Drive), etc.

[0027] The control unit 12 provides various information in response to requests from external devices. The control unit 12 appropriately transmits data for screen display to the information terminal 20 and the confirmation terminal 30, causing various screens to be displayed on these display units. In this embodiment, "screen" is a concept that includes not only the entire display area of ​​the display unit or an independent display unit such as a web page, but also a portion of the display area that is partially generated, updated, and expanded within a single display screen or web page. That is, in this specification, "screen" refers to a collection of information displayed on the display unit, and includes not only modes in which the entire display area is switched (such as page transitions), but also modes in which a portion of the display area is dynamically generated, updated, and expanded within the same display screen.

[0028] The control unit 12 appropriately stores information transmitted from external devices such as the information terminal 20, confirmation terminal 30, operation terminal 60, and model server 80 in the storage unit 13. The control unit 12 is composed of one or more processors. More specifically, the control unit 12 is composed of a processing unit such as a CPU, MPU, or GPU, and the various functions of the control unit 12 are realized by the processing unit and a control program such as the management program 13p. Some of the various functions of the control unit 12 may be realized by hardware.

[0029] The control unit 12 includes an information processing means 12a, an output processing means 12b, an acquisition processing means 12c, and a response management means 12d. The information processing means 12a performs basic processing of the system, such as processing related to access and login to the system from the information terminal 20 and the confirmation terminal 30.

[0030] The output processing means 12b transmits analysis data to the information terminal 20 that generates questions for analyzing the subject's speech patterns, and causes the information terminal 20 to output these questions in at least one of voice and text. In this embodiment, the output processing means 12b displays a response acquisition screen PC based on the analysis data on the display unit 24 of the information terminal 20. The output processing means 12b extracts question information from the storage unit 13 to generate analysis data. If multiple questions are stored in the storage unit 13, the output processing means 12b may extract a set number of questions based on predetermined extraction conditions, or it may extract them randomly.

[0031] The output processing means 12b transmits question data, which includes questions about predetermined materials, to the information terminal 20, and causes the information terminal 20 to output one or more of these questions in at least one of voice and text. The question data is information for outputting one or more questions about the materials. In this embodiment, the output processing means 12b causes the display unit 24 of the information terminal 20 to display a question screen Ps based on the question data. The output processing means 12b may also transmit the question data to the information terminal 20 to display a confirmation screen Pk based on the question data on the display unit 24.

[0032] The output processing means 12b may extract one or more questions from the storage unit 13 to generate question data. In this case, the output processing means 12b may extract one or more pre-set questions from the question table, extract one or more questions based on predetermined extraction conditions, or extract one or more questions randomly. The output processing means 12b may cause the language model 80m to generate one or more questions. In this case, the prompt data for generating estimated answers may include elements for generating one or more questions.

[0033] The acquisition processing means 12c acquires information relating to the subject's voice spoken in response to a question as response data via the sound pickup unit 27 of the information terminal 20. Hereinafter, the subject's voice spoken in response to a question will also be referred to as "spoken voice". The acquisition processing means 12c acquires information relating to the subject's voice response to the question (answer voice information) as response data. The subject's voice response to a question is the voice spoken by the subject when they orally answer the question, and will hereinafter simply be referred to as "answer voice". The answer voice information may be audio data of the answer voice, or it may be answer text converted from the answer voice by speech recognition processing. That is, the conversion of the answer voice into text may be performed by the information processing unit 22 of the information terminal 20, or by the control unit 12 (acquisition processing means 12c). The acquisition processing means 12c registers a log related to the acquisition of the response data.

[0034] The acquisition processing means 12c may transmit answer confirmation data for verifying the acquired answer data to the information terminal 20 and display a confirmation screen Pk based on the answer confirmation data on the display unit 24. In this embodiment, the confirmation screen Pk allows at least one of (hereinafter also referred to as "modification, etc.") of the text (answer text) based on the answer data. If the answer text is modified, etc. on the confirmation screen Pk, the acquisition processing means 12c acquires and registers a log related to the modification, etc. from the information terminal 20.

[0035] If the confirmation screen Pk has a replenishment button Bh, the acquisition processing means 12c acquires information relating to the subject's response audio picked up by the sound pickup unit 27 of the information terminal 20 after the replenishment button Bh is pressed, as part of the response data, and also acquires and registers a log relating to the replenishment of the response. For example, the log relating to the replenishment of the response includes information indicating that the response has been replenished, and information about the replenished response.

[0036] The response management means 12d calculates the similarity (hereinafter also simply referred to as "similarity") between the response data corresponding to a question and the estimated response, which is the answer estimated for the question. If multiple questions are set for the material, the response management means 12d may calculate the similarity for the response data of each of the multiple questions. The response management means 12d performs the similarity calculation using the response text, but if the response text has been modified, it performs the similarity calculation using the modified text. Various well-known calculation methods can be used for calculating the similarity, such as methods using morphological analysis and vectorization, methods using Transformer models such as BERT, or combinations thereof. In each figure in this embodiment, an example is shown in which the similarity is displayed by converting the cosine similarity between the estimated response text data and the response text into a percentage.

[0037] The response management means 12d generates status management data indicating the subject's understanding of the material using the calculated similarity. If the response management means 12d calculates the similarity for each of the answer data for multiple questions, it generates status management data using the multiple calculated similarities. The response management means 12d may also generate response certification data indicating the question-and-answer process regarding the material using the calculated similarity, logs related to response corrections, and logs related to response supplementation.

[0038] The response management means 12d may obtain evaluation data indicating the evaluation result of the response to a question, based on the similarity associated with the question. The evaluation data is information that indicates a graded ranking related to the similarity between the response text and the estimated response, such as "○", "△", "×", "Excellent", "Good", "Acceptable", "Poor", "S", "A", "B", "C", "Pass", "Fail". The response management means 12d may obtain the evaluation data by comparing the similarity with an evaluation table, by comparing the similarity with one or more thresholds, or by using a function that associates the similarity with the evaluation data.

[0039] The answer management means 12d of this embodiment has a function to determine a comprehension score using similarity. The comprehension score may be based on a simple calculation, such as the average of the similarity scores of multiple questions. In this case, if there is only one question, the similarity score becomes the comprehension score. The answer management means 12d may also determine a provisional score, such as the average of the similarity scores of multiple questions, and then determine the comprehension score by comparing the provisional score with multiple thresholds set for rating. In this case, the comprehension score may be expressed in a tiered manner, such as "AAA", "AA", "A", "BBB", "BB", "B", "CCC", "CC", "C", "D", or "S", "A", "B", "C", "D".

[0040] As shown in Table 1 below, if a weight coefficient is set for each question, the comprehension score or provisional score can be calculated using a score calculation formula such as Equation 1 below. The weight coefficient can be set arbitrarily according to the importance of the question, etc. Table 1 is a schematic example of a question table. Equation 1 is an example of a score calculation formula when the number of questions is "3". The numerator and denominator of the score calculation formula will change depending on the number of questions.

[0041] [Table 1]

[0042]

number

[0043] Here, the similarity scores corresponding to questions 1 through n are denoted as R1 through Rn, and the weight coefficients corresponding to questions 1 through n are denoted as M1 through Mn. In Equation 1, a, b, and c correspond to the question numbers. That is, Ra, Rb, and Rc are the similarity scores corresponding to questions a, b, and c, respectively, and Ma, Mb, and Mc are the weight coefficients corresponding to questions a, b, and c, respectively. However, the database storing the question content and the database storing the weight coefficients may be set up separately. In short, the combination of information stored in the database can be arbitrarily set and changed.

[0044] If the score calculation formula is for calculating a provisional score, the response management means 12d may calculate the comprehension score by referring the obtained provisional score to a score correspondence table as illustrated in Table 2. Table 2 is a schematic example of a score correspondence table. As shown in Table 2, the score correspondence table associates multiple numerical ranges divided according to the magnitude of the provisional score with multiple comprehension scores assigned in stages to each numerical range. The numerical ranges in the score correspondence table can be set arbitrarily. However, the method for calculating the comprehension score from the provisional score is not limited to the above, and the response management means 12d may calculate the comprehension score using a function that directly or indirectly associates the provisional score and the comprehension score.

[0045] [Table 2]

[0046] In this embodiment, the response management means 12d is configured to obtain an estimated response using response data. For example, a template containing data and generation instruction information may be stored in the storage unit 13, and the response management means 12d may be configured to incorporate the acquired response data into the template to generate prompt data. Here, we will illustrate two methods for calculating estimated answers.

[0047] [First method for calculating estimated answers] The response management means 12d takes prompt data, which includes materials, response data, and generation instruction information, as input to the language model 80m, and obtains an estimated response corresponding to a question from the language model 80m. The generation instruction information related to the first method is information that causes the system to generate an estimated response to a question about materials based on the materials and response data. The prompt data may include a question. The generation instruction information may also be information that causes the system to generate an estimated response to a question based on materials and response data, along with the question based on the materials.

[0048] The generation instruction information includes instruction information that represents a predetermined instruction. The instruction information uses terms such as instruction text and purpose as headings. The generation instruction information may also include reference information that optionally includes output conditions, target, number of characters, difficulty level, etc. The reference information uses terms such as constraints and constraint conditions as headings.

[0049] A schematic example of the structure of prompt data (prompt for generation) based on the first method is shown below. First, we will illustrate prompt data that includes "generation instruction information that generates an estimated answer to a question based on the provided materials and response data." This prompt data assumes that the question has been set in advance. The question may be pre-created, for example, and may be generated using a trained model such as the language model 80m. The following is a schematic example of prompt data containing three questions.

[0050] # the purpose Based on the provided materials, please create estimated answers to each of the following three questions. # Materials (········) # question Question 1: ... Question 2: ... Question 3: ... # Response data (········) # Restrictions • [Estimated Response] should be created assuming that the response is from someone who uses language similar to that found in the response data. Please briefly quote the relevant section from the source material that supported your answer at the end of each answer. • For the [evidence] section, please extract the relevant passage from the document word for word without changing a single word. • Output format: Please strictly adhere to the following format. # Output format Q[number]: [question text] A [Number]: [Estimated Answer]... [basis]····

[0051] In this example, the content of "# Purpose" corresponds to instruction information, and the content of "# Question," "# Constraints," and "# Output Format" corresponds to reference information. Under "# Materials," for example, text data as materials may be inserted, or information about the source of the materials (such as URLs or database information) may be included. Materials may be directly attached as files such as PDFs. Database information consists of, for example, connection information indicating the location of the materials, authentication information regarding access rights, and operation information regarding the acquisition of the materials. Under "# Response Data," text data as response data may be inserted. Response data may be directly attached as audio or video files. Note that the content of "# Materials" and "# Response Data" corresponds to reference information for prompt data, but in this embodiment, materials and response data are considered separately from the generation instruction information. In this example, the combination of information related to the first sentence (first point) of "# Purpose," "# Response Data," and "# Constraints" is particularly distinctive, but these configurations can be appropriately modified without departing from the spirit of this invention.

[0052] A template containing "generation instruction information that generates estimated answers to questions about the materials based on the materials and response data" may be stored in the storage unit 13. The response management means 12d then incorporates the acquired response data into the template to generate prompt data (generation prompts).

[0053] Next, we will provide an example of prompt data that includes "a question based on the materials, along with generation instruction information that generates an estimated answer based on the materials and response data for that question." The following is a schematic example of the prompt data structure when generating a single question (a configuration that presents a single question).

[0054] # Materials (········) # Response data (········) # Instructions (task definition) Based on the above materials, please create a set of questions to measure comprehension and corresponding estimated answers. # Constraints • For the [Estimated Response], please create it by faithfully reproducing the "respondent's wording" as seen in the [Response Data], and by putting yourself in the respondent's shoes. • Your [Estimated Answer] should be based solely on the information provided in the [Materials]. Do not include anything not mentioned in the materials, and exclude your own speculation or external knowledge. Output format: Please strictly adhere to the following format. # Output format Q[number]: [question text] A [Number]: [Estimated Answer] [Evidence] (Quote the relevant section from the supporting document)

[0055] In this example, the content of "# Instructions" corresponds to instruction information, and the content of "# Constraints" and "# Output Format" corresponds to reference information. If you want to generate multiple questions, you can adjust the content of "# Instructions" to, for example, "Based on the above materials, please create a set of ○ questions to measure the level of understanding of the content, and the estimated answers [estimated answers] for each question." Note that the content of "# Materials" and "# Response Data" corresponds to reference information for the prompt data, but in this embodiment, the materials and response data are considered separately from the generation instruction information. In this example, the combination of information related to the first sentence (first point) of "# Response Data," "# Instructions," and "# Constraints" is particularly distinctive, but these configurations can be changed as appropriate without departing from the purpose of this invention.

[0056] In each of the above examples, the reference information includes information to refer to the contents of the [response data], but this information may also be included in the instruction information, for example, as shown below. The content of "# Purpose" in the prompt data of the first example can also be adjusted in the same way as shown below.

[0057] # Instructions (task definition) Based on the [Materials], please create estimated answers to the following [Questions]. When doing so, please ensure that the [Estimated Answers] faithfully reproduce the "respondent's wording" as gleaned from the [Response Data].

[0058] A template containing "generation instruction information that generates estimated answers to questions about the materials based on the materials and response data" may be stored in the storage unit 13. The response management means 12d then incorporates the acquired response data into the template to generate prompt data (generation prompts).

[0059] [Second method for calculating estimated answers] The response management means 12d may take prompt data, which includes materials, response data, and generation instruction information that causes the language model 80m to rewrite a standard answer corresponding to a question based on the response data, as input to the language model 80m, and obtain an estimated answer corresponding to a question from the language model 80m. That is, the generation instruction information related to the second method is information that causes the language model 80m to rewrite a standard answer corresponding to a question based on the response data. The prompt data may contain a question. The standard answers may be prepared in advance and stored in the storage unit 13. For example, the storage unit 13 may store a standard answer DB in which multiple questions and standard answers corresponding to each question are associated. The response management means 12d may obtain a set number of questions and standard answers from the standard answer DB and insert them into a template.

[0060] Here, we will provide a schematic example of the structure of prompt data (rewrite prompt) based on the second method. The following is a schematic example of prompt data containing two questions.

[0061] # Materials (········) # Instructions (task definition) Based on the [Materials], please create [Estimated Answers] for the following [Questions], while comprehensively covering the content of the [Standard Answers] and faithfully reproducing the "respondent's language style" as gleaned from the [Response Data]. The goal is to rewrite the original [Standard Answers] in a style similar to that of the respondents in the [Response Data]. # question Question 1: ... Question 2: ... # Standard answer • Standard answer to Question 1: ... • Criteria answer for Question 2: ... # Response data (········) # Guidelines for rewriting • Maintaining Information: Do not change the content (outline) of the [Standard Response]. Do not add new elements or supplementary steps that are not included in the [Standard Response], even if they are included in the [Materials]. • Tone transformation: Without changing the message of the [Standard Response], rewrite only the "phrasing" and "speech style" to match the style of the respondent in the [Response Data]. # Restrictions • Facts: The basis for your answer must be based on [documents]. The scope of your answer must not deviate from the [standard answer]. Output format: Please strictly adhere to the following format. # Output format Q[number]: [question text] A [Number]: [Estimated Answer] [Basis] (A short, word-for-word quote from the source material that supports the description in [Estimated Answer])

[0062] In this example, the content of "# Instructions" corresponds to instruction information, while the content of "# Questions," "# Standard Answers," "# Rewriting Guidelines," "# Constraints," and "# Output Format" corresponds to reference information. Note that the content of "# Documents" and "# Response Data" corresponds to reference information for prompt data, but in this embodiment, documents and response data are considered separately from the generation instruction information. In this example, the combination of information related to "# Instructions," "# Response Data," and "# Rewriting Guidelines" is particularly distinctive, but these configurations can be appropriately modified without departing from the spirit of this invention. However, the prompt data may be configured to generate one or more questions and corresponding standard answers. In this case, the generation instruction information is organized so that response data is not used to generate the standard answers.

[0063] It is advisable to store a template containing "generation instruction information that rewrites the standard answer corresponding to the question based on the response data" in the storage unit 13. The response management means 12d then incorporates the acquired response data into the template to generate prompt data (rewrite prompt).

[0064] The response management means 12d transmits status management data, generated using one or more similarity scores, relating to the subject's understanding of the material, to the confirmation terminal 30, and displays a status confirmation screen Pm based on the status management data on the display unit 34. The status confirmation screen Pm is a screen that shows the subject's understanding of the material. The status confirmation screen Pm should be configured to include information indicating whether or not the response has been corrected and information indicating whether or not the response has been supplemented.

[0065] The response management means 12d transmits response certification data, which is generated using one or more similarity scores and shows the question-and-answer process regarding the document, to the information terminal 20, and displays a certification screen Pz based on the response certification data on the display unit 24. The certification screen Pz is for certifying the history of the answers to questions regarding the document, the question-and-answer process, and its completion. The certification screen Pz should be configured to include information indicating whether the answer has been modified and information indicating whether the answer has been supplemented.

[0066] The response management means 12d may have a function to identify the modified parts and estimate the type of modification by comparing the original text, which is the response text before modification, with the modified text, which is the response text after modification. That is, the response management means 12d may be configured to identify one or more modified parts from the original text to the modified text and generate type data that estimates the type of modification of the modified parts. The type of modification refers to, for example, deletion, insertion, correction of typographical errors, change of expression, or supplementation of content. Of these, typographical errors include errors caused by oral mistakes such as the speaker's slip of the tongue, errors in the speech recognition / conversion process, omission of characters, notational defects (incorrect conversion, etc.), and display defects (garbled characters, etc.). Note that in Figures 13 and 14, "typographical errors" is used as the type of modification, abbreviating "correction of typographical errors."

[0067] The response management means 12d may sequentially identify the modified portion each time it is modified. In this case, the response management means 12d may also estimate the type of modification at the time of identification, or it may estimate one or more types of modifications after all modified portions have been identified. The response management means 12d may also identify one or more modified portions by comparing the original text and the modified text after all modifications have been completed.

[0068] The response management means 12d may use difference extraction algorithms such as SES (Shortest Edit Script) or LCS (Longest Common Subsequence) when estimating the type of modification, or it may use vector operations based on the extracted features, or it may use a trained model such as the language model 80m. However, the response management means 12d may be configured to perform preprocessing on the response text, such as so-called filler removal or tweak removal.

[0069] The response management means 12d may be configured to generate information for the certification screen Pz (or a dataset containing such information) based on the response certification data in VC (Verifiable Credential) format. To ensure the authenticity of the information for the certification screen Pz, the response management means 12d may use a blockchain (distributed ledger) on a P2P network to issue the information in a manner that is difficult to tamper with. The information processing means 12a may issue a URL for displaying the certification screen Pz, or a two-dimensional code that encodes such a URL, and transmit it to the information terminal 20.

[0070] Information terminal 20 is used for tasks such as logging in participants, providing them with questions and inquiries, obtaining their responses, and modifying the response text. Information terminal 20 is not limited to devices owned by participants; it may also be a device lent to them. Each participant uses information terminal 20 to answer questions and inquiries. Information terminal 20 has at least a microphone to pick up ambient sounds. Information terminal 20 may have a built-in camera or an external camera may be connected. Ideally, information terminal 20 should be a portable device such as a smartphone, tablet PC, or notebook PC.

[0071] As illustrated in Figure 2, the information terminal 20 includes a communication unit 21, an information processing unit 22, a storage unit 23, a display unit 24, an input unit 25, a camera 26a, a camera 26b, a sound pickup unit 27, and a notification unit 28. The communication unit 21 is an interface for the information processing unit 22 to communicate with external devices such as the management device 10 via wired or wireless connection. The display unit 24 is, for example, a liquid crystal display (LCD) and displays various information according to instructions from the information processing unit 22. The input unit 25 receives input operations from examinees, etc., and outputs an operation signal to the information processing unit 22 according to the content of the received operation. The information terminal 20 may also have a touch panel in which the display unit 24 and the input unit 25 are stacked. In such a configuration, the display unit 24 functions as a display panel that displays images including characters, etc., and the input unit 25 functions as a detection means for detecting touch operations.

[0072] Cameras 26a and 26b each have an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) and capture videos and still images. Camera 26a is a so-called in-camera located on the same side as the display unit 24. Camera 26b is a so-called out-camera located on the opposite side from the display unit 24. Since either camera 26a or camera 26b can be used for shooting with the information terminal 20, when referring to cameras 26a and 26b without distinction, they are collectively referred to as camera 26. However, the information terminal 20 may have one camera built in or connected, or it may not have a camera at all.

[0073] The information processing unit 22, in cooperation with the management device 10, displays various screens (such as web pages), including the response acquisition screen Pc, introduction screen Pd, question screen Ps, confirmation screen Pk, and certification screen Pz, as well as related information and subwindows, on the display unit 24 in response to operation signals from the input unit 25. The information processing unit 22 receives operations from the subject via each screen and executes processing according to the content of the operation. The information processing unit 22 activates the sound collection unit 27 in response to instructions from the subject via the input unit 35 or instructions from the management device 10.

[0074] For example, the information processing unit 22 starts recording sound in the sound recording unit 27 in response to a sound recording command from the management device 10, and transmits the sound recorded by the sound recording unit 27 to the management device 10. The information processing unit 22 may also continuously transmit the audio related to the subject's response recorded by the sound recording unit 27 to the management device 10, or it may transmit the response audio or the response text (a text version thereof) to the management device 10 when the subject has finished responding.

[0075] If the management system 100 is configured to display a confirmation screen Pk, the transcription of the answer audio is performed on the information terminal 20 side, so the information terminal 20 only needs to send the answer text as answer data to the management device 10. If the answer text is modified, the information terminal 20 sends the final modified answer text as answer data to the management device 10. The information processing unit 22 may start taking pictures with the camera 26 in response to a shooting command from the management device 10 and send the resulting video to the management device 10.

[0076] The information processing unit 22 is composed of one or more processors. More specifically, the information processing unit 22 is composed of arithmetic processing units such as a CPU, MPU, or GPU, and the various functions of the information processing unit 22 are realized by the arithmetic processing units and the operation program. Some of the various functions of the information processing unit 22 may be realized by hardware. The storage unit 23 stores various information in addition to the operation program of the information processing unit 22. The operation program of the information processing unit 22 also includes a segmentation program that is transmitted from the management device 10 in a timely manner. The segmentation program constitutes the management program 13p. The information processing unit 22 can perform some processes, such as switching the display of various elements on the screen, without communicating with the management device 10, through operations based on the segmentation program. The operation program of the information processing unit 22 is software that works in cooperation with arithmetic processing units such as a CPU, MPU, or GPU to realize various functions. The storage unit 23 is composed of RAM and ROM, flash memory, eMMC, SSD, or HDD. The sound pickup unit 27 is a microphone that picks up ambient sounds. The notification unit 28 includes a speaker and outputs sound or voice.

[0077] Users register personal information such as their name, unique identification information (such as an ID or email address), and a unique password in this system via an information terminal 20 or the like. At least one of the identification information and password may be assigned by the management device 10. Users can operate this system by, for example, entering their identification information and password on a login screen provided by the management device 10, and can register answers to questions and inquiries guided by various pieces of information.

[0078] The verification terminal 30 is a PC or similar device used by an administrator to verify the subject's understanding of the materials. As shown in Figure 3, the verification terminal 30 includes a communication unit 31, a verification processing unit 32, a storage unit 33, a display unit 34, and an input unit 35. The communication unit 31 is an interface for the verification processing unit 32 to communicate with external devices such as the management device 10 via wired or wireless connection. The display unit 34 is, for example, a liquid crystal display and displays various information according to instructions from the verification processing unit 32. The input unit 35 receives input operations from examinees, etc., and outputs an operation signal to the verification processing unit 32 according to the content of the received operation. The input unit 35 is, for example, composed of a keyboard and a pointing device such as a mouse or trackball.

[0079] The confirmation processing unit 32, in cooperation with the management device 10, displays various screens (such as a status confirmation screen Pm) and related information / subwindows on the display unit 34 in response to operation signals from the input unit 35. The confirmation processing unit 32 accepts operations from examinees, etc., through each screen and executes processing according to the content of the operation. The confirmation processing unit 32 is composed of a processing unit such as a CPU, MPU, or GPU, and the various functions of the confirmation processing unit 32 are realized by the processing unit and the operation program. Some of the various functions of the confirmation processing unit 32 may be realized by hardware.

[0080] The storage unit 33 stores various information in addition to the operation program of the verification processing unit 32. The operation program of the verification processing unit 32 also includes a partitioning program that is transmitted from the management device 10 in a timely manner. The partitioning program constitutes the management program 13p. The verification processing unit 32 can perform several processes, such as switching the display of various elements on the screen, without communicating with the management device 10, based on the operations of the partitioning program. The operation program of the verification processing unit 32 is software that works in cooperation with a processing unit such as a CPU, MPU, or GPU to realize various functions. The storage unit 33 is composed of RAM and ROM, flash memory, eMMC, SSD, or HDD.

[0081] Next, referring to Figures 4 to 11, we will explain examples of the configuration of each screen related to answering the questions. Each figure shows an example where the information terminal 20 is a smartphone. In the following, redundant explanations of components common to each screen, as well as explanations of some components of each screen, will be omitted as appropriate. The same matters will be omitted as appropriate in Figures 12 to 14, which will be discussed later.

[0082] Figure 4 is an explanatory diagram illustrating the response acquisition screen PC displayed on the display unit 24 of the information terminal 20. The response acquisition screen PC has an explanation unit Eo, a question unit Qo, a promotion unit U, and a record button Bk. The explanation unit Eo is where explanatory information for answering the question is displayed. The question unit Qo is where questions for analyzing the subject's language use are displayed. The "○○○○○○" in the question unit Qo contains items according to the subject's attributes, such as their motivation for applying to our company, self-introduction, strengths and weaknesses, hobbies, success stories, research content at university, ongoing in-house projects, how they spend their holidays, family, recent Good & New, recommended travel destinations, etc. The questions in the question unit Qo should be unrelated to the materials in order to elicit natural answers and avoid duplication of content with the questions. The time (□ minutes) for the question unit Qo can be set arbitrarily. However, the time display is not required.

[0083] The record button Bk is a virtual button that activates the sound pickup unit 27 and starts recording. When the subject presses the record button Bk, recording starts via the sound pickup unit 27. At that time, the information processing unit 22 may switch the record button Bk on the response acquisition screen Pc to a stop button to stop recording, or it may switch the response acquisition screen Pc to a recording screen (not shown) that includes a stop button. If a time limit for answering a question is set in advance, the information processing unit 22 may stop recording when that time limit has elapsed. The facilitation unit U is where a message is displayed to encourage the user to press the record button Bk.

[0084] When the information processing unit 22 displays the response acquisition screen PC, it may output the contents of the question section Qo to the notification unit 28 as audio. In this case, the information processing unit 22 does not need to display the question section Qo on the response acquisition screen PC. In other words, the information processing unit 22 may be configured to output the question by at least one of audio and text. The response acquisition screen PC may have a back button R to return to the state (screen, etc.) that was displayed immediately before.

[0085] The response acquisition screen PC does not need to have an explanation unit Eo. When the information processing unit 22 displays the response acquisition screen PC, it may output the contents of the explanation unit Eo to the notification unit 28 as audio. The response acquisition screen PC does not need to have a facilitation unit U. When the information processing unit 22 displays the response acquisition screen PC, it may output the contents of the facilitation unit U to the notification unit 28 as audio.

[0086] Figure 5 is an explanatory diagram illustrating an introductory screen Pd displayed on the display unit 24 of the information terminal 20. The introductory screen Pd includes an explanation section Ea, a promotion section U, and an OK button Ba. The explanation section Ea is where explanations for one or more questions to be presented are displayed. The OK button Ba is a virtual button for transitioning to the question screen Ps. When the information processing unit 22 displays the introductory screen Pd, it may also output the contents of the explanation section Ea to the notification unit 28 as audio. In this case, the explanation section Ea does not need to be displayed on the introductory screen Pd. The information terminal 20 may also be configured not to display the introductory screen Pd.

[0087] Figures 6 to 8 are explanatory diagrams illustrating the question screen Ps displayed on the display unit 24 of the information terminal 20. The question screen Ps has a question unit Qn, a promotion unit U, and a record button Bk. Figures 6 to 8 show an example in which three questions are presented to the target person. Specifically, this is an example in which a company presents questions to a job seeker (the target person) about materials such as the "homepage," "brochure," and "video," and asks for their answers.

[0088] When the information processing unit 22 displays the question screen Ps, it may output the contents of the question section Qn to the notification unit 28 as audio. In this case, the information processing unit 22 does not need to display the question section Qn on the question screen Ps. In other words, the information processing unit 22 may be configured to output the question by at least one of audio and text.

[0089] When the record button Bk is pressed by the subject, the information processing unit 22 starts recording via the sound pickup unit 27. At this time, the information processing unit 22 may switch the record button Bk on the question screen Ps to a stop button to stop recording, or it may switch the question screen Ps to a recording screen (not shown) that includes a stop button. If a time limit for answering the question is set in advance, the information processing unit 22 stops recording when that time limit has elapsed.

[0090] However, candidates undergoing interviews and similar processes experience psychological stress such as tension and anxiety, which may prevent them from speaking as they normally would. This can lead to inappropriate vocabulary choices or omissions of important information. In other words, response texts that are simply transcripts of the audio responses may not accurately reflect the candidate's true intentions or abilities, potentially making them unreliable as information for talent evaluation. Therefore, the management system 100 in this embodiment is configured to allow for the modification (fine-tuning) and supplementation of the response texts.

[0091] More specifically, when the information processing unit 22 stops recording due to a stop operation by the subject or the expiration of the time limit, it displays a confirmation screen Pk, as shown in Figure 9 or Figure 10, on the display unit 24. The confirmation screen Pk in Figure 9 has a text section T, a confirmation button Bo, a correction button Bs, and a supplement button Bh. The text section T is where the answer text, which is a text transcription of the subject's voice response to the question, is displayed.

[0092] The text section T in Figure 9 is in an uneditable state. When the edit button Bs is pressed, it switches to a confirmation screen Pk like the one in Figure 10, where the answer text can be edited. If the information terminal 20 is a smartphone or tablet, the confirmation screen Pk displays an operation section W (simplified in Figure 10) with multiple virtual buttons (keys) assigned to various functions, allowing for actions such as deleting and inserting characters. The confirmation button Bo is a virtual button used to confirm the text in the text section T as the answer text. The text edited in the text section T is registered as the edited text. However, when recording is stopped, the information processing unit 22 may display a confirmation screen Pk like the one in Figure 10, which has an editable text section T, without going through the confirmation screen Pk like the one in Figure 9. Note that the text section T in Figure 9 does not necessarily need to be inverted; it is merely a visual convention to distinguish it from the text section T in Figure 10.

[0093] On the other hand, allowing unlimited modification of responses by the respondents could undermine the significance of verbal responses and raise concerns about the objective reliability of the data. Therefore, in this embodiment, the number of characters that can be modified in the response text of the confirmation screen Pk is limited to a predetermined range. In other words, the information processing unit 22 is configured to limit the number of characters that can be modified in the response text included in the text section T of the confirmation screen Pk to a predetermined range.

[0094] For example, the text field T may have either a limit on the number of characters that can be deleted (allowed deletions) or a limit on the number of characters that can be added (allowed additions). The text field T may also be configured so that the number of characters that can be added depends on the number of characters that have been deleted. For example, the text field T may be configured so that when a certain number of characters are deleted from the answer text, "a certain number + α" characters can be added. α is a predetermined number that can be changed at will.

[0095] In a configuration where the text section T has editing restrictions, if the user wishes to supplement the answer audio beyond the editable range, they press the supplement button Bh to add the audio answer. The supplement button Bh is a virtual button that accepts audio supplements to the answer to the question, functions similarly to the record button Bk, and may be switched to a stop button as appropriate. In other words, the supplement button Bh is a virtual button that activates the sound collection unit 27 and starts recording.

[0096] When recording is complete after pressing the replenishment button Bh, a confirmation screen Pk, as shown in Figure 11, is displayed on the display unit 24. The response audio (replenishment audio) recorded by pressing the replenishment button Bh is managed separately from the response audio (main audio) recorded by pressing the record button Bk. The text section T on the confirmation screen Pk in Figure 11 has a main text section Ta that displays the main text, which is the main audio converted into text, and a supplementary text section Tb that displays the supplementary text, which is the supplementary audio converted into text. When the edit button Bs is pressed, at least the supplementary text section Tb becomes editable. When the edit button Bs is pressed, both the main text section Ta and the supplementary text section Tb may become editable.

[0097] However, the management system 100 may be configured so that the answer text cannot be modified or supplemented. In this case, once the registration of the answer audio via the question screen Ps is complete, for example, a message confirming receipt of the answer will be displayed on the display unit 24 of the information terminal 20, and the question and answer session regarding the materials will be completed.

[0098] Figures 6 to 8 illustrate several questions, but a few more questions are provided below. For example, if the target group is new graduates, the following questions might be considered. (1) Please describe in detail the most challenging aspect of the "XXXX (e.g., extracurricular activities)" section of your resume, and the role you played in overcoming it. (2) How do you think the knowledge and experience you gained in "XXXX" at university or elsewhere will be useful in our company's work? (3) How would you like to contribute to our company as a member, leveraging your strengths? (1) is a question that delves deeper into the candidate's experience, (2) is a question about the application of what they have learned, and (3) is a question about their aspirations after joining the company.

[0099] For example, if the target person is a mid-career hire, the following questions might be considered. (1) Regarding the achievement "XXXX" in your resume, please describe the biggest challenge you faced and the specific role you played in overcoming it. (2) In what specific situations do you think the specialized skills listed as "XXXX" on your resume can be immediately utilized as valuable assets in our company's operations? (3) Leveraging your past experience, what challenges do you plan to address and what value will you create at our company after joining? (1) is a question that delves deeper into their achievements, (2) is a question about the immediate usability of their skills, and (3) is a question about their potential contributions after joining the company.

[0100] For example, if the target person is a highly specialized professional, the following questions might be considered. (1) Please describe the biggest challenge you faced in the "XXX work" and the specific initiatives you led to resolve it. (2) Reflect on your decisions and actions in the "XXX task" and logically explain what was effective and what could be improved. (3) Please tell us about your future plans for how you intend to utilize the knowledge you gained from this practical experience to contribute to the development of your field of expertise and society. (1) Questions concerning the clarification of the core aspects and roles in practice, (2) questions concerning the validity and application of professional judgment, and (3) questions concerning ethics and prospects for social contribution.

[0101] Next, an example of a status confirmation screen Pm displayed on the display unit 34 will be described with reference to Figures 12 and 13. The status confirmation screen Pm in Figure 12 includes a question display unit Qc where a question is displayed, a question display unit Qm where a question is displayed, an answer display unit Tm where the answer text is displayed, a similarity column D showing the similarity for each answer, an evaluation column H showing evaluation data based on the similarity, and a score column Sc showing the comprehension score. The question display unit Qc may function as a virtual button for displaying the text data of the spoken audio. The status confirmation screen Pm may also display the text data of the spoken audio.

[0102] Figure 13 shows an example of a status confirmation screen Pm that supports a configuration allowing for modification and supplementation of the answer text. The status confirmation screen Pm in Figure 13 includes a question display section Qc, a question display section Qm, a source text section Tx that displays the original text which is the answer text before modification, a modified text section Tz that displays the modified text which is the answer text after modification, a type section Sm that indicates the type of modification, a similarity column D, an evaluation column H, and a score column Sc that displays the comprehension score. If the answer has been supplemented, the supplemented text is also displayed in the modified text section Tz.

[0103] The information processing unit 22 is configured to apply highlighting to the modified parts of the modified text in the modified text section Tz according to the type of modification. In other words, the modified parts of the modified text have a different appearance from other parts (unmodified parts). The information processing unit 22 may also apply highlighting to the modified parts of the original text in the original text section Tx according to the type of modification. Possible forms of highlighting include changing the color, thickness, or typeface (font) of the characters themselves, adding a background color, underlining, adding a border, inserting symbols, or combinations thereof.

[0104] In Figure 13, various forms of highlighting are used for the modified sections (deletions, corrections of typographical errors, insertions, changes in expression, or additions) in the original text section Tx and the modified text section Tz. The forms of highlighting shown in Figure 13 are merely examples, and any form of highlighting can be applied to each modified section depending on the type of modification. In Figure 13, the highlighting of modified sections in the original text section Tx (strikethrough, shading, highlighting, insertion symbols, color changes, etc.) and the highlighting of modified sections in the modified text section Tz (underlining, shading, highlighting, etc.) correspond to information indicating whether or not a modification has been made (information indicating that a modification has been made).

[0105] In the assumption shown in Figure 13, the revised answer text corresponding to question 3 includes supplementary text. Therefore, the revised text section Tz corresponding to question 3 consists of a main text section T1 where the main text is displayed and a supplementary text section T2 where the supplementary text is displayed. The presence or absence of the supplementary text section T2 indicates whether or not the answer has been supplemented. In other words, the revised text section Tz corresponding to question 3 contains information indicating that the answer has been supplemented. The type section Sm contains information indicating whether or not the answer has been revised and information indicating whether or not the answer has been supplemented. The type section Sm corresponding to question 3 indicates that the answer has been supplemented.

[0106] Next, with reference to Figure 14, an example of the certification screen Pz displayed on the display unit 24 of the information terminal 20 will be described. The certification screen Pz in Figure 14 has a score field Scr, a remarks display unit Sd, and a question information unit Qr that displays information related to each question. The remarks display unit Sd is where information related to the comprehension score, such as advice for the subject, is displayed. The question information unit Qr exemplified in Figure 14 displays information indicating whether the answer has been corrected, information indicating whether the answer has been supplemented, the type of correction, the similarity, and the evaluation data.

[0107] Next, an example of the overview of the management method in this embodiment will be described based on the sequence chart in Figure 15. This sequence chart mainly illustrates the processing flow when there is only one question presented to the subject. This sequence chart is compatible with the management system 100, which does not have a function for confirming the answer text (such as a correction function).

[0108] When a user accesses the system via a URL or the like using the information terminal 20 and performs the login operation (step S101), the information processing unit 22 of the information terminal 20 sends a login request to the management device 10 (step S102). For example, the user performs the access process to the system at a predetermined time on a pre-specified day. The control unit 12 of the management device 10 compares the user's identification number, etc., included in the login request with the identification information, etc., in the user DB, and if the match is successful (step S103), sends initial data to the information terminal 20 (step S104).

[0109] The information processing unit 22 displays an initial screen based on the initial data on the display unit 24 (step S105), accepts a predetermined operation to request a question-and-answer session regarding the materials (step S106), and sends a start request to the management device 10 (step S107). Upon receiving the start request, the control unit 12 sends the analysis data to the information terminal 20 (step S108).

[0110] The information processing unit 22 displays a response acquisition screen PC on the display unit 24, which outputs questions for analyzing the subject's speech patterns based on the analysis data. At this time, the information processing unit 22 may output a predetermined sound from the notification unit 28, depending on the settings. When the information processing unit 22 receives an operation to start recording, such as when the record button Bk is pressed (step S110), it activates the sound collection unit 27 and starts recording the voice uttered by the subject in response to the questions (step S111). When the information processing unit 22 meets the termination conditions, such as when an operation to stop recording is performed or the time limit has elapsed, it registers the recorded voice as spoken voice (step S112) and transmits information related to the spoken voice as response data to the management device 10 (step S113).

[0111] The control unit 12 uses the received response data to generate prompt data for generating an estimated answer. For example, the control unit 12 inserts the data and response data into a template in the storage unit 13 and generates prompt data that includes the data, the response data, and generation instruction information. The prompt data may be configured to generate one or more questions. The control unit 12 may also generate prompt data that includes the data, the response data, and generation instruction information that causes the control unit 12 to rewrite a standard answer corresponding to a question based on the response data. The prompt data may be configured to generate one or more questions along with a standard answer corresponding to that question (step S114).

[0112] The control unit 12 sends the generated prompt data to the model server 80, causing the language model 80m to input the prompt data (step S115). As a result, the language model 80m generates an estimated answer, and the generated estimated answer is sent from the model server 80. When the control unit 12 obtains the estimated answer (step S116), it stores the estimated answer in the storage unit 13 and sends the question data to the information terminal 20 (step S117).

[0113] The information processing unit 22 displays an introductory screen Pd based on the question data on the display unit 24 (step S118). In response to a screen advance operation such as pressing the OK button Ba (step S119), the information processing unit 22 displays a question screen Ps on the display unit 24 for outputting one or more questions about the document (step S120).

[0114] When the information processing unit 22 receives a recording start operation, such as when the recording button Bk is pressed (step S121), it activates the sound collection unit 27 and starts recording the voice of the person answering the question (step S122). When the termination condition is met, the information processing unit 22 registers the recorded voice as the answer voice (step S123) and transmits the information related to the answer voice as answer data to the management device 10 (step S124).

[0115] When the control unit 12 receives the response data, it reads the estimated response from the storage unit 13 (step S125) and calculates the similarity between the response text based on the response data and the estimated response. If the response data transmitted from the information terminal 20 is audio data, the control unit 12 converts it to text to generate the response text. If the response audio is converted to text on the information terminal 20, the response data transmitted from the information terminal 20 corresponds to the response text (step S126).

[0116] The control unit 12 generates status management data indicating the subject's understanding of the material using one or more similarity values. The control unit 12 also generates answer certification data indicating the question-and-answer process regarding the material using one or more similarity values. In doing so, the control unit 12 may calculate an understanding score representing the subject's understanding of the material based on one or more similarity values ​​and include the calculated understanding score in the status management data and answer certification data (step S127). Then, the control unit 12 sends a processing completion notification to the information terminal 20 (step S128).

[0117] Upon receiving a processing completion notification, the information processing unit 22 displays, for example, answer completion information indicating that the question-and-answer session has been completed on the display unit 24 (step S129). However, the control unit 12 may also send answer certification data to the information terminal 20 in step S128 and display the certification screen Pz on the display unit 24. Although not shown in Figure 15, the control unit 12 may send status management data to the confirmation terminal 30 and display the status confirmation screen Pm on the display unit 24.

[0118] Each of the above processes has been explained in the order of the step numbers shown in Figure 15, but the order of the processes is not limited to this explanation, and it is also possible to change or omit some of the processes. For example, the response acquisition screen Pc may be set as the initial screen, and the processes in steps S104 to S107 may be omitted. The management system 100 may be configured not to perform the display process of the introduction screen Pd (steps S118, S119). When the control unit 12 receives response data (step S113), it may perform a comparison process with the voice of the subject that has been registered in advance. If the subject is presented with multiple questions, the processes in steps S120 to S124 may be executed for each question, as illustrated by the dashed arrows in Figure 15, or after the processes in steps S120 to S123 have been executed for each question, multiple response data (response data sets) may be transmitted in step S124.

[0119] Next, referring to the flowchart in Figure 16, we will explain the operation flow from the point of receiving question data (corresponding to step S117 in Figure 15) onward by the information terminal 20 of the management system 100, which is equipped with a function for verifying the answer text.

[0120] When the information processing unit 22 of the information terminal 20 receives question data (step S201), it displays an introductory screen Pd as shown in Figure 5 on the display unit 24 based on the question data (step S202). When the information processing unit 22 receives a screen advance operation (step S203 / Yes), it displays question screens Ps as shown in Figures 6 to 8 on the display unit 24 based on the question data (step S204).

[0121] When the information processing unit 22 receives a recording start operation (step S205 / Yes), it starts recording the voice related to the subject's response (step S206). When the termination condition is met (step S207 / Yes), the information processing unit 22 displays a confirmation screen Pk as shown in Figure 9 on the display unit 24 (step S208).

[0122] When registration is instructed, such as by pressing the confirmation button Bo (step S209 / Yes), the information processing unit 22 registers the answer text displayed on the confirmation screen Pk as the final answer text associated with that question (step S210). If there are any remaining questions (questions not yet presented) (step S211 / Yes), the information processing unit 22 proceeds to the process in step S204 and displays the subsequent questions on the display unit 24. In other words, the information processing unit 22 repeatedly executes the processes in steps S204 to S211 (S222) until there are no remaining questions.

[0123] On the other hand, if the user requests correction of the response text displayed on the confirmation screen Pk, such as by pressing the correction button Bs (step S212 / Yes), the information processing unit 22 displays the confirmation screen Pk in correction mode as shown in Figure 10 on the display unit 24 and changes the content of the text section T according to the correction operation by the user (step S213). Subsequently, if registration is instructed, such as by pressing the confirmation button Bo (step S214 / Yes), the process proceeds to step S210.

[0124] If, on the confirmation screen Pk, the supplement button Bh is pressed or otherwise requested to supplement the answer (step S212 / No, step S215 / Yes), the information processing unit 22 starts recording audio for supplementing the answer to that question (step S216). If, after step S213, the request for supplementing the answer is made (step S214 / No, S215 / Yes), the information processing unit 22 also starts recording audio for supplementing the answer to that question (step S216).

[0125] When the termination condition is met (step S217 / Yes), the information processing unit 22 displays a confirmation screen Pk as shown in Figure 11 on the display unit 24 (step S218). When registration of the answer text displayed on the confirmation screen Pk is instructed (step S219 / Yes), the information processing unit 22 proceeds to the process of step S210. On the other hand, when correction of the answer text displayed on the confirmation screen Pk is requested (step S219 / No, S220 / Yes), the information processing unit 22 displays a confirmation screen Pk on the display unit 24 that allows correction of at least the supplementary text section Tb, and changes the displayed content of the text section T according to the correction operation by the subject (step S221). When registration of the answer text displayed on the confirmation screen Pk is instructed (step S222 / Yes), the process proceeds to the process of step S210.

[0126] Once the processing of answers to all questions is complete (step S211 / No), the information processing unit 22 sends an answer dataset containing the answer text corresponding to each question to the management device 10. This allows the management device 10 to generate information indicating the subject's understanding of the material (step S223). Based on the processing completion notification sent from the management device 10, the information processing unit 22 displays the answer completion information on the display unit 24 (step S224).

[0127] When the information processing unit 22 receives an operation requesting proof of response (step S225 / Yes), it sends an audit trail request to the management device 10. This operation may be performed from a predetermined management screen or the like (step S226). When the information processing unit 22 receives the proof of response data sent from the management device 10 in response to the audit trail request (step S227), it displays a proof screen Pz, as shown in Figure 14, on the display unit 24 based on the proof of response data. This allows the subject to know the objective judgment result of their response (step S228).

[0128] Each of the above processes has been explained in the order of the step numbers shown in Figure 16, but the order of the processes is not limited to this explanation, and it is also possible to change or omit parts of the processes. For example, if multiple questions are set, the information processing unit 22 may be configured to send the answer text to the management device 10 each time the answer text for one question is determined. In this case, the information processing unit 22 may also obtain information for the next question from the management device 10. That is, when the registration of the answer text displayed on the confirmation screen Pk is instructed (step S209 / Yes), the information processing unit 22 may send the answer text to the management device 10 as answer data (step S210).

[0129] Next, referring to the flowchart in Figure 17, the processing flow from the transmission of question data by the management device 10 of the management system 100 (corresponding to step S117 in Figure 15) onward will be explained. Note that the example operation of the management device 10 in Figure 17 corresponds to the example operation of the information terminal 20 in Figure 16. In Figure 17, it is assumed that multiple questions are presented to the subject.

[0130] The control unit 12 of the management device 10 transmits question data to the information terminal 20 (step S301) and receives the answer data set from the information terminal 20 (step S302). Then, the control unit 12 obtains estimated answers for multiple questions from the storage unit 13 and, by comparing the answer text with the estimated answer, calculates the similarity between the answer text and the estimated answer for each question (step S303).

[0131] Next, the control unit 12 uses the similarity scores of each of the multiple questions to obtain a comprehension score representing the subject's understanding of the material (step S304). Then, the control unit 12 generates status management data using the similarity scores of each of the multiple questions and the comprehension score based on them (step S305). Finally, the control unit 12 sends a processing completion notification to the information terminal 20 (step S306).

[0132] When a request for evidence is sent from the confirmation terminal 30 (step S307), the control unit 12 sends status management data back to the confirmation terminal 30. As a result, the status confirmation screen Pm is displayed on the display unit 34 (step S308). When a request for evidence is sent from the information terminal 20 (step S307), the control unit 12 sends response certification data back to the information terminal 20. As a result, the certification screen Pz is displayed on the display unit 24 (step S308). The above processes have been explained in the order of the step numbers shown in Figure 17, but the order of the processes is not limited to this explanation, and it is also possible to change or omit some of the processes.

[0133] As described above, the management device 10 of this embodiment transmits analysis data that outputs a question for analyzing the subject's speech patterns, causing the information terminal 20 to output the question, and acquires information related to the subject's voice uttered in response to the question as response data via the sound pickup unit 27 of the information terminal 20. The management device 10 then receives prompt data including data, response data, and generation instruction information that generates an estimated answer to the question based on the data and response data, and obtains an estimated answer corresponding to the question from the language model 80m. Alternatively, the management device 10 receives prompt data including data, response data, and generation instruction information that generates an estimated answer to the question based on the data and response data, and obtains an estimated answer corresponding to the question from the language model 80m. With this configuration, the response data related to the subject's voice in response to the question is reflected in the estimated answer compared with the answer voice data, thereby improving the accuracy of the analysis of the answer to the question. For example, the similarity between the answer data corresponding to the question and the estimated answer can be determined with high accuracy. Incidentally, oral responses tend to reveal the depth of understanding and thought process more readily than written or manual input, making them suitable for assessing how well a person grasps the essence of the material. Furthermore, having participants answer orally is useful for evaluating their flexibility and presentation skills in situations involving psychological stress such as tension.

[0134] The control unit 12 may receive prompt data that includes a document, response data, a question based on the document, and generation instruction information that generates an estimated answer based on the document and response data for that question, and obtain an estimated answer corresponding to the question from the language model 80m. With this configuration, there is no need to prepare questions in advance, so immediate action can be taken even when the period from the determination of the document to the presentation of the question is short. For example, automating the generation of such questions is useful when daily reports are used as documents and knowledge and handover matters are confirmed when employees report to work the next day. A template including the document and generation instruction information may be stored in the storage unit 13, and the control unit 12 may be configured to incorporate the response data into the template to generate prompt data. This speeds up processing.

[0135] However, in situations where questions are presented to the same person over time and their understanding of the materials is checked each time, the response data generated once may be registered in the storage unit 13 and reused repeatedly. That is, the management system 100 may omit the processing corresponding to steps 108 to S113 in Figure 15 for the second and subsequent question-and-answer sessions, and in step S114, read the person's response data from the storage unit 13 to generate prompt data.

[0136] The embodiments described above are examples of management devices, management systems, management programs, and management methods, and the technical scope of the present invention is not limited to these embodiments. For example, the various screens illustrated in each figure are merely examples, and the arrangement of each element displayed on each screen, as well as the shapes, fonts, and expressions, can be changed as appropriate, and each element can be freely selected or omitted. For example, the appearance and character configuration of various virtual buttons are not limited to the examples in each figure and can be changed arbitrarily. Although each figure is formally illustrated in grayscale, colors are used appropriately for characters, figures, etc., on the actual screens. Note that each screen may differ somewhat depending on the type of browser and the models of the information terminal 20 and the confirmation terminal 30. The information processing unit 22 may activate the camera 26b and start recording when the record button Bk is pressed on the response acquisition screen Pc or the question screen Ps. The information processing unit 22 may also transmit a video consisting of audio and video to the management device 10, and the management device 10 may transmit the video to the confirmation terminal 30 for viewing.

[0137] Here, Figures 18 to 22 illustrate different configurations of the question screen Ps and confirmation screen Pk compared to Figures 6 to 10. Figures 18 to 22 show examples where three questions are presented to the subject. Components similar to those in Figures 6 to 10 are given the same reference numerals, and their explanations are omitted or simplified. Figure 18 is an example of the question screen Ps in standby mode, Figure 19 is an example of the question screen Ps in recording mode, and Figures 20 to 22 are examples of the confirmation screen Pk (question screen Ps in confirmation mode). The status unit Eb displays the progress of the questions. When the record button Bk is pressed on the question screen Ps in standby mode (see Figure 18), the record button Bk switches to the stop button Bt, recording begins via the sound pickup unit 27, and the time display tn indicating the recording time progresses. In other words, it switches to the question screen Ps in recording mode as shown in Figure 19.

[0138] When the stop button Bt is pressed on the recording question screen Ps (see Figure 19), for example, the stop button Bt switches to an image Bu indicating it cannot be operated, recording stops, and the progress of the time display tn stops. In other words, it switches to a confirmation screen Pk as shown in Figure 20. At this time, the information processing unit 22 converts the answer audio into text to generate the answer text, and displays it in the text section T in an editable state. Alternatively, when the stop button Bt is pressed on the recording question screen Ps (see Figure 19), the system may be configured so that the stop button Bt switches to a record button Bk, allowing for additional recording. The text section T in Figures 21 and 22 also displays the answer text in an editable state through the same process as described above. Then, when the confirm button Bo is pressed on the confirmation screen Pk as shown in Figure 22, it switches to a pre-submission screen (not shown) for, for example, final confirmation and submission of the answer text. However, it is also possible to omit the final confirmation on the pre-submission screen and configure the system so that the response text before and after the revision is sent each time from the confirmation screen Pk.

[0139] The management system 100 may be configured such that the answer text cannot be modified, and only the addition of audio to the answer to the question is possible. Specifically, the confirmation screen Pk (see Figures 9 and 11) may be configured so that it does not have a modification button Bs, but only a supplement button Bh as a virtual button related to modification, etc. This is because, in oral responses, even if a minor slip of the tongue remains in the answer text, it often does not affect the actual evaluation. Furthermore, considering special circumstances due to psychological stress, it is considered reasonable from the standpoint of fairly evaluating personnel to provide an opportunity for the subject to supplement their answer if they later realize that there is a omission in the content. However, if scoring is done using a uniform standard without considering whether or not supplementation has been made, it may lack fairness from a different perspective. Therefore, it is advisable to keep a log of the supplementation of the answer so that information indicating whether or not supplementation has been made can be confirmed. However, the management system 100 may also be configured so that the answer text cannot be modified or supplemented.

[0140] The above description illustrates an example where language model 80m is a large-scale language model stored on an external server, but is not limited to this. Language model 80m may be a trained model constructed by machine learning to generate estimated answers based on materials and questions (including rewriting of standard answers). Language model 80m may be constructed by supervised learning using a DNN, by semi-supervised learning or unsupervised learning, or by methods other than deep learning. Language model 80m may be stored in the management device 10. The above table information and database configuration is merely an example. Various table information and databases can be constructed by arbitrarily selecting and combining multiple pieces of information. The above configuration of each prompt data is merely an example, and the specific content, order of items, and constraints can be appropriately changed, added, or selected without departing from the spirit of this disclosure. [Explanation of Symbols]

[0141] 10 Management device, 11 Communication unit, 12 Control unit, 12a Information processing means, 12b Output processing means, 12c Acquisition processing means, 12d Response management means, 13 Storage unit, 13p Management program, 20 Information terminal, 21 Communication unit, 22 Information processing unit, 23 Storage unit, 24 Display unit, 25 Input unit, 26, 26a, 26b Camera, 27 Sound collection unit, 28 Notification unit, 30 Confirmation terminal, 31 Communication unit, 32 Confirmation processing unit, 33 Storage unit, 34 Display unit, 35 Input unit, 60 Operation terminal, 80 Model server, 80m Language model, 100 Management system, Ba OK button, Bo Confirm button, Bh Refill button, Bk Record button, Bs Correction button, D Similarity field, Ea, Eo Explanation field, H Evaluation field, Pc Response acquisition screen, Pd Introduction screen, Pk Confirmation screen, Pm Status confirmation screen, Ps Question screen, Pz Proof screen, Qc Question display section, Qm Question display section, Qn Question section, Qo Question section, Qr Question information section, R Back button, Sc Score field, Sd Remarks display section, Sm Type section, T Text section, T1 Main text section, T2 Supplementary text section, Ta Main text section, Tb Supplementary text section, Tm Answer display section, Tx Original text section, Tz Modified text section, U Facilitation section, W Operation section.

Claims

1. Output processing means that transmits analysis data to an information terminal to output questions for analyzing the subject's language use, and outputs the questions to the information terminal in at least one of voice and text; and transmits question data including questions about predetermined materials to the information terminal, and outputs one or more of the questions to the information terminal in at least one of voice and text; The acquisition processing means acquires information relating to the voice of the subject that responds to the question as response data via the sound pickup unit of the information terminal, and acquires information relating to the voice of the subject that responds to the question as response data, The system includes a response management means for determining the similarity between the response data corresponding to the aforementioned question and the estimated response, which is the estimated answer to the said question. The aforementioned response management means is A management device that takes prompt data, which includes the aforementioned materials, the response data, and generation instruction information that causes the language model to generate an estimated answer to the question based on the materials and the response data, as input to a language model, and obtains the estimated answer corresponding to the question from the language model.

2. Output processing means that transmits analysis data to an information terminal to output questions for analyzing the subject's language use, and outputs the questions to the information terminal in at least one of voice and text, and transmits question data including questions about predetermined materials to the information terminal, and outputs one or more of the questions to the information terminal in at least one of voice and text, The acquisition processing means acquires information relating to the voice of the subject that responds to the question as response data via the sound pickup unit of the information terminal, and acquires information relating to the voice of the subject that responds to the question as response data, The system includes a response management means for determining the similarity between the response data corresponding to the aforementioned question and the estimated response, which is the estimated answer to the said question. The aforementioned response management means is A management device that takes prompt data, which includes the aforementioned materials, the response data, and generation instruction information that causes a standard answer corresponding to the question to be rewritten based on the response data, as input to a language model, and obtains the estimated answer corresponding to the question from the language model.

3. It has a storage unit that stores a template including the aforementioned materials and the generation instruction information, The aforementioned response management means is The management device according to claim 1 or 2, which incorporates the response data into the template to generate the prompt data.

4. The acquisition processing means is The system transmits the response confirmation data for verifying the aforementioned response data to the information terminal, and displays a confirmation screen based on the response confirmation data on the display unit of the information terminal. The aforementioned confirmation screen is The management device according to claim 1 or 2, wherein the response text, which is text based on the aforementioned response data, can be modified.

5. The aforementioned confirmation screen is The number of characters that can be edited in the response text is limited to a predetermined range, and there is a button to accept voice additions to the answer to the question. The acquisition processing means is The management device according to claim 4, wherein after the press of the replenishment button, the sound pickup unit of the information terminal acquires information relating to the subject's response voice as part of the response data, and registers a log relating to the replenishment of the response.

6. The output processing means is Based on the aforementioned similarity, status management data indicating the subject's understanding of the aforementioned materials is transmitted to the confirmation terminal, and a status confirmation screen based on the status management data is displayed on the display unit of the confirmation terminal. The aforementioned status confirmation screen is, The management device according to claim 5, comprising information indicating whether or not the aforementioned modifications have been made, and information indicating whether or not the aforementioned response has been supplemented.

7. The acquisition processing means is The system transmits the response confirmation data for verifying the aforementioned response data to the information terminal, and displays a confirmation screen based on the response confirmation data on the display unit of the information terminal. The aforementioned confirmation screen is It has a supplement button that accepts voice supplements for the answers to the aforementioned questions, The acquisition processing means is The management device according to claim 1 or 2, wherein after the press of the replenishment button, the sound pickup unit of the information terminal acquires information relating to the subject's response voice as part of the response data, and registers a log relating to the replenishment of the response.

8. An information terminal equipped with a display unit, The system includes a management device that outputs analytical data for analyzing the subject's language use, and a question data including questions related to pre-determined materials, which is transmitted to the information terminal. The aforementioned information terminal is Based on the analysis data, the question is output in at least one of voice and text, and based on the question data, one or more questions are output in at least one of voice and text. The aforementioned control device is The information related to the voice of the subject who responded to the question is acquired as response data via the sound pickup unit of the aforementioned information terminal. The language model receives prompt data, which includes the aforementioned materials, the response data, and generation instruction information that causes the language model to generate an estimated answer to the question based on the materials and the response data, as input to the language model, and obtains an estimated answer, which is an estimated answer to the question, from the language model. Information relating to the voice of the subject's response to the aforementioned question is obtained as response data. A management system for determining the similarity between the answer data corresponding to the aforementioned question and the estimated answer corresponding to the aforementioned question.

9. An information terminal equipped with a display unit, The system includes a management device that outputs analytical data for analyzing the subject's language use, and a question data including questions related to pre-determined materials, which is transmitted to the information terminal. The aforementioned information terminal is Based on the analysis data, the question is output in at least one of voice and text, and based on the question data, one or more questions are output in at least one of voice and text. The aforementioned control device is The information related to the voice of the subject who responded to the question is acquired as response data via the sound pickup unit of the aforementioned information terminal. The language model takes prompt data, which includes the aforementioned materials, the response data, and generation instruction information that causes a standard answer corresponding to the question to be rewritten based on the response data, as input to the language model, and obtains an estimated answer, which is the estimated answer to the question, from the language model. Information relating to the voice of the subject's response to the aforementioned question is obtained as response data. A management system for determining the similarity between the answer data corresponding to the aforementioned question and the estimated answer corresponding to the aforementioned question.

10. Output processing means that transmits analysis data to an information terminal to output questions for analyzing the subject's language use, and outputs the questions to the information terminal in at least one of voice and text; transmits question data including questions about predetermined materials to the information terminal, and outputs one or more of the questions to the information terminal in at least one of voice and text; Acquisition processing means that, via the sound pickup unit of the information terminal, acquires information relating to the voice of the subject that responded to the question as response data, and acquires information relating to the voice of the subject that answered the question as response data. Furthermore, one or more processors are configured to function as a response management means for determining the similarity between the response data corresponding to the question and the estimated response which is the estimated answer to the question. The aforementioned response management means is A management program that takes prompt data, which includes the aforementioned materials, the response data, and generation instruction information that causes the language model to generate an estimated answer to the question based on the materials and the response data, as input to a language model, and obtains the estimated answer corresponding to the question from the language model.

11. Output processing means that transmits analysis data to an information terminal to output questions for analyzing the subject's language use, and outputs the questions to the information terminal in at least one of voice and text; transmits question data including questions about predetermined materials to the information terminal, and outputs one or more of the questions to the information terminal in at least one of voice and text; Acquisition processing means that, via the sound pickup unit of the information terminal, acquires information relating to the voice of the subject that responded to the question as response data, and acquires information relating to the voice of the subject that answered the question as response data. Furthermore, one or more processors are configured to function as a response management means for determining the similarity between the response data corresponding to the question and the estimated response which is the estimated answer to the question. The aforementioned response management means is A management program that takes prompt data, which includes the aforementioned materials, the response data, and generation instruction information that causes a standard answer corresponding to the question to be rewritten based on the response data, as input to a language model, and obtains the estimated answer corresponding to the question from the language model.

12. The steps include: transmitting analytical data that generates questions for analyzing the subject's language use to an information terminal, and having the information terminal output the questions in at least one of voice and text; The steps include: acquiring information related to the voice of the subject that responded to the question as response data via the sound pickup unit of the information terminal; A step of obtaining an estimated answer, which is an estimated answer to a question, from a language model, using prompt data, which includes predetermined materials, response data, and generation instruction information that causes the language model to generate an estimated answer to a question about the materials based on the materials and the response data, as input to a language model; The steps include: transmitting question data including the aforementioned questions to the information terminal, causing the information terminal to output one or more of the aforementioned questions in at least one of voice and text; The steps include: acquiring information related to the voice of the subject's response to the question as response data via the sound pickup unit of the information terminal; A management method in which one or more processors perform the steps of determining the similarity between the answer data corresponding to the question and the estimated answer corresponding to the question.

13. The steps include: transmitting analytical data that generates questions for analyzing the subject's language use to an information terminal, and having the information terminal output the questions in at least one of voice and text; The steps include: acquiring information related to the voice of the subject that responded to the question as response data via the sound pickup unit of the information terminal; A step of obtaining an estimated answer, which is an estimated answer to a question, from a language model, using prompt data, which includes predetermined materials, response data, and generation instruction information that causes a standard answer corresponding to a question about the materials to be rewritten based on the response data, as input to a language model; The steps include: transmitting question data including the aforementioned questions to the information terminal, causing the information terminal to output one or more of the aforementioned questions in at least one of voice and text; The steps include: acquiring information related to the voice of the subject's response to the question as response data via the sound pickup unit of the information terminal; A management method in which one or more processors perform the steps of determining the similarity between the answer data corresponding to the question and the estimated answer corresponding to the question.

Images