Skill test generation system, skill test generation server, skill test generation method, and skill test generation program

Abstract

The objective is to provide a skill test generation system, skill test generation server, skill test generation method, and skill test generation program that can maintain consistent quality of generated questions without requiring a great deal of time and effort in the skill test question generation process. [Solution] A skill test generation system comprising a generation unit that generates questions related to a skill test for job seekers, a quality assurance unit that evaluates the quality of the questions generated by the generation unit, and a control unit that causes the generation unit and the quality assurance unit to generate the skill test.

Description

Technical Field

[0001] The present disclosure relates to a skill test generation system, a skill test generation server, a skill test generation method, and a skill test generation program.

Background Art

[0002] In recent years, in job hunting and job matching services using the Internet (so-called job hunting sites), in order to objectively measure the vocational abilities and aptitudes of job seekers, skill tests (CBT: Computer Based Testing) implemented online using a web browser or the like are widely used. Employing companies use such test results, in addition to documentary information such as resumes and work experience documents, as materials for judging whether the practical skills possessed by job seekers meet the standards required by the company.

[0003] In a conventional general skill test providing system, templates of a large number of pre-created question data are stored in a database (question library) for each category such as programming languages, accounting knowledge, language proficiency, and logical thinking ability. When an employing company selects test items according to the job types and requirements to be recruited, the system extracts questions that meet the conditions from the database randomly or according to a predetermined logic, constructs a test set, and presents it to the terminal of the job seeker.

[0004] In the above system, the generation process of the question data registered in the database basically depends on manual work. Specifically, a question writer with knowledge of the target specialized field analyzes general job requirements and technological trends to create question texts, options, correct answers, and explanatory texts. In general, the created data is double-checked by proofreaders and supervisors to ensure the absence of typos and the accuracy of the content, and then released after being registered in the system.

[0005] Furthermore, quality control of the created tests is also performed manually. For example, a process may be included in which a third party (tester) actually takes the pre-release test and measures the accuracy rate and response time to confirm whether the difficulty level of the questions is as expected and whether any inappropriate questions are included. In this way, in conventional technologies, human judgment and work are involved in many processes, from the planning and creation of questions to verification and implementation into the system.

[0006] Patent Document 1 describes a job recruitment and job seeker support system and method that enables the rapid and simple selection of suitable personnel based on objective and rational selection criteria. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2008-176666 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, generating questions for conventional skill tests required a significant amount of time and effort from human hands. Because it was done manually, the quality of the generated questions depended on the abilities of the person doing the generating, making it impossible to maintain a consistent level of quality.

[0009] Therefore, the purpose of this disclosure is to provide a skill test generation system, a skill test generation server, a skill test generation method, and a skill test generation program that can maintain uniform quality of generated questions without requiring a great deal of time and effort in generating skill test questions. [Means for solving the problem]

[0010] A skill test generation system according to a first aspect of this disclosure comprises a generation unit that generates questions relating to a skill test which is a test for job seekers; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; and a control unit that causes the generation unit and the quality assurance unit to generate the skill test.

[0011] A skill test generation server according to a second aspect of this disclosure comprises a generation unit that generates questions relating to a skill test which is a test for job seekers; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; and a control unit that causes the generation unit and the quality assurance unit to generate the skill test.

[0012] A skill test generation method according to a third aspect of this disclosure includes a generation step of generating questions relating to a skill test which is a test for job seekers, a quality assurance step of evaluating the quality of the questions generated by the generation step, and a control step of causing the generation step and the quality assurance step to generate the skill test.

[0013] A skill test generation program according to a fourth aspect of this disclosure is characterized by comprising: a generation unit that generates questions relating to a skill test which is a test for job seekers; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; and a computer functioning as a control unit that causes the generation unit and the quality assurance unit to generate the skill test. [Effects of the Invention]

[0014] According to this disclosure, it is possible to maintain consistent quality in the generation of skill test questions without requiring a great deal of time and effort. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram showing the overall configuration of the skill test generation system 1 according to this embodiment. [Figure 2] This is a functional block diagram of the skill test generation server 100 according to this embodiment. [Figure 3] FIG. is a basic flowchart of the skill test generation system 1 according to the present embodiment. [Figure 4] FIG. is a more detailed flowchart of the generation unit 131 according to the present embodiment. [Figure 5] FIG. is a more detailed flowchart of the quality assurance unit 132 according to the present embodiment. [Figure 6] FIG. is a configuration example diagram of the keyword data table 600 according to the present embodiment. [Figure 7] FIG. is a configuration example diagram of the natural language text data table 610 according to the present embodiment. [Figure 8] FIG. is a configuration example diagram of the job offer data table 620 according to the present embodiment. [Figure 9] FIG. is a configuration example diagram of the setting information table 630 referred to by the wrong answer specialization unit 1313 according to the present embodiment. [Figure 10] FIG. is a configuration example diagram of the setting information table 640 referred to by the explanation specialization unit 1314 according to the present embodiment. [Figure 11] FIG. is a configuration example diagram of the setting information table 650 referred to by the quality check unit 1322 and the correction unit 1323 according to the present embodiment. [Figure 12] FIG. is a functional block diagram of the skill test generation server 100 showing another example according to the present embodiment. [Figure 13] FIG. is a block diagram showing a hardware configuration example of the computer 900 functioning as the skill test generation server 100 and the job seeker terminal 200 according to the present embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0016] Hereinafter, embodiments for implementing the present invention will be described in detail. Note that the present invention is not limited to the following embodiments, and various modifications are possible within the scope of the gist thereof. In addition, the drawings used in the following description schematically show the characteristic parts for the sake of easy understanding of the features of the present invention, and the dimensional ratios of each component are not necessarily the same as the actual ones.

[0017] FIG. 1 is a schematic diagram showing the overall configuration of a skill test generation system 1 according to this embodiment. The skill test generation system 1 generates, manages, and provides questions related to a skill test for measuring the specialized skills and abilities of job seekers. The skill test generation system 1 includes a skill test generation server 100 and a plurality of job seeker terminals 200. The skill test generation server 100 and each job seeker terminal 200 are communicably connected to each other via a network NW.

[0018] The network NW is a communication network that enables information communication between devices, and is constituted by, for example, the Internet, WAN (Wide Area Network), LAN (Local Area Network), dedicated line, mobile communication network (for example, 4G, 5G, LTE), or a combination thereof. As the communication protocol, for example, TCP / IP (Transmission Control Protocol / Internet Protocol) or the like is used, but it is not limited thereto.

[0019] <able> The skill test generation server 100 is an information processing device that forms the core of the skill test generation system 1, automatically generates questions for the skill test, guarantees the quality thereof, and further analyzes the test results to improve the question generation process. The skill test generation server 100 can be realized, for example, as a virtual server constructed in a cloud computing environment, a physical server computer, or a distributed system in which a plurality of computers operate in cooperation.

[0020] The job seeker terminal 200 is a terminal device used by job seekers to take skill tests provided by the skill test generation server 100. The job seeker terminal 200 is, for example, a smartphone, tablet, personal computer (PC), notebook PC, or any other device with information processing capabilities. The job seeker terminal 200 typically accesses the skill test generation server 100 via a web browser or dedicated application software to view questions, input answers, and check results.

[0021] Figure 2 is a functional block diagram of the skill test generation server 100 according to this embodiment. The skill test generation server 100 comprises a communication unit 110, a storage unit 120, and a processing unit 130. The skill test generation server 100 is realized through the cooperation of hardware implemented by the computer 900 and a skill test generation program stored in the storage unit 120.

[0022] The communication unit 110 has the function of performing data communication with the job seeker terminal 200 and external systems via the network NW. For example, it can send generated skill test question data to the job seeker terminal 200 and receive answer data from the job seeker terminal 200.

[0023] The memory unit 120 stores programs and data necessary for processing by the processing unit 130. In addition to the skill test generation program, the memory unit 120 stores data such as basic data, setting information, generated data, quality data, and feedback data.

[0024] The basic data may be the information that forms the basis for the generation unit 131 when generating questions. The basic data may include a keyword data table 600 that associates "keywords" indicating specific skills or knowledge with the "related categories" to which they belong, a natural language text data table 610 that associates "text content" such as technical documents and explanatory articles with their "source / type", and a job posting data table 620 that associates the "job title" being recruited, the required "essential skills", and the "job description".

[0025] The configuration information may be information that defines the rules and templates for each functional unit to perform its processing. For example, the configuration information may include a configuration information table 630 that defines the principles of error design and types of cognitive traps used by the error specialization unit 1313, a configuration information table 640 that defines the components and tone and manner of explanations used by the explanation specialization unit 1314, and a configuration information table 650 that defines the check items and correction rules used by the quality check unit 1322.

[0026] The generated data may be intermediate and final products generated during the question generation process. The generated data may include, for example, a draft question generated by the basic generation unit 1312, a candidate incorrect answer generated by the incorrect answer specialization unit 1313, an explanatory text generated by the explanatory text specialization unit 1314, and question data generated by integrating these.

[0027] The quality data may be data related to processing by the quality assurance unit 132. The quality data may include, for example, the judgment result (pass / fail) by the quality check unit 1322, the reason for failure, and the correction history by the correction unit 1323.

[0028] The feedback data may be data related to processing by the feedback unit 133. The feedback data may include, for example, answer data collected from the job seeker terminal 200, statistical indicators such as the difficulty level and discrimination index for each question calculated by the statistical analysis unit 1331, evaluation results by the effectiveness evaluation unit 1332, and updated generation parameters.

[0029] The processing unit 130 controls the operation of the entire skill test generation server 100 by reading and executing the skill test generation program from the storage unit 120. The processing unit 130 oversees a series of processes, including generating questions related to the skill test, which is a test for job applicants, and evaluating their quality. The processing unit 130 has as its main functional units a control unit 1300, a generation unit 131, a quality assurance unit 132, and a feedback unit 133.

[0030] The control unit 1300 causes the generation unit 131 and the quality assurance unit 132 to generate skill tests. The control unit 1300 may also cause the feedback unit 133 to update the parameters that the generation unit 131 uses to generate questions based on the results of the skill tests.

[0031] The generation unit 131 generates questions related to a skills test for job seekers. In this embodiment, the generation unit 131 does not generate questions with a single function, but rather features a configuration in which multiple specialized functional units work together to generate higher quality questions in stages. Specifically, the generation unit 131 includes a generation control unit 1311, a basic generation unit 1312, an incorrect answer specialized unit 1313, and an explanation specialized unit 1314. Each function of the generation unit 131 may be implemented, for example, by a large-scale language model (LLM) that is given a specific role using prompt engineering techniques.

[0032] The generation control unit 1311 has the function of controlling the operation of the entire generation unit 131. The generation control unit 1311 acquires basic data (e.g., job type, required skills, related technical documents, etc.) that will be used as the basis for generating questions from the memory unit 120, and passes appropriate instructions and data to the basic generation unit 1312, the incorrect answer specialization unit 1313, and the explanation specialization unit 1314 to execute processing. Finally, the generation control unit 1311 integrates the outputs from each unit (draft questions, candidate incorrect answers, and explanation text) and formats them into a single completed question data.

[0033] The basic generation unit 1312 has the function of creating the core part of the question. Based on the input basic data, the basic generation unit 1312 generates a "draft question" that includes the question statement (scenario or question) which forms the framework of the question, and the intent of the question which indicates what it wants to measure. For example, based on the instruction to measure the skills of a "front-end engineer" regarding "React performance optimization," the basic generation unit 1312 may generate a problem scenario including specific code snippets and the intent of the question which knowledge or judgment ability of the job seeker the question is intended to test, as text.

[0034] The error-specialization unit 1313 intentionally designs mistakes that test-takers are likely to make from a psychological and educational technology perspective, generating candidate "engaging error answers" that are not merely mistakes. The error-specialization unit 1313 operates based on principles defined in the setting information table 630. Specifically, it receives the draft question and correct answer generated by the basic generation unit 1312 as input and generates candidate error answers based on the following principles. Note that the basic generation unit 1312 may also generate candidate error answers directly from the basic data. The principles may include principles of error design and types of cognitive traps.

[0035] The principle of incorrect answer design may be such that the incorrect answer specialization unit 1313 designs incorrect answers according to at least one of the following principles: "options that are partially correct but lack an important perspective," "options that are generally correct but inappropriate in the context of this question," and "options that are based on typical misconceptions that beginners are prone to."

[0036] The error-specialization unit 1313 further sets cognitive traps that exploit the test-taker's thinking habits. For example, it may consider traps of the following types to enrich the variety of incorrect answers: "superficial similarity" (making the test-taker judge based only on the superficial features of the situation), "simplification" (making the test-taker oversimplify a complex problem), "short-term perspective" (making the test-taker judge based on short-term benefits without considering long-term effects), and "partial optimization" (making the test-taker choose a partial solution without considering overall optimization). Based on these principles, the error-specialization unit 1313 may generate multiple candidate incorrect answers along with the reasons why each answer is wrong.

[0037] The explanation specialization unit 1314 generates detailed explanations to maximize the learning effect of test takers. The explanation specialization unit 1314 not only shows the correct and incorrect answers, but also explains in depth from a practical perspective why the correct answer is correct and why the other options are incorrect. The explanation specialization unit 1314 operates based on the components defined in the configuration information table 640. The explanation specialization unit 1314 receives the draft question, the correct answer, and the candidate incorrect answers as input, and may generate an explanation that includes the following elements. The explanation specialization unit 1314 may also generate an explanation directly from the basic data. The elements may include the components of the explanation and the tone and manner.

[0038] The components of the explanation may include information of high value as learning content, such as: "the essential reason for the correct answer," "the reason why each incorrect answer is inappropriate (not just finding mistakes, but pointing out which perspectives are lacking)," "points for practical application," "related theories and frameworks," and "frequently occurring failure patterns and how to avoid them."

[0039] The tone and manner should also be controlled in the style of explanatory text. For example, as an "expert in learning design for corporate training," the use of technical jargon should be kept to a minimum, while the text should be written in a tone that balances reliability and clarity, including practical advice.

[0040] The quality assurance unit 132 is a module that evaluates whether the questions generated by the generation unit 131 have appropriate quality as a skill measurement tool, and makes corrections or adjustments as necessary. This ensures that the quality of the automatically generated questions is maintained at a certain level or higher. The quality assurance unit 132 includes a quality control unit 1321, a quality check unit 1322, a correction unit 1323, and a format adjustment unit 1324.

[0041] The quality control unit 1321 has the function of controlling the operation of the entire quality assurance unit 132. The quality control unit 1321 receives question data from the generation unit 131 and first instructs the quality check unit 1322 to make a quality judgment. Based on the judgment result, if the question is unsatisfactory, it instructs the correction unit 1323 to make corrections, and if it is satisfactory (or after correction), it instructs the format adjustment unit 1324 to make final adjustments, thereby managing the flowchart of the question data.

[0042] The quality check unit 1322 has the function of automatically evaluating the quality of a question based on predetermined objective quality standards and determining whether it is "pass" or "fail." The quality check unit 1322 performs the evaluation based on the check items defined in the setting information table 650. The quality check unit 1322 receives the question data, the measurement target of the question, and the target audience information as input and checks the following items. The items may include measurement accuracy, clarity of correct answer, functionality of the choices, realism, and judgment criteria.

[0043] Measurement accuracy may be measured by whether the questions purely measure only the target skills set for measurement. Furthermore, measurement accuracy may also be measured by whether other unrelated knowledge or abilities influence the answers.

[0044] The clarity of a correct answer may be defined as whether the correct answer to a question is uniquely determined without any room for interpretation. Alternatively, the clarity of a correct answer may be defined as whether the question is not ambiguous in its wording or set up in a way that allows for multiple interpretations.

[0045] The functionality of the answer choices may be measured by whether each choice (correct / incorrect) reflects a different set of judgment criteria or thought processes, and functions as a meaningful choice. The functionality of the answer choices may also be measured by whether there are no obviously inappropriate choices or choices that are substantially the same as other choices.

[0046] Realism can mean whether the scenarios and situations presented in the questions are realistic and could actually occur in practical work. It can also mean whether the scenarios are not unrealistic and fail to accurately measure the test-taker's practical abilities.

[0047] The judgment criteria may be whether the judgment required in the question is consistent with the criteria for good decision-making in practice. The judgment criteria may be such that the quality check unit 1322 evaluates these items and, if there are items that do not meet the criteria, determines them as "failed" and outputs the reason.

[0048] This section explains the "check content" in the quality evaluation performed by the Quality Check Department 1322. The Quality Check Department 1322 is configured to check questions from three main perspectives: "Form (automatic correction)," "Content (suggestion for correction)," and "Practical quality."

[0049] First, the checks related to "format (automatic correction)" are criteria for the format of the questions and are items that can be judged and corrected mechanically based on rules. Specifically, the quality check unit 1322 performs a character limit check to confirm whether the question scenario or problem statement is within an appropriate length (for example, within the range of 200 to 250 characters). It also performs a sentence ending consistency check to confirm whether the end of each question is consistent with the specified style (for example, target endings such as "~ or." or "~ select.").

[0050] Next, the check items related to "Content (Proposal for Revision)" are criteria concerning the quality and validity of the questions, and are items that require semantic analysis. Quality Check Department 1322 evaluates the following items. These items include measurement accuracy, clarity of correct answer, and functionality of the answer choices.

[0051] Measurement accuracy can be evaluated by assessing whether the questions purely measure only the target skills they are intended to measure, that is, whether they include unnecessary background knowledge or are merely reading comprehension tests.

[0052] The clarity of the correct answer can be evaluated by assessing whether the correct answer is objectively unique and leaves no room for interpretation. In terms of clarity of the correct answer, ambiguous questions that even experts disagree on may be judged as inappropriate.

[0053] The functionality of the answer choices evaluates whether each choice (especially the incorrect answer) functions effectively to identify the test-taker's abilities. The functionality of the answer choices may involve checking for obviously unnatural choices or overlapping content with other choices, and determining whether each choice reflects a different set of judgment criteria or patterns of misunderstanding.

[0054] Furthermore, the checks related to "practical quality" include items to confirm whether the questions are relevant to the actual work environment. The quality check department 1322 may perform a "realism" check to evaluate whether the scenario set in the question is a realistic situation that could occur in an actual work environment (whether it is realistic), and a "judgment criteria" check to evaluate whether the judgment required for the correct answer in the question matches the criteria for good decision-making in practice.

[0055] The quality check unit 1322 may comprehensively evaluate the questions based on these check contents and, if the criteria are not met, output a judgment result that includes specific correction instructions (for example, "deviation from measurement target" or "option C is not functioning").

[0056] The correction unit 1323 automatically corrects questions that have been judged as "failed" by the quality check unit 1322, based on the reasons for failure pointed out. The correction unit 1323 may perform multiple correction processes depending on the reasons for failure, rather than a single correction logic. For example, the correction unit 1323 may perform at least one of the following correction processes. The correction process may include unifying sentence-ending expressions, regenerating incorrect answers, and reinforcing explanatory texts. The quality check unit 1322 may then determine whether the questions corrected by the correction unit 1323 are pass or fail.

[0057] The standardization of sentence-ending expressions is permitted if the sentence-ending expressions such as "~ka." or "~shinasai." are not consistent throughout the entire question; in such cases, you may standardize them to the specified format.

[0058] If the quality check unit 1322 indicates that the "option functionality" is low (e.g., the incorrect answer is not appealing), the incorrect answer specialization unit 1313 may be instructed again to generate new incorrect answer candidates using different principles or traps, and replace the original incorrect answer.

[0059] If the quality check unit 1322 points out that the "clarity of the correct answer" is low, the explanation specialization unit 1314 may be instructed to describe in more detail the basis for the correct answer and the reasons why the incorrect answer is inappropriate, thereby reinforcing the explanation.

[0060] The formatting adjustment unit 1324 has the function of formatting questions that have passed the quality check, or questions that have been corrected by the correction unit 1323, into the final output format. The formatting adjustment unit 1324 may, for example, adjust the total number of characters in the question to a specified range (e.g., 200-250 characters), add HTML tags or Markdown notation for display on a web page, or combine each element such as the question, answer choices, and explanations into structured data such as JSON (JavaScript Object Notation) format.

[0061] Next, the "final format" processed by the format adjustment unit 1324 will be described. The final format is the data format in which the completed question data, after going through the quality assurance process, is ultimately output from the system. The format adjustment unit 1324 may have the function of formatting the question data that has passed the quality check (or the question data that has been corrected by the correction unit 1323) into a final format suitable for display on the job seeker terminal 200 and for linking with external systems.

[0062] Specifically, the formatting adjustment unit 1324 converts the question text, answer choice list, correct answer data, explanation text, and metadata (difficulty level, tags, etc.) into a structured data format such as JSON (JavaScript Object Notation) or XML. The formatting adjustment unit 1324 may also perform processing to add HTML tags or Markdown notation so that the data is displayed appropriately in the web browser of the job seeker terminal 200. The formatting adjustment unit 1324 may also unify line break codes and add decorative information for emphasis.

[0063] Furthermore, the formatting adjustment unit 1324 may perform adjustments such as unifying character codes to prevent garbled characters in different environments, or adding display layout information for the question text. By formatting the test into this final format, the generated skill test becomes immediately ready for distribution, enabling consistent display on client applications such as job seeker terminals 200.

[0064] The feedback unit 133 analyzes the results (answer data) of the skill test after it has been used by job seekers, and builds a feedback loop to improve the quality of the questions and the generation process. This allows the feedback unit 133 to self-improve with each iteration of the test, enabling it to generate more accurate skill tests. The feedback unit 133 includes a statistical analysis unit 1331 and an effectiveness evaluation unit 1332. The answer data collected by the feedback unit 133 is stored in the memory unit 120.

[0065] The statistical analysis unit 1331 has the function of quantitatively analyzing the performance of each question using statistical test theory based on the answer data of a large number of job seekers stored in the memory unit 120. Examples of statistical test theories used include Item Response Theory (IRT) and Classical Test Theory (CTT).

[0066] Item Response Theory (IRT) is a theory that focuses on the response patterns of test takers to individual questions (items) rather than the total test score, and uses probabilistic models to estimate the test taker's ability level and the characteristics of the questions.

[0067] The IRT (Individualized Test Response) expresses the relationship between a test-taker's potential ability score (θ) and the probability that the test-taker will answer a particular question correctly using a mathematical model (item characteristic curve) such as a logistic curve. The mathematical model incorporates item parameters as question-specific characteristic values, such as "difficulty (b parameter)" which represents the difficulty of the question, "discriminative power (a parameter)" which represents the steepness of the change in the probability of answering correctly due to ability differences, and "guessing (c parameter)" which represents the probability that a low-ability test-taker will answer correctly by chance.

[0068] Because IRT can separate and handle the test-taker's ability scores from the parameters of the questions, it can be applied to comparing (equalizing) the abilities of test-takers who have taken different sets of questions, and to adaptive tests that optimize the questions to suit the abilities of individual test-takers.

[0069] Classical Test Theory (CTT) is a statistical theory that uses test scores (observed scores) as the basis for analysis, and has been a standard method used in the fields of psychometrics and educational measurement. In classical test theory, observed scores are defined as the sum of the test-taker's invariant "true score" and the "measurement error" that fluctuates randomly with each measurement.

[0070] Analysis based on CTT calculates reliability coefficients using the variance and covariance of the entire test, and evaluates the properties of the test and questions using indicators such as the correct answer rate (pass rate) for each question and the correlation (discriminative index) between the score of a question and the total test score. It is characterized by its relatively simple calculation process and the use of an intuitively understandable indicator, the total score (raw score). Based on these theories, the statistical analysis unit 1331 calculates indicators for each question, such as the following. The indicators include the difficulty level of the question and the discriminative index (discriminative power) of the question.

[0071] The difficulty level of a question may be an indicator of the level of ability required to answer that question correctly. The difficulty level of a question may be calculated by considering not only the correct answer rate but also the ability level of the test taker.

[0072] The discriminative index (discriminative power) of a question may be an indicator of how well the question can distinguish between highly skilled and less skilled test-takers. The discriminative index (discriminative power) of a question may be such that a question with a higher discriminative index performs better as a measurement tool.

[0073] The effectiveness evaluation unit 1332 evaluates the effectiveness of each question based on indicators (difficulty level, discrimination index, etc.) calculated by the statistical analysis unit 1331. For example, the effectiveness evaluation unit 1332 identifies questions with a significantly low discrimination index, or questions that are extremely difficult or too easy for the intended target group, as "questions that require improvement."

[0074] Furthermore, the effectiveness evaluation unit 1332 feeds the evaluation results back into the "generation parameters" that the generation unit 131 uses to generate questions. For example, if it is found that "this type of scenario setting tends to result in a low discrimination index," it adjusts the prompts used by the basic generation unit 1312 to suppress the generation of such scenarios. Also, if it is found that a particular incorrect answer pattern is not selected by anyone (i.e., does not function as an incorrect answer), it updates the parameters by changing the weighting of the incorrect answer designs referenced by the incorrect answer specialization unit 1313, thereby reducing the frequency of use of that pattern.

[0075] The job seeker terminal 200 is a client device used by job seekers to take skills tests. As mentioned above, this includes smartphones and PCs. The job seeker terminal 200 also has a hardware configuration similar to that of the computer 900.

[0076] According to this embodiment, a skill test generation system 1 is provided that can efficiently and continuously generate high-quality skill test questions by linking three main functions: question generation, quality assurance, and feedback. In particular, by configuring the generation unit 131 and the quality assurance unit 132 with a combination of multiple specialized functional units, it is possible to automatically generate questions that are closer to, or even better than, questions created by humans.

[0077] Figure 3 shows a basic flowchart of the skill test generation system 1 according to this embodiment. This flowchart is realized when the control unit 1300 of the skill test generation server 100 executes a program stored in the storage unit 120. First, the control unit 1300 (or processing unit 130) of the skill test generation server 100 starts the skill test generation process based on instructions from an administrator or the like.

[0078] When processing begins, the control unit 1300 first instructs the generation unit 131 to generate questions for a skills test for job seekers (step S102). This instruction may include information such as the number of questions to be generated, the theme, difficulty level, and target job types.

[0079] Upon receiving this instruction, the generation unit 131 uses its internal logic and various data stored in the memory unit 120 to execute the question generation process. The generated questions consist of a problem statement, multiple answer choices (including at least one correct answer and multiple incorrect answers), and a dataset containing information about the correct answer. Once the generation unit 131 has generated the specified number of questions, it outputs the results to the control unit 1300.

[0080] Next, the control unit 1300 inputs the question data generated by the generation unit 131 to the quality assurance unit 132 and instructs it to evaluate the quality of the questions (step S104). The quality assurance unit 132 evaluates each input question based on predetermined quality standards.

[0081] These quality standards include clarity of the questions, appropriateness of difficulty level, consistency with the skills to be measured, and validity of the answer choices. The quality assurance unit 132 determines whether each question meets the quality standards and reports the result to the control unit 1300. If a question is determined not to meet the quality standards, the reason and proposed corrections may also be reported.

[0082] Next, the control unit 1300 integrates the results of the question generation process by the generation unit 131 (step S102) and the quality evaluation process by the quality assurance unit 132 (step S104) to generate the final skill test (step S106).

[0083] Specifically, the control unit 1300 selects a set of questions that the quality assurance unit 132 has determined to meet the quality standards, and combines them to form a single skill test. The control unit 1300 may randomize the order of the questions or adjust their arrangement according to their difficulty level. The generated skill test data is stored in the storage unit 120 and becomes available for distribution upon request from the job seeker terminal 200. Through this series of processes, a high-quality skill test for objectively measuring the skills of job seekers is efficiently generated.

[0084] Figure 4 shows a more detailed flowchart of the generation unit 131 according to this embodiment. The flowchart is mainly executed inside the generation unit 131 of the skill test generation server 100. First, when the processing unit 130 starts the skill test generation process (step S201), the generation control unit 1311 of the generation unit 131 controls the details of the process.

[0085] The generation control unit 1311 acquires the basic data necessary for generating the skill test (step S202). The basic data includes, for example, the target job type for the skill test, the required skill level, and the theme of the questions. Based on this information, the generation control unit 1311 extracts information that will be used to generate the questions from the keyword data table 600, the natural language text data table 610, and the job posting data table 620 stored in the memory unit 120. For example, if the target job type is "data scientist," the generation control unit 1311 identifies "Python" and "machine learning" as required skills from the job posting data table 620 and acquires technical documents and explanatory articles related to these keywords from the natural language text data table 610.

[0086] Next, the generation control unit 1311 inputs the acquired basic data into the basic generation unit 1312 to generate a draft question that will form the framework of the question (step S203). The basic generation unit 1312 is configured, for example, using a large-scale language model (LLM), and interprets the input basic data (text data and keywords) to generate a question statement that includes specific scenarios and tasks for testing particular skills, as well as the intent behind the question, which indicates what the question aims to measure. The generated draft question is the core part of the problem, and does not yet have a detailed explanation.

[0087] Next, the generation control unit 1311 inputs the draft question generated by the basic generation unit 1312 to the incorrect answer specialization unit 1313 to generate incorrect answer candidates (step S204). The incorrect answer specialization unit 1313 understands the content and intent of the draft question and generates multiple high-quality incorrect answers based on psychological and educational technology insights, such as typical errors that test-takers are likely to make, options that are partially correct but not entirely correct, or options that people tend to choose when they misunderstand the situation.

[0088] The generation control unit 1311 inputs the draft question, the correct answer (which may be generated by the basic generation unit 1312 along with the draft question, or may be defined separately), and the incorrect answer candidates generated by the incorrect answer specialization unit 1313 into the explanation specialization unit 1314, causing it to generate a detailed explanation (step S205). The explanation specialization unit 1314 generates a text that not only explains the rationale for why a particular option is correct, but also logically explains why each incorrect answer is wrong. Furthermore, the explanation specialization unit 1314 creates an explanation that enhances the learning effect for test takers by including practical application examples and related knowledge.

[0089] Finally, the generation control unit 1311 integrates the draft question (question text) generated by the basic generation unit 1312, the candidate incorrect answers generated by the incorrect answer specialization unit 1313, and the explanation text generated by the explanation specialization unit 1314, along with the correct answer information, and formats them into a single completed question data (step S206). This integrated question data becomes the final output of the generation unit 131.

[0090] After this, the process moves to the quality evaluation step (step S207) by the quality assurance unit 132. If the quality evaluation results in the quality meeting the predetermined standard (step S208: Yes) and the required number of questions are available (step S209: No), the processing unit 130 instructs the generation control unit 1311 to repeat the question generation process from step S202.

[0091] Figure 5 shows a more detailed flowchart of the quality assurance unit 132 according to this embodiment. The flowchart is mainly executed within the quality assurance unit 132 of the skill test generation server 100. The questions generated by the generation unit 131 are passed to the quality control unit 1321 of the quality assurance unit 132.

[0092] The quality control unit 1321 first sends the input question to the quality check unit 1322 (step S301).

[0093] The quality check unit 1322 automatically evaluates the quality of the questions based on pre-set quality standards and makes a pass or fail judgment (step S302). These quality standards include measurement accuracy, clarity of correct answers, functionality of the answer choices, realism, and judgment criteria, which will be described later.

[0094] If the quality check unit 1322 determines (step S303) that a question is unacceptable (step S303: No), the quality control unit 1321 sends the unacceptable question, along with data indicating the reason for the unacceptability, to the correction unit 1323 (step S304). The correction unit 1323 then performs a correction process on the question according to the reason for the unacceptability.

[0095] This correction process includes at least three patterns. The first pattern is that, if the reason for failure is "inconsistency in writing style or expression," the correction unit 1323 may perform a process to unify sentence-ending expressions (for example, "~ka.", "~dearu.", etc.) or the notation of technical terms between multiple questions or within a single question.

[0096] The second pattern is that, if the reason for failure is "low quality of incorrect answer" or "insufficient functionality of the answer choices," the correction unit 1323 may regenerate a more attractive and discriminable incorrect answer using the same logic as the incorrect answer specialization unit 1313, or a different logic.

[0097] The third pattern is that, if the reason for failure is "insufficient explanation," the correction unit 1323 may use the same logic as the explanation specialization unit 1314 to reinforce the explanation text, such as clarifying the basis for the correct answer or the reason for the incorrect answer, or adding specific examples.

[0098] The correction unit 1323 performs at least one or more of these correction processes, generates the corrected question, and sends it back to the quality control unit 1321. The corrected question may also be returned to the quality check process (starting from step S301) by the quality check unit 1322.

[0099] On the other hand, if the question is judged to be acceptable in step S303 (step S303: Yes), or if the correction by the correction unit 1323 is completed in step S304, the quality control unit 1321 inputs the question into the format adjustment unit 1324 (step S305).

[0100] The formatting adjustment unit 1324 formats the question data into a final format (e.g., JSON or XML format) that can be displayed and executed on the job seeker terminal 200 (step S306). This formatting process includes unifying character codes, tagging, and adding display layout information.

[0101] Finally, the question data, which has been finalized by the format adjustment unit 1324, is passed to the processing unit 130 as the final output of the quality assurance unit 132 by the quality control unit 1321 (step S307). This completes the question, which is guaranteed to be of quality and is formatted for use by the system.

[0102] According to one embodiment of the present invention, the feedback process is performed after the skill test has actually been used by the job seeker. First, when the job seeker takes the skill test using the job seeker terminal 200, the answer data for each question (selected option, answer time, etc.) is transmitted to the skill test generation server 100 via the network NW. The server's communication unit 110 receives this data and stores the answer data in the storage unit 120 in a format that can be used for processing by the feedback unit 133.

[0103] Next, the statistical analysis unit 1331 of the feedback unit 133 performs a statistical analysis based on a sufficient amount of answer data stored in the memory unit 120. Specifically, the statistical analysis unit 1331 calculates evaluation indicators for each question using statistical test theories such as Item Response Theory (IRT) and Classical Test Theory (CTT).

[0104] The calculated metrics include the difficulty level of the questions (correct answer rate or difficulty parameter b in the IRT), the discrimination index of the questions (an indicator showing how accurately the test-taker's abilities can be identified, such as the discrimination parameter a in the IRT), and the selection rate for each option.

[0105] Next, the effectiveness evaluation unit 1332 evaluates the effectiveness of each question based on various indicators calculated by the statistical analysis unit 1331. The effectiveness evaluation unit 1332 identifies questions that require improvement, such as questions that are too difficult or too easy, questions with a low (or negative) discrimination index, or questions where a particular incorrect answer is not selected at all or is selected more often than the correct answer.

[0106] The effectiveness evaluation unit 1332 then generates feedback information for the generation parameters used when the generation unit 131 generates questions, based on the evaluation results. For example, if the evaluation result indicates that "questions containing certain keywords tend to be too difficult," the evaluation unit provides feedback to the basic generation unit 1312 to adjust the prompts and constraints when generating draft questions using those keywords.

[0107] Furthermore, if the evaluation result indicates that "a certain type of incorrect answer is not functioning," the incorrect answer specialization unit 1313 generates information to modify the logic and parameters that generate that type of incorrect answer. This feedback information is passed to the generation unit 131 via the processing unit 130 and reflected in the next question generation cycle. This feedback loop allows the skill test generation system 1 to continuously improve the quality of questions by making self-improvements based on actual test data.

[0108] According to one embodiment of the present invention, when the error-specializing unit 1313 starts processing, it receives as input information a draft question (question text) generated by the basic generation unit 1312, the correct answer to that question, and the target that the question is trying to measure (measurement target).

[0109] The error-specialization unit 1313 aims, as a principle of error design, to generate plausible errors that not only are simply wrong but also lure test-takers into cognitive traps. This principle includes at least three types.

[0110] The first type is the "partially correct option," which is correct in some respects but insufficient or inappropriate in the core conditions or context that the question is asking about.

[0111] The second type is "an option that is generally correct but inappropriate in the context of the question," which is correct in terms of general knowledge but cannot be applied under the specific circumstances or constraints presented in the question.

[0112] The third type is "options that are easily chosen due to typical misunderstandings," which are choices based on common misconceptions or mistakes that beginners often fall into.

[0113] Based on these principles, the error-specialization unit 1313 generates incorrect answers by considering specific types of cognitive traps. For example, the "surface similarity" trap is a method that generates options that are superficially similar to keywords or expressions in the question but have different meanings.

[0114] This allows us to filter out test-takers who make judgments based only on superficial features without a deep understanding of the situation. Other traps include "short-term perspective," which induces judgments that consider only short-term benefits or impacts and lack a long-term viewpoint, and "partial optimization," which induces judgments that adhere to partial optimization rather than overall optimization. The error identification unit 1313 combines these principles and methods to generate multiple candidate errors that are highly discriminatory to the correct answer of the question and have diagnostic value, and outputs them to the generation control unit 1311.

[0115] According to one embodiment of the present invention, when the explanation-specialized unit 1314 starts processing, it receives as input information the full text of the question, its correct answer, a list of generated incorrect answers, and the problems and learning points contained in the question.

[0116] The explanation-focused unit 1314 aims not only to show the correct answer but also to support the improvement of test takers' practical skills by combining multiple components to generate rich explanatory text.

[0117] The first component is "the essential reason for the correct answer to the question," which involves a logical and step-by-step explanation of why that particular option is correct.

[0118] The second component is "the reason why each incorrect answer to the question is inappropriate," which specifically points out why each incorrect answer choice is wrong and what misunderstanding it is based on. This allows test-takers to accurately understand where the error occurred in their own thought process.

[0119] The third component is "Practical Application Points," which explains, with specific examples, how the knowledge and skills that were the theme of the question can be used in actual work.

[0120] The fourth component is "Theories and Frameworks Related to the Question," which introduces the academic theories and business frameworks that underlie the question, promoting a deeper understanding.

[0121] The fifth component is "Frequently Occurring Failure Patterns and How to Avoid Them," which provides examples of common failures seen in practice related to the topic, along with specific advice on how to prevent them. The explanation specialization unit 1314 generates an explanatory text that conforms to the specified number of characters and tone and manner by combining at least one or more of these components, and outputs it to the generation control unit 1311.

[0122] According to one embodiment of the present invention, when the quality check unit 1322 starts processing, it receives as input information the question to be evaluated (including the question text, answer choices, and correct answer information), the target objective that the question aims to measure, and the profile of the expected test taker (target audience).

[0123] The quality check unit 1322 evaluates the quality of the questions from multiple perspectives based on several check items. The first check item is "measurement accuracy," which evaluates whether the questions purely measure only the target skills required of job seekers, and whether the results are not influenced by unrelated knowledge or reading comprehension.

[0124] The second criterion is "clarity of the correct answer," which evaluates whether the correct answer to a question is objectively unique and leaves no room for interpretation. Ambiguous questions that lead to differing opinions among experts will be considered unacceptable.

[0125] The third check item is "option functionality," which evaluates whether each option functions appropriately. Specifically, it checks whether all incorrect answers are plausible and appealing to test-takers, and whether each option reflects a different set of judgment criteria or patterns of misunderstanding, rather than being merely a paraphrase.

[0126] The fourth check item is "realism," which evaluates whether the scenarios and situations presented in the questions are realistic and could actually occur in real work situations. Unrealistic settings are considered inappropriate because they undermine the test taker's sense of conviction.

[0127] The fifth check item is "judgment criteria," which evaluates whether the judgment required in the question matches the criteria for good decision-making in practice. The quality check unit 1322 evaluates at least one or more of these check items, makes an overall determination of whether it passes or fails, and outputs the result to the quality control unit 1321.

[0128] Figure 6 shows an example of the configuration of the keyword data table 600 according to this embodiment. The keyword data table 600 is a table that manages keyword information that serves as the starting point for generating questions for the skill test. This table includes the fields "Data ID", "Keyword", and "Related Category".

[0129] The "Data ID" is an identifier used to uniquely identify each keyword. The "Keyword" stores specific technical terms or skill names related to the theme of the question (e.g., "Python," "Project Management," etc.). The "Related Category" indicates the broad category to which the keyword belongs (e.g., "Programming Language," "Business Skills," etc.).

[0130] The generation unit 131 (basic generation unit 1312) refers to this table and determines keywords that will be the theme of the question based on the specified category or random selection.

[0131] Figure 7 shows an example of the configuration of the natural language text data table 610 according to this embodiment. The natural language text data table 610 is a table that manages unstructured text data that is referenced when generating the question text, the basis for the correct answer, or the explanatory text. This table includes the fields "Data ID", "Text Content", and "Source / Type".

[0132] The "Text Content" field stores natural language text such as the main body of a technical document, the content of an explanatory article, or an excerpt from a textbook (for example, "Explanation of React's hook functionality..."). The "Source / Type" field indicates the source and type of the information ("Technical Document," "Explanatory Article," etc.).

[0133] The generation unit 131 uses RAG (Retrieval-Augmented Generation) technology or the like to extract text related to the theme of the question from this table and inputs this as context into the LLM to generate factually accurate questions and explanations.

[0134] Figure 8 shows an example of the configuration of the job posting data table 620 according to this embodiment. The job posting data table 620 is a table that manages the definition of the requirements for personnel that a company is seeking. This table includes the fields "Data ID", "Job Title", "Required Skills", and "Job Description".

[0135] The "Job Title" field contains the name of the position being advertised (e.g., "Frontend Engineer," "Data Scientist," etc.). The "Required Skills" field contains the specific technologies and tools required for that job (e.g., "HTML, CSS, etc.," "Python, SQL, etc."). The "Job Description" field contains the specific tasks expected after joining the company (e.g., "Implementing UI for Web applications").

[0136] Based on the information in this table, the generation control unit 1311 instructs each generation engine to generate questions with scenarios that are tailored to the practical work content of a specific job type.

[0137] Figure 9 shows an example of the configuration of the setting information table 630 referenced by the error-specialization unit 1313 according to this embodiment. The setting information table 630 is prompt configuration information that defines the behavior of the error-specialization unit 1313 (AI agent). This table includes "task definition," "input information," "principles of error design," "type of cognitive trap," "output format," and "quality criteria."

[0138] The "Task Definition" states that the test-taker should act as a psychology and education expert. The "Principles of Designing Errors" define guidelines for creating high-quality errors, such as "options that are partially correct but lack a crucial perspective" or "options that are generally correct but inappropriate in this situation." The "Types of Cognitive Traps" define specific methods for testing test-takers, such as "Surface Similarity" (making the test-taker judge based only on the superficial features of the situation) and "Short-Term Perspective."

[0139] The incorrect answer specialization unit 1313 outputs the incorrect answer candidates and their intentions in the specified JSON format (distractors list, etc.) according to these definitions.

[0140] Figure 10 shows an example of the configuration of the setting information table 640 referenced by the explanation specialization unit 1314 according to this embodiment. The setting information table 640 is prompt configuration information that defines the behavior of the explanation specialization unit 1314 (AI agent). This table includes "task definition," "input information," "component elements of explanation," "tone & manner," and "output format."

[0141] The "Task Definition" states that the task involves creating explanations to support skill development as a learning design expert. The "Components of the Explanation" stipulate that the explanation should include not only the "essential reasoning behind the correct answer," but also the "reasons why each incorrect answer is inappropriate (what perspective is lacking)," "points for practical application," and "related theories and frameworks."

[0142] The "Tone & Manner" section specifies a writing style that is not preachy and encourages positive learning. The explanatory section 1314 outputs the explanatory text as structured data (reasoning_correct, practical_tips, etc.) according to these definitions.

[0143] Figure 11 shows an example of the configuration of the setting information table 650 referenced by the quality check unit 1322 and the correction unit 1323 according to this embodiment. The setting information table 650 is definition information for controlling the operation of the quality assurance unit 132. This table includes "task definition," "input," "check items," "output," and "correction rules."

[0144] The "check items" are categorized into "Format (automatic correction)," "Content (suggested corrections)," and "Practical Quality." For format, this includes adjusting character count and ensuring consistent sentence endings, while for content, it defines "measurement accuracy," "clarity of correct answers," and "function of answer choices." Furthermore, for practical quality, consistency in "realism" and "judgment criteria" is checked.

[0145] According to the "output" definition, the quality check unit 1322 classifies the issues into "critical," "major," and "minor," points them out, and outputs JSON data including revised solutions and reasons for the revisions.

[0146] (Other embodiments) Figure 12 is a functional block diagram of the skill test generation server 100, showing another example according to this embodiment. In this modified example, the skill test generation server 100 includes an acquisition unit 134 and a prompt generation unit 135, in addition to the configuration shown in Figure 2.

[0147] The skill test generation server 100 described above may also comprise an acquisition unit 134 that acquires basic data entered regarding the content of a skill test, which is a test for job seekers; a prompt generation unit 135 that generates prompts used in the generation model from the basic data; a generation unit 131 that inputs the prompts into the generation model and generates questions, which are the generation results, from the generation model; a quality assurance unit 132 that evaluates the quality of the questions generated by the generation unit 131; and a control unit 1300 that causes the acquisition unit 134, the prompt generation unit 135, the generation unit 131, and the quality assurance unit 132 to generate the skill test.

[0148] The acquisition unit 134 has the function of acquiring basic data entered regarding the content of the skills test, which is a test given to job seekers. The basic data here includes information such as the job type, required skills, job description, and the difficulty level and target audience of the test. The acquisition unit 134 accepts information entered by the administrator or job posting information linked from an external system as basic data.

[0149] The prompt generation unit 135 has the function of generating prompts (instructions) used by the generation model (e.g., a large-scale language model) used by the generation unit 131, based on the basic data acquired by the acquisition unit 134. In generating prompts, it processes the keywords and conditions included in the basic data into a predetermined format or template that is easy for the generation model to interpret. Alternatively, the prompt generation unit 135 may acquire a prompt template that has been stored in advance in the storage unit 120 or the like, and generate prompts by processing it based on the basic data.

[0150] The prompt generation unit 135 may generate prompts by obtaining prompts that have been pre-generated from an external source. The prompts may also include the content of basic data.

[0151] In this configuration, the generation unit 131 inputs the prompts generated by the prompt generation unit 135 into the generation model and generates questions (question text, answer choices, explanations, etc.) based on the output results from the generation model. The control unit 1300 controls the series of operations of the acquisition unit 134, prompt generation unit 135, generation unit 131, and quality assurance unit 132 to generate the skill test. As a result, highly accurate prompts that conform to the input conditions are automatically prepared, and the quality of question generation by the generation unit 131 can be made more stable.

[0152] <Basic Computer Configuration> Figure 13 is a block diagram showing an example of the hardware configuration of a computer 900 that functions as a skill test generation server 100 and a job seeker terminal 200 according to this embodiment. The computer 900 includes a control unit 901, a storage unit 902, a communication unit 903, an input unit 904, and an output unit 905. The control unit 901, storage unit 902, communication unit 903, input unit 904, and output unit 905 are electrically connected to each other via a communication bus 910.

[0153] The control unit 901 includes a CPU (Central Processing Unit, also called a processor) and controls various parts of the computer 900, as well as reading and executing various programs stored in the storage unit 902.

[0154] The memory unit 902 includes a main memory such as DRAM (Dynamic Random Access Memory) and an auxiliary memory such as a hard disk, and is a device that stores various programs for running the operating system and various applications of the computer 900, as well as data used by these programs. The processes shown in the flowcharts described above are realized by the control units 901 of each computer constituting the skill test generation server 100 and the job seeker terminal 200 executing the programs stored in their respective memory units 902.

[0155] The communication unit 903 is a device for communicating with external devices and sends and receives data according to the instructions of the control unit 901. Each computer that makes up the skill test generation server 100 and the job seeker terminal 200 uses this communication unit 903 to communicate with other devices via the network NW.

[0156] The input unit 904 is a device that receives input from an external source and supplies it to the control unit 901, and includes, for example, a keyboard, mouse, touch panel, and camera. The output unit 905 is a device that outputs the processing results of the control unit 901 to the outside, and includes, for example, a display and speaker.

[0157] <Program> Here, we will describe the programs for realizing each functional unit of the skill test generation server 100 according to this embodiment.

[0158] The skill test generation server 100 is implemented on the computer 900. The operation of each component of the skill test generation server 100 is stored in the auxiliary storage device of the storage unit 902 in the form of a program. The control unit 901 reads the program from the auxiliary storage device of the storage unit 902, expands it into the main memory of the storage unit 902, and executes the above processing according to the program. The control unit 901 also reserves a storage area in the main memory of the storage unit 902 corresponding to the storage unit 120 described above, according to the program.

[0159] Specifically, the program is a skill test generation program that causes the computer 900 to function as a generation unit that generates questions related to a skills test, which is a test for job applicants; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; and a control unit that causes the generation unit and the quality assurance unit to generate the skills test.

[0160] The auxiliary storage device of the memory unit 902 is an example of a tangible medium that is not temporary. Other examples of tangible mediums that are not temporary include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memory connected via the input unit 904. Furthermore, if this program is distributed to the computer 900 via the network NW, the computer 900 that receives the program may load it into the main memory of the memory unit 902 and execute the above processing.

[0161] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in the auxiliary storage device of the memory unit 902.

[0162] <Note> (Note 1) A generation unit that generates questions for a skills test, which is a test for job applicants, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and A skill test generation system equipped with the following features.

[0163] (Note 2) The system includes a feedback unit that updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The skill test generation system as described in Appendix 1, wherein the control unit causes the generation unit, the quality assurance unit, and the feedback unit to generate the skill test.

[0164] (Note 3) The skill test generation system according to Appendix 1 or 2, wherein the generation unit generates the questions based on the basic data input regarding the content of the skill test to be generated.

[0165] (Note 4) The generating unit is A basic generation unit that generates a draft question including the question text that forms the framework of the aforementioned question and the intent behind the question, An error-specializing unit that generates candidate incorrect answers for the aforementioned draft question, A specialized explanation unit that generates an explanation text including the basis for the correct answer and the reason for the incorrect answer corresponding to the aforementioned draft question, The generation control unit generates the basic questions by inputting the basic data into the basic generation unit, generates the incorrect answer candidates by inputting the incorrect answer specialized unit, generates the explanation text by inputting the correct question and the incorrect answer candidates into the explanation text, and generates the questions based on the correct question, the incorrect answer candidates and the explanation text. A skill test generation system, including any of the systems described in Appendix 1 to 3.

[0166] (Note 5) The aforementioned quality assurance unit includes a quality check unit that determines whether the quality of the question is acceptable or unacceptable based on pre-set quality standards, A correction unit for making corrections to the aforementioned question, A formatting adjustment unit that formats the aforementioned questions into the final format, A quality control unit is configured to input the question generated by the generation unit into the quality check unit and determine whether the question is pass or fail, and if the question is failing, to input the failing question into the correction unit and output a corrected question, and to input the question that the quality check unit determined to be passable, or the corrected question from the correction unit, into the format adjustment unit and output a final adjusted question. A skill test generation system having any of the features described in Appendix 1 to 4.

[0167] (Note 6) The aforementioned modification unit performs at least one of the following modification processes based on the reason for failure: unifying the sentence-ending expressions of the questions, regenerating incorrect answers to the questions, or reinforcing the explanatory text for the questions, as described in any of the appendices 1 to 5 of the skill test generation system.

[0168] (Note 7) The aforementioned feedback unit is A memory unit for storing answer data of the job seeker who took the aforementioned skills test, A statistical analysis unit that calculates the difficulty level of the question or the discrimination index of the question using statistical test theory based on the aforementioned answer data, The effectiveness evaluation unit evaluates the effectiveness of the question based on the calculated indicators and provides feedback to the generation parameters used in generating the question in the generation unit. A skill test generation system as described in Appendix 2, having the following features.

[0169] (Note 8) The aforementioned incorrect answer specialization unit accepts input information including the question, the correct answer to the question, and the measurement target of the question, and, in principle, generates the incorrect answer design based on superficial similarity that makes it easy to select based on superficial characteristics of the situation, including at least one of the following: an option that is partially correct, an option that is generally correct but inappropriate in the situation of the question, and an option that is likely to be selected due to a typical misunderstanding, thereby generating the incorrect answer candidates as described in Appendix 4.

[0170] (Note 9) The aforementioned specialized explanation unit accepts input information including all of the aforementioned questions, the correct answers to the questions, the incorrect answers to the questions, and the problems, and generates the explanation text using at least one of the following as components: the essential reason for the correct answer to the question, the reason why the incorrect answer to the question is inappropriate, points for practical application, theories and frameworks related to the question, and frequently occurring failure patterns and methods for avoiding them, as described in Appendix 4 of the skill test generation system.

[0171] (Note 10) The quality check unit receives input information including the question, the target set for the question, and the target person for the question, and checks at least one of the following: measurement accuracy, which indicates whether only the target skills required of the job seeker are being judged; clarity of correct answer, which indicates whether there is room for interpretation as to whether the answer to the question is correct or not; functionality of the options, which indicates whether each option of the question reflects a different judgment axis; realism, which indicates whether the situation in the question is a situation that could actually occur; and judgment criteria, which indicates whether the question is consistent with decision-making in practical work, and determines whether it is a pass or a fail, as described in Appendix 5.

[0172] (Note 11) An acquisition unit that acquires basic data entered regarding the content of the skills test, which is a test for job applicants, A prompt generation unit that generates prompts used in the generation model from the aforementioned basic data, A generation unit inputs the aforementioned prompt into the generation model and generates a question, which is the generation result, from the generation model. A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and A skill test generation system equipped with the following features.

[0173] (Note 12) A generation unit that generates questions for a skills test, which is a test for job applicants, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and A skill test generation server equipped with the following features.

[0174] (Note 13) A generation step that generates questions for a skills test, which is a test for job applicants, A quality assurance step which evaluates the quality of the questions generated by the generation step, The generation step and the quality assurance step are further controlled to generate the skill test. A method for generating skill tests that includes the following:

[0175] (Note 14) Computers, A generation unit that generates questions for a skills test, which is a test for job applicants. A quality assurance unit for evaluating the quality of the questions generated by the generation unit, and a skill test generation program for functioning as a control unit that causes the generation unit and the quality assurance unit to generate the skill test. [Explanation of symbols]

[0176] 1: Skill Test Generation System 100: Skill Test Generation Server 110: Communications Department 120: Storage section 130: Processing Unit 131: Generation part 132: Quality Assurance Department 133: Feedback Department 134: Acquisition Department 135: Prompt generation unit 200: Job seeker terminal 600: Keyword Data Table 610: Natural Language Text Data Table 620: Job posting data table 630: Configuration Information Table 640: Configuration Information Table 650: Configuration Information Table 900: Computer 901: Control Unit 902: Storage section 903: Communications Department 904: Input section 905: Output section 910: Bus 1300: Control Unit 1311: Generation Control Unit 1312 :Basic generator 1313: Incorrect Answer Specialization Department 1314: Specialized Commentary Section 1321: Quality Control Department 1322: Quality Check Department 1323: Correction section 1324: Format adjustment section 1331:Statistical Analysis Department 1332: Effectiveness Evaluation Department

Claims

1. A generation unit that generates questions for a skills test, which is a test for job applicants, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and Equipped with, The generating unit is The system generates the questions based on the basic data entered regarding the content of the skills test to be generated, A basic generation unit that generates a draft question including the question text that forms the framework of the aforementioned question and the intent behind the question, An error-specializing unit that generates candidate incorrect answers for the aforementioned draft question, A specialized explanation unit generates an explanation text that includes the basis for the correct answer corresponding to the draft question and the reasons for incorrect answers, including the candidate incorrect answers. The system comprises a basic generation unit which inputs the basic data to generate the draft question, an incorrect answer specialization unit which inputs the draft question to generate the incorrect answer candidates, an explanation specialization unit which inputs the draft question and the incorrect answer candidates to generate the explanation text, and a generation control unit which generates the question based on the draft question, the incorrect answer candidates, and the explanation text. The aforementioned Quality Assurance Department A quality check unit that determines whether the quality of the aforementioned question is acceptable or unacceptable based on pre-set quality standards, A correction unit for making corrections to the aforementioned question, A formatting adjustment unit that formats the aforementioned questions into the final format, A skill test generation system comprising: a quality check unit which inputs the questions generated by the generation unit and determines whether the questions are pass or fail; if the questions are failing, a correction unit which inputs the failing questions and outputs the corrected questions; and a quality control unit which inputs the questions that the quality check unit determined to be passable, or the corrected questions from the correction unit, and outputs the questions after final adjustments.

2. The system includes a feedback unit that updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The skill test generation system according to claim 1, wherein the control unit causes the generation unit, the quality assurance unit, and the feedback unit to generate the skill test.

3. The skill test generation system according to claim 1, wherein the modification unit performs at least one of the following modification processes based on the reason for failure: unifying the sentence-ending expressions of the question, regenerating the candidate incorrect answers for the question, or reinforcing the explanatory text for the question.

4. The aforementioned feedback unit is A memory unit for storing answer data of the job seeker who took the aforementioned skills test, A statistical analysis unit that calculates the difficulty level of the question or the discrimination index of the question using statistical test theory based on the aforementioned answer data, The effectiveness evaluation unit evaluates the effectiveness of the question based on the calculated difficulty level of the question or the discrimination index of the question, and provides feedback to the generation parameters used in generating the question in the generation unit. The skill test generation system according to claim 2, comprising:

5. The skill test generation system according to claim 1, wherein the error-specialization unit receives input information including the question, the correct answer to the question, and the measurement target of the question, and generates the error-specialization unit based on superficial similarity that makes it easy to select based only on the superficial characteristics of the situation, based on an error-specialization that includes at least one of the following: an option that is partially correct, an option that is generally correct but inappropriate in the situation of the question, and an option that is likely to be selected due to a typical misunderstanding.

6. The skill test generation system according to claim 1, wherein the explanation-specialized unit accepts input information including all of the questions, the correct answers to the questions, the incorrect answers including the candidate incorrect answers to the questions, and the problems, and generates the explanation text using at least one of the following as components: the essential reason for the correct answer to the question, the reason why the incorrect answers including the candidate incorrect answers to the questions are inappropriate, points for practical application, theories and frameworks related to the questions, and frequently occurring failure patterns and methods for avoiding them.

7. The skill test generation system according to claim 1, wherein the quality check unit receives input information including the question, the target setting of the question, and the target person of the question, and checks at least one of the following: measurement accuracy indicating whether only the target skills required of the job seeker are being judged; clarity of correct answer indicating whether there is room for interpretation as to whether the answer to the question is correct or not; functionality of the options indicating whether each option of the question reflects a different judgment axis; realism indicating whether the situation of the question is a situation that could actually occur; and judgment criteria indicating whether the question is consistent with decision-making in practical work.

8. An acquisition unit that acquires basic data entered regarding the content of the skills test, which is a test for job applicants, A prompt generation unit that generates prompts used in the generation model from the aforementioned basic data, A generation unit inputs the aforementioned prompt into the generation model and generates a question, which is the generation result, from the generation model. A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and Equipped with, The aforementioned basic data includes job posting data from the employer, The prompt generation unit embeds information including required skills and job description contained in the job posting data into the format of task definitions and input information, constructs instruction statements including specific system prompts for the generation model and prompts including user prompts, or obtains prompts that have been pre-generated from an external source. The generation unit inputs the prompt constructed or acquired by the prompt generation unit into the generation model, and generates the question based on the generation result output from the generation model, A basic generation unit that generates a draft question including the question text that forms the framework of the aforementioned question and the intent behind the question, An error-specializing unit that generates candidate incorrect answers for the aforementioned draft question, A specialized explanation unit generates an explanation text that includes the basis for the correct answer corresponding to the draft question and the reasons for incorrect answers, including the candidate incorrect answers. The system comprises a basic generation unit which inputs the basic data to generate the draft question, an incorrect answer specialization unit which inputs the draft question to generate the incorrect answer candidates, an explanation specialization unit which inputs the draft question and the incorrect answer candidates to generate the explanation text, and a generation control unit which generates the question based on the draft question, the incorrect answer candidates, and the explanation text. The aforementioned Quality Assurance Department A quality check unit that determines whether the quality of the aforementioned question is acceptable or unacceptable based on pre-set quality standards, A correction unit for making corrections to the aforementioned question, A formatting adjustment unit that formats the aforementioned questions into the final format, A skill test generation system comprising: a quality check unit which inputs the questions generated by the generation unit and determines whether the questions are pass or fail; if the questions are failing, a correction unit which inputs the failing questions and outputs the corrected questions; and a quality control unit which inputs the questions that the quality check unit determined to be passable, or the corrected questions from the correction unit, and outputs the questions after final adjustments.

9. A generation unit that generates questions for a skills test, which is a test for job applicants, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and Equipped with, The generation unit generates the questions based on the basic data input regarding the content of the skill test to be generated, A basic generation unit that generates a draft question including the question text that forms the framework of the aforementioned question and the intent behind the question, An error-specializing unit that generates candidate incorrect answers for the aforementioned draft question, A specialized explanation unit generates an explanation text that includes the basis for the correct answer corresponding to the draft question and the reasons for incorrect answers, including the candidate incorrect answers. The system comprises a basic generation unit which inputs the basic data to generate the draft question, an incorrect answer specialization unit which inputs the draft question to generate the incorrect answer candidates, an explanation specialization unit which inputs the draft question and the incorrect answer candidates to generate the explanation text, and a generation control unit which generates the question based on the draft question, the incorrect answer candidates, and the explanation text. The aforementioned Quality Assurance Department A quality check unit that determines whether the quality of the aforementioned question is acceptable or unacceptable based on pre-set quality standards, A correction unit for making corrections to the aforementioned question, A formatting adjustment unit that formats the aforementioned questions into the final format, A skill test generation server comprising: a quality check unit which inputs the questions generated by the generation unit and determines whether the questions are pass or fail; if the questions are failing, a correction unit which inputs the failing questions and outputs the corrected questions; and a quality control unit which inputs the questions that the quality check unit determined to be passable, or the corrected questions from the correction unit, and outputs the questions after final adjustments.

10. A method for generating a skill test performed by a computer, A generation step that generates questions for a skills test, which is a test for job applicants, A quality assurance step which evaluates the quality of the questions generated by the generation step, The generation step and the quality assurance step are further controlled to generate the skill test. It has, In the generation step, the questions are generated based on the basic data entered regarding the content of the skill test to be generated, The basic generation step involves generating a draft question that includes the core question text and the intent behind the question, An error-specialization step that generates incorrect answer candidates, which are candidates for incorrect answers to the aforementioned draft question, A specialized explanation step that generates an explanation text including the basis for the correct answer corresponding to the aforementioned draft question and the reasons for incorrect answers, including the aforementioned candidate incorrect answers, The basic generation step includes inputting the basic data to generate the draft question; the incorrect answer specialization step includes inputting the draft question to generate the incorrect answer candidates; the explanation specialization step includes inputting the draft question and the incorrect answer candidates to generate the explanation text; and the generation control step includes generating the question based on the draft question, the incorrect answer candidates, and the explanation text. The aforementioned quality assurance step is: A quality check step that determines whether the quality of the aforementioned question is acceptable or unacceptable based on pre-set quality standards, A correction step to make revisions to the above question, A formatting adjustment step to format the aforementioned questions into the final format, A skill test generation method comprising: a quality check step in which the question generated in the generation step is input and it is determined whether the question is a pass or a fail; if the question is a fail, a correction step in which the failing question is input and a corrected question is output; and a quality control step in which the question determined to be a pass in the quality check step, or the corrected question from the correction step, is input and the final adjusted question is output.

11. Computers, A generation unit that generates questions for a skills test, which is a test for job applicants, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, The generation unit and the quality assurance unit are configured to include a control unit that generates the skill test and and make it work The generation unit generates the questions based on the basic data input regarding the content of the skill test to be generated, The aforementioned generation unit, A basic generation unit that generates a draft question including the question text that forms the framework of the aforementioned question and the intent behind the question, An error-specializing unit that generates candidate incorrect answers for the aforementioned draft question, A specialized explanation unit generates an explanation text that includes the basis for the correct answer corresponding to the draft question and the reasons for incorrect answers, including the candidate incorrect answers. The basic generation unit is configured to input the basic data and generate the draft question; the incorrect answer specialization unit is configured to input the draft question and generate the incorrect answer candidates; the explanation specialization unit is configured to input the draft question and the incorrect answer candidates and generate the explanation text; and the unit functions as a generation control unit that generates the question based on the draft question, the incorrect answer candidates, and the explanation text. The aforementioned quality assurance department, A quality check unit that determines whether the quality of the aforementioned question is acceptable or unacceptable based on pre-set quality standards, A correction unit for making corrections to the aforementioned question, A formatting adjustment unit that formats the aforementioned questions into the final format, A skill test generation program that functions as a quality control unit, inputting the questions generated by the generation unit into the quality check unit and determining whether the questions are pass or fail; if the questions are failing, inputting the failing questions into the correction unit and outputting the corrected questions; and inputting the questions that the quality check unit determined to be passable, or the corrected questions from the correction unit, into the format adjustment unit and outputting the questions after final adjustments.

Images