Skill test generation system, skill test generation server, skill test generation method, and skill test generation program

Abstract

The objective is to provide a skill test generation system, skill test generation server, skill test generation method, and skill test generation program that can generate skill test questions of appropriate difficulty from input basic data without requiring a great deal of time and effort in generating skill test questions. [Solution] The system comprises a generation unit that generates skill test questions based on basic data input from the recruiting party, a quality assurance unit that evaluates the quality of the generated questions, a feedback unit that updates the generation parameters based on the test results, and a control unit that controls these. The quality assurance unit is a skill test generation system that has a function to check the difficulty level of the questions according to set criteria and correct them if they fail.

Description

Technical Field

[0001] The present invention relates to a skill test generation system, a skill test generation server, a skill test generation method, and a skill test generation program.

Background Art

[0002] In recent years, in job hunting and recruitment matching services using the Internet (so-called job hunting websites), in order to objectively measure the vocational ability and suitability of job seekers, skill tests (CBT: Computer Based Testing) implemented online using a web browser or the like are widely used. Employing companies use such test results in addition to document information such as resumes and work experience documents as materials for judging whether the practical skills possessed by job seekers meet the standards required by the company.

[0003] In a conventional general skill test providing system, templates of a large number of pre-created question data are stored in a database (question library) for each category such as programming languages, accounting knowledge, language proficiency, and logical thinking ability. When an employing company selects test items according to the job type and requirements to be recruited, the system extracts questions that meet the conditions from the database randomly or according to a predetermined logic, constructs a test set, and presents it to the job seeker's terminal.

[0004] In the system as described above, the generation process of the question data registered in the database basically depends on manual work. Specifically, a question writer with knowledge of the target specialized field analyzes general job requirements and technological trends to create question texts, options, correct answers, and explanatory texts. In order to ensure the presence or absence of typos and the accuracy of the content, the created data is generally double-checked by a corrector or supervisor and then released after being registered in the system.

[0005] Furthermore, the difficulty level of the created tests is also managed manually. For example, a process may be included in which a third party (tester) actually takes the pre-release test and measures the accuracy rate and response time to confirm whether the difficulty level of the questions is as intended and whether any inappropriate questions are included. In this way, in conventional technologies, human judgment and work are involved in many processes, from the planning and creation of questions to verification and implementation into the system.

[0006] Patent Document 1 describes a job recruitment and job seeker support system and method that enables the rapid and simple selection of suitable personnel based on objective and rational selection criteria. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2008-176666 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, in the conventional skill test question generation process, it was not possible to objectively evaluate and guarantee the quality of the generated questions, particularly their difficulty level, specialization, and relevance to practical work, without requiring significant time and effort. Furthermore, because adding questions required manual effort and the number of questions could not be prepared, candidates could prepare using so-called past questions, making it impossible to accurately measure the true skills of job seekers. As a result, it was not possible to properly evaluate the job seekers' work skills, potentially leading to increased recruitment costs and hiring mismatches.

[0009] Therefore, the present invention aims to provide a skill test generation system, a skill test generation server, a skill test generation method, and a skill test generation program that can generate skill test questions of appropriate difficulty from input basic data without requiring a great deal of time and effort in generating skill test questions. [Means for solving the problem]

[0010] A skill test generation system according to a first aspect of the present invention comprises: a generation unit that generates questions related to a skill test, which is a test for job seekers, based on basic data input by the employer that is recruiting the job seekers; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; a feedback unit that updates the parameters used by the generation unit to generate the questions based on the results of the skill test; and a control unit that causes the generation unit, the quality assurance unit, and the feedback unit to generate the skill test, wherein the basic data is input regarding the content of the skill test to be generated. The quality assurance unit includes at least one of the selected keywords, natural language text relating to the content of the generated skill test, and the job posting data from the recruiting party. The quality assurance unit includes a quality check unit that determines whether the difficulty level of the question is pass or fail based on pre-set quality standards, a modification unit that modifies the difficulty level of the question, and a quality control unit that inputs the question generated by the generation unit to the quality check unit and causes it to determine whether the difficulty level of the question is pass or fail, and if the question is failing, inputs the failing question to the modification unit and outputs the question with the difficulty level modified.

[0011] A skill test generation server according to a second aspect of the present invention comprises: a generation unit that generates questions related to a skill test, which is a test for job seekers, based on basic data input by the employer that is recruiting the job seekers; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; a feedback unit that updates the parameters used by the generation unit to generate the questions based on the results of the skill test; and a control unit that causes the generation unit, the quality assurance unit, and the feedback unit to generate the skill test, wherein the basic data is input with respect to the content of the skill test to be generated. The quality assurance unit includes at least one of the following: keywords, natural language text relating to the content of the skill test to be generated, and job posting data from the recruiting party. The quality assurance unit includes a quality check unit that determines whether the difficulty level of the question is pass or fail based on pre-set quality standards, a modification unit that modifies the difficulty level of the question, and a quality control unit that inputs the question generated by the generation unit to the quality check unit and causes it to determine whether the difficulty level of the question is pass or fail, and if the question is failing, inputs the failing question to the modification unit and outputs the question with the difficulty level modified.

[0012] A third aspect of the present invention relates to a skill test generation method comprising: a generation step of generating questions relating to a skill test, which is a test for job seekers, based on basic data input by the employer recruiting the job seekers; a quality assurance step of evaluating the quality of the questions generated by the generation step; a feedback step of updating the parameters used by the generation step to generate the questions based on the results of the skill test; and a control step of causing the generation step, the quality assurance step, and the feedback step to generate the skill test, wherein the basic data relates to the content of the skill test to be generated. The quality assurance step includes at least one of the specified keywords, natural language text relating to the content of the skill test to be generated, and the job posting data from the recruiting party, and the quality assurance step includes a quality check step that determines whether the difficulty level of the question is pass or fail based on a predetermined quality standard, a modification step that modifies the difficulty level of the question, and a quality control step that inputs the question generated in the generation step into the quality check step and determines whether the difficulty level of the question is pass or fail, and if the question is failing, inputs the failing question into the modification step and outputs the question with the difficulty level modified.

[0013] A skill test generation program according to a fourth aspect of the present invention comprises a computer, a generation unit that generates questions relating to a skill test, which is a test for job seekers, based on basic data input by the employer recruiting the job seekers; a quality assurance unit that evaluates the quality of the questions generated by the generation unit; a feedback unit that updates the parameters used by the generation unit to generate the questions based on the results of the skill test; and a control unit that causes the generation unit, the quality assurance unit, and the feedback unit to generate the skill test, wherein the basic data is input with respect to the content of the skill test to be generated. The quality assurance unit includes at least one of the following: keywords, natural language text relating to the content of the skill test to be generated, and job posting data from the recruiting party. The quality assurance unit is characterized by functioning as a quality control unit that includes a quality check unit that determines whether the difficulty level of the question is pass or fail based on pre-set quality standards, a modification unit that modifies the difficulty level of the question, and a quality control unit that inputs the question generated by the generation unit to the quality check unit and causes it to determine whether the difficulty level of the question is pass or fail, and if the question is failing, inputs the failing question to the modification unit and outputs the question with the difficulty level modified. [Effects of the Invention]

[0014] According to the present invention, in generating skill test questions, it is possible to generate questions of appropriate difficulty from input basic data without requiring a great deal of time and effort. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram showing the overall configuration of the skill test generation system 1 according to this embodiment. [Figure 2] This is a functional block diagram of the skill test generation server 100 according to this embodiment. [Figure 3] This figure shows a basic flowchart of the skill test generation system 1 according to this embodiment. [Figure 4]It is a diagram showing a more detailed flowchart of the generation unit 131 according to this embodiment. [Figure 5] It is a diagram showing a more detailed flowchart of the quality assurance unit 132 according to this embodiment. [Figure 6] It is a diagram showing a configuration example of the keyword data table 600 according to this embodiment. [Figure 7] It is a diagram showing a configuration example of the natural language text data table 610 according to this embodiment. [Figure 8] It is a diagram showing a configuration example of the job offer data table 620 according to this embodiment. [Figure 9] It is a diagram showing a configuration example of the setting information table 630 referred to by the wrong answer specialization unit 1313 according to this embodiment. [Figure 10] It is a diagram showing a configuration example of the explanation specialization unit 1314 according to this embodiment. [Figure 11] It is a diagram showing a configuration example of the setting information table 650 referred to by the quality check unit 1322 and the correction unit 1323 according to this embodiment. [Figure 12] It is a functional block diagram of the skill test generation server 100 showing another example according to this embodiment [Figure 13] It is a block diagram showing a hardware configuration example of the computer 900 that functions as the skill test generation server 100 and the job seeker terminal 200 according to this embodiment.

Mode for Carrying Out the Invention

[0016] <## Hereinafter, embodiments for carrying out the present invention will be described in detail. Note that the present invention is not limited to the following embodiments, and various modifications are possible within the scope of the gist.

[0017] Figure 1 is a schematic diagram showing the overall configuration of the skill test generation system 1 according to this embodiment. The skill test generation system 1 is a system that automatically generates skill tests for evaluating the skills of job seekers. As shown in Figure 1, the skill test generation system 1 comprises a skill test generation server 100 and one or more job seeker terminals 200 used by job seekers. The skill test generation server 100 and the job seeker terminals 200 are connected to each other so as to be able to communicate with each other via a network NW, which is a communication line network such as the Internet, LAN (Local Area Network), or WAN (Wide Area Network).

[0018] The skill test generation server 100 is an information processing device that automatically generates skill test questions based on basic data provided by the recruiting party, guarantees their quality, and provides a series of functions to improve the generation process by providing feedback on the test results. The skill test generation server 100 can be implemented, for example, by one or more server computers built in a cloud computing environment.

[0019] The job seeker terminal 200 is a terminal device operated by the job seeker, which receives and displays the skill test generated by the skill test generation server 100, accepts answer input from the job seeker, and transmits it to the skill test generation server 100. The job seeker terminal 200 is an information processing terminal such as a smartphone, tablet, personal computer (PC), or notebook PC. The job seeker terminal 200 uses the services provided by the skill test generation server 100 via a web browser or dedicated application software. Although not shown in the diagram, the recruiting party's terminal, operated by the recruiting party, is also connected to the skill test generation server 100 via a network NW, and can input basic data that forms the basis for skill test generation, check the generated skill test, and view the job seeker's test results.

[0020] Figure 2 is a functional block diagram of the skill test generation server 100 according to this embodiment. Functionally, the skill test generation server 100 comprises a communication unit 110, a storage unit 120, and a processing unit 130.

[0021] The communications unit 110 is responsible for data communication with external devices via the network NW.

[0022] The memory unit 120 stores various information used by the skill test generation server 100 for processing. The memory unit 120 stores the program for executing the skill test generation process, as well as basic data input from the recruiting party. The basic data includes at least one of the following: keywords input regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and job posting data from the recruiting party.

[0023] Specifically, the memory unit 120 stores a keyword data table 600, a natural language text data table 610, a job posting data table 620, and the like. The keyword data table 600 stores data IDs associated with keywords representing skills (e.g., "Python", "Project Management") and related categories. The natural language text data table 610 stores data IDs associated with text content such as technical documents and explanatory articles, and their sources and types. The job posting data table 620 stores data IDs associated with job posting information such as job title, required skills, and job description.

[0024] The basic data may be the input information that forms the basis for the generation unit 131 when generating questions related to a skills test for job seekers. Based on this basic data, the generation unit 131 may determine the theme, difficulty level, and direction of the scenario for the questions. Specifically, the basic data may consist of the following elements:

[0025] The basic data may include information defining technical terms and concepts that are the subject of the question. This information may be managed, for example, as a keyword data table 600, where "keywords" that indicate specific skills or knowledge are associated with "related categories," which are the broad classifications to which those keywords belong. The generation unit 131 may refer to this and select specific themes, such as "Python" or "project management."

[0026] The basic data may include text information that serves as the basis for the questions and explanations. This may be managed as a natural language text data table 610 and consist of "text content" such as technical documents, explanatory articles, or textbooks, and "source / type" indicating the source of that information. The generation unit 131 may generate factual and accurate questions by referencing this text using RAG (Retrieval-Augmented Generation) technology or the like.

[0027] The basic data may include job information that defines the practical context. This job information is managed as the job posting data table 620, which defines the "job title" being recruited, the "essential skills" required for that job title, and the specific "job duties." This allows the generation unit 131 to generate questions that are not merely knowledge-based but are tailored to situations that may occur in actual work environments.

[0028] The basic data is not limited to information in an internal database, but may also include test difficulty settings entered by the administrator, profiles of the target test-takers, or even job postings themselves linked from external systems. Furthermore, the generation unit 131 may receive the basic data as input and reflect it in instructions (prompts) to an internal generation model (LLM, etc.) to generate a skill test that matches the requested specifications.

[0029] Basic data may include information that includes at least one of the following: the job description offered by the employer, the skills required by the employer for job seekers, and the trends in the industry to which the employer belongs. This information may be obtained from job posting data or by direct input by the employer's representative.

[0030] The basic data may be the pre-created questions themselves. The generation unit 131 may create so-called similar questions based on the pre-created questions, with equivalent items and difficulty levels.

[0031] The difficulty level may include, for example, a standard score based on the score distribution of past test takers. For example, if a standard score of 65 is specified, the generation unit 131 may generate questions of a difficulty level such that a test taker with a standard score of 65 can achieve a passing score.

[0032] Required skills may be specified in the form of so-called national qualifications, private qualifications, etc. For example, if the Information Processing Engineer Examination is specified as a private qualification, the generation unit 131 may generate questions that are similar to those asked in the Information Processing Engineer Examination as the theme of the questions.

[0033] When a job posting is input as basic data, the control unit 1300 may have the basic generation unit 1312 present the job seeker with the themes for the questions. There may be multiple themes for the questions.

[0034] The memory unit 120 stores information generated or used by each part of the processing unit 130. For example, it stores draft questions, candidate incorrect answers, explanatory texts, and completed question data generated by the generation unit 131, quality standard data and revision history data used by the quality assurance unit 132, applicant response data and statistical analysis results used by the feedback unit 133, and updated generation parameters.

[0035] The processing unit 130 is a functional unit realized by the control unit 901 in the aforementioned hardware configuration executing a predetermined program, and performs a series of processes related to skill test generation. The processing unit 130 controls the operation of the entire skill test generation server 100. The processing unit 130 includes a control unit 1300, a generation unit 131, a quality assurance unit 132, and a feedback unit 133.

[0036] The control unit 1300 causes the generation unit 131 and the quality assurance unit 132 to generate skill tests. The control unit 1300 may also cause the feedback unit 133 to update the parameters that the generation unit 131 uses to generate questions based on the results of the skill tests.

[0037] The generation unit 131 generates questions related to a skills test for job applicants based on basic data stored in the memory unit 120. The generation unit 131 includes a generation control unit 1311, a basic generation unit 1312, an incorrect answer specialization unit 1313, and an explanation specialization unit 1314. The generation unit 131 can be implemented using an AI model such as a large-scale language model (LLM).

[0038] The generation control unit 1311 comprehensively controls the processing flow within the generation unit 131. The generation control unit 1311 operates the basic generation unit 1312, the incorrect answer specialization unit 1313, and the explanation specialization unit 1314 sequentially or in parallel, and integrates the outputs of each unit to generate the final question.

[0039] The basic generation unit 1312 generates draft questions, including the core question text and the intent behind the question, based on the basic data. For example, the basic generation unit 1312 generates draft questions that test knowledge of Python data analysis libraries and questions that test evaluation metrics for machine learning models, based on information such as "Required skills: Python, machine learning knowledge" and "Job description: Data analysis and model building" included in the job posting data.

[0040] The basic generation unit 1312 may, when a job posting is entered as basic data, present the employer with a selection of recommended skill test themes. The basic generation unit 1312 may also create questions based on the themes selected by the employer.

[0041] The incorrect answer specialization unit 1313 generates plausible incorrect answer options, which are candidate incorrect answers, for the draft questions generated by the basic generation unit 1312. The incorrect answer specialization unit 1313, acting as an expert in psychology and educational technology, creates options that test-takers are likely to get wrong. The incorrect answer specialization unit 1313 generates candidate incorrect answers based on the principles defined in the setting information table 630.

[0042] Specifically, the error-specialization unit 1313 receives input information including the question, the correct answer to the question, and the measurement target of the question. As a principle of error design, the error-specialization unit 1313 designs candidate errors to include at least one of the following: an option that is partially correct but lacks an important perspective, an option that is generally correct but inappropriate under specific circumstances of the question, and an option that relates to a typical misconception that beginners are prone to.

[0043] Furthermore, the error-specializing unit 1313 generates clever error candidates by utilizing perspectives such as "superficial similarity," which causes users to make incorrect judgments based only on the superficial characteristics of a situation; "short-term perspective," which causes users to consider only short-term effects; and "partial optimization," which causes users to lose sight of the overall optimality, as types of cognitive traps.

[0044] The explanation-specialized unit 1314 generates detailed explanations to support the practical skill improvement of test takers, based on the draft questions, correct answers, and incorrect answer candidates generated by the incorrect answer-specialized unit 1313. The explanation-specialized unit 1314 creates explanations for questions as an expert in learning design in corporate training. The explanation-specialized unit 1314 generates explanations based on the components defined in the setting information table 640 related to explanation text generation.

[0045] The explanation-focused unit 1314 accepts input information including all questions, correct answers, incorrect answers, and problems. The components of the explanation include at least one of the following: the reason why the correct answer is essentially correct, the reason why each incorrect answer is inappropriate (not just pointing out that it is "wrong," but explaining what perspective is lacking), application points on how the knowledge can be applied in practice, theories and frameworks related to the question, and common failure patterns in practice and how to avoid them.

[0046] The quality assurance unit 132 ensures the overall quality of the skill test by evaluating the quality of the questions generated by the generation unit 131 and making corrections as necessary. The quality assurance unit 132 includes a quality control unit 1321, a quality check unit 1322, a correction unit 1323, and a format adjustment unit 1324.

[0047] The quality control unit 1321 comprehensively controls the processing flow within the quality assurance unit 132. The quality control unit 1321 inputs the generated questions into the quality check unit 1322 and, depending on the judgment result, passes the questions to the correction unit 1323 or the format adjustment unit 1324.

[0048] The quality check unit 1322 evaluates the difficulty level and appropriateness of the questions based on pre-set quality standards and determines whether they pass or fail. As a quality control expert for the questions, the quality check unit 1322 evaluates the content of the questions from multiple perspectives. The quality check unit 1322 makes its judgment based on the check items defined in the quality check setting information table 650.

[0049] The quality check department 1322 evaluates the difficulty level of a question based on at least one of the following: the specialization of the terminology included in the question, the number of thinking steps required to answer the question, and the correct answer rate for similar questions in past skill tests.

[0050] The difficulty level of a question may be an indicator of the level of ability required for a test-taker to answer the question correctly, or the magnitude of the cognitive load required to solve the question. Difficulty level is defined and managed from both a predictive assessment during the generation stage before the question is released and a statistical assessment based on the results after the test is actually administered.

[0051] During the question generation stage, the quality check unit 1322 may estimate the difficulty level by analyzing the components of the question. Specifically, the following three perspectives may be considered. These three perspectives may include "specialization of terminology," "number of thinking steps," and "similarity to past data."

[0052] "Specialized terminology" may be a criterion for determining the difficulty level based on whether the keywords included in the question and answer choices are common or require advanced specialized knowledge (for example, their frequency of appearance and level in the dictionary data of the memory unit 120).

[0053] The "number of thinking steps" can be a criterion for judgment based on the number of steps in the solution process, determining whether it is sufficient to recall a single piece of knowledge to arrive at the correct answer (low difficulty), or whether it requires logical reasoning combining multiple conditions or complex matching of preconditions (high difficulty).

[0054] "Similarity to past data" may be used to estimate the difficulty level of a question by referring to data on the correct answer rate and answer time of similar questions that have been asked in the past. Furthermore, the quality check unit 1322 may evaluate the difficulty level not only as an absolute value, but also in relation to the set "target audience".

[0055] For example, if the target audience is at a "junior level," questions that ask about advanced "judgment criteria" or "exceptions in practical work" may be judged by the quality check unit 1322 as excessively difficult (failure) and may be subject to modification by the correction unit 1323, such as simplifying the wording or adding hints.

[0056] Furthermore, after the skill test is conducted, the feedback unit 133 (statistical analysis unit 1331) may calculate the difficulty level as a mathematical "difficulty level (b parameter)" using a statistical model such as Item Response Theory (IRT). If this calculated difficulty level deviates significantly from the assumption made during generation, the feedback unit 133 may feed this information back into the parameters for the next question generation. This can contribute to improving the accuracy of question generation.

[0057] Furthermore, the quality check unit 1322 accepts input information including the question, the objective set for the question, and the target group for the question (e.g., junior level, senior level), and evaluates multiple check items. The check items include at least one of the following: "measurement accuracy," which indicates whether only the target skills required of job seekers can be determined; "clarity of correct answer," which indicates whether there is no room for interpretation as to whether the answer to the question is correct or not; "functionality of options," which indicates whether each option in the question reflects a different judgment axis and is not merely a paraphrase; "realism," which indicates whether the scenario presented in the question is a situation that could actually occur in practice; and "judgment criteria," which indicates whether the correct answer selection in the question is consistent with good decision-making in practice.

[0058] The modification unit 1323 modifies the difficulty level and content of questions that have been judged as unacceptable by the quality check unit 1322. Based on the reason for the failure (for example, "option C is not functioning" or "the scenario is not realistic"), the modification unit 1323 performs specific modification processes.

[0059] As specific methods of correction, the correction unit 1323 will perform at least one of the following: change the wording of the question (to make it clearer, replace technical terms with simpler language, etc.), replace the content of the question (change the scenario to be more realistic, etc.), or delete the question itself if it is deemed inappropriate.

[0060] Furthermore, the correction unit 1323 also has the function of performing more specific correction processes based on the reason for failure, such as unifying sentence-ending expressions across multiple questions, regenerating incorrect answer choices that do not function properly, or reinforcing explanatory text to aid the test-taker's understanding.

[0061] The formatting adjustment unit 1324 formats the questions that the quality check unit 1322 has determined to be acceptable, or the questions after correction by the correction unit 1323, into the final format for when they are presented as skills tests. The formatting adjustment unit 1324 may, for example, assign question numbers, standardize the symbols of the answer choices (A, B, C, D, etc.), or convert them to a specific data format such as HTML or XML.

[0062] The feedback unit 133 updates the parameters used by the generation unit 131 to generate questions based on the results of the skills test conducted, and continuously improves the accuracy and quality of question generation. The feedback unit 133 includes a statistical analysis unit 1331 and an effectiveness evaluation unit 1332.

[0063] The feedback unit 133 retrieves the job seeker's answer data (correct / incorrect answers for each question, answer time, etc.) transmitted from the job seeker terminal 200 and stored in the storage unit 120.

[0064] The statistical analysis unit 1331 performs analysis using statistical test theory (e.g., Item Response Theory (IRT) or Classical Test Theory (CTT)) based on the accumulated answer data. Through this analysis, the statistical analysis unit 1331 calculates objective indicators such as the difficulty level of each question (e.g., correct answer rate) and the discriminant index, which indicates how accurately the abilities of the test takers can be identified.

[0065] Item Response Theory (IRT) is a theory that focuses on the test-taker's response patterns to individual questions (items) rather than the total test score, and uses probabilistic models to estimate the test-taker's ability level and the characteristics of the questions.

[0066] In the IRT, the relationship between a test-taker's potential ability score (θ) and the probability that the test-taker will answer a particular question correctly is represented by a mathematical model (item characteristic curve) such as a logistic curve. This model incorporates item parameters as question-specific characteristic values, such as "difficulty (b parameter)" which represents the difficulty of the question, "discriminative power (a parameter)" which represents the steepness of the change in the probability of answering correctly due to ability differences, and "guessing (c parameter)" which represents the probability that a low-ability test-taker will answer correctly by chance.

[0067] Because it allows for the separation and treatment of the test-taker's ability scores from the parameters of the questions, it can be applied to comparing (equalizing) the abilities of test-takers who have taken different sets of questions, and to adaptive tests that optimize the questions to suit the abilities of individual test-takers.

[0068] Classical Test Theory (CTT) is a statistical theory that uses test scores (observed scores) as the basis for analysis, and it is a standard method used in the fields of psychometrics and educational measurement. In this theory, observed scores are defined as the sum of the test-taker's invariant "true score" and the "measurement error" that fluctuates randomly with each measurement.

[0069] CTT-based analysis calculates reliability coefficients using the variance and covariance of the entire test, and evaluates the characteristics of the test and questions using indicators such as the correct answer rate (pass rate) for each question and the correlation (discriminant index) between the score of a question and the total test score. Its features include a relatively simple calculation process and the use of the total score (raw score), which is an intuitively easy-to-understand indicator.

[0070] The effectiveness evaluation unit 1332 evaluates the effectiveness of each question based on the indicators calculated by the statistical analysis unit 1331. For example, it identifies questions that are too difficult or too easy, or questions with low discrimination indices. The feedback unit 133 then provides feedback to the generation parameters used by the generation unit 131 to generate the questions based on the evaluation results.

[0071] The feedback unit 133 generates information to adjust the rules and weights referenced by the basic generation unit 1312 and the incorrect answer specialization unit 1313, such as "generate questions with more thought steps from this type of keyword" or "avoid generating incorrect answers of this pattern because they have low discriminatory power," and stores this information in the memory unit 120. This improves the quality of question generation in subsequent attempts.

[0072] As described above, the skill test generation system 1 of this embodiment can efficiently provide high-quality and objective skill tests by automating a series of cycles including automatic question generation, quality assurance, and self-improvement through feedback.

[0073] Figure 3 is a diagram showing a basic flowchart of the skill test generation system 1 according to this embodiment. First, the processing unit 130 of the skill test generation server 100 starts the skill test generation process. The control unit 1300 built into the processing unit 130 (the processing unit 130 may also have the functions of the control unit 1300) instructs the generation unit 131 to generate questions related to the skill test for job seekers.

[0074] In response to this instruction, the generation unit 131 executes a process to generate questions related to the skills test based on the basic data entered by the recruiting party (step S102). The basic data here refers to information that includes at least one of the keywords entered regarding the content of the skills test to be generated, natural language text regarding the content of the skills test to be generated, and the job posting data from the recruiting party.

[0075] These basic data are transmitted, for example, from a terminal device (not shown) operated by a recruiting representative to the skill test generation server 100 via a network NW, received via the communication unit 110, and temporarily stored in the storage unit 120. The generation unit 131 reads these basic data from the storage unit 120 and uses them for the question generation process.

[0076] Next, the control unit 1300 instructs the quality assurance unit 132 to evaluate the quality of the questions generated by the generation unit 131. In response to this instruction, the quality assurance unit 132 executes a process to evaluate the quality of the generated questions (step S104). In the quality evaluation process, the difficulty level and appropriateness of the content of the questions are checked based on pre-set quality standards.

[0077] Once the quality evaluation process is complete, the control unit 1300 generates a final skill test based on the evaluation results from the quality assurance unit 132 (step S106). In this step, a predetermined number of questions that meet the quality standards, or questions that have been modified to meet the quality standards, are collected and configured into a single skill test. The generated skill test is stored in the storage unit 120 and distributed to the job seeker terminal 200 as needed.

[0078] Furthermore, the skill test generation system 1 includes a feedback mechanism to utilize the results of the skill test in future question generation. Specifically, when a job seeker takes a skill test using the job seeker terminal 200, their answer data is transmitted to the skill test generation server 100.

[0079] The feedback unit 133 updates the parameters that the generation unit 131 uses to generate questions based on this answer data. This feedback loop allows the system to be used repeatedly, enabling it to automatically generate higher quality and more effective questions.

[0080] Figure 4 shows a more detailed flowchart of the generation unit 131 according to this embodiment. This process is mainly performed in cooperation with the generation unit 131 and the quality assurance unit 132.

[0081] First, the control unit 1300 (control unit) of the skill test generation server 100 starts the skill test generation process (step S201). The control unit instructs the generation control unit 1311 within the generation unit 131 to start processing.

[0082] The generation control unit 1311 acquires basic data that will be used as the basis for generating questions (step S202). As mentioned above, the basic data includes keywords, natural language text, and job posting data provided by the recruiting party. For example, the job posting data includes the job title, required skills, and job description.

[0083] Next, the generation unit 131 generates a draft of the question using the acquired basic data (step S203). Specifically, this process is handled by the basic generation unit 1312 within the generation unit 131. The basic generation unit 1312 analyzes the contents of the basic data and generates a draft of the question that includes the question text, which forms the framework of the question, and the intent behind the question.

[0084] Next, the incorrect answer specialization unit 1313 within the generation unit 131 generates incorrect answer candidates based on the draft questions generated by the basic generation unit 1312 (step S204).

[0085] Furthermore, the explanation-specialized unit 1314 within the generation unit 131 generates an explanation text for test takers based on the draft question and the candidate incorrect answers (step S205). The explanation text includes the rationale for why a particular option is the correct answer and the reasons why each incorrect option is wrong.

[0086] Finally, the generation control unit 1311 integrates the draft question generated by the basic generation unit 1312, the candidate incorrect answers generated by the incorrect answer specialization unit 1313, and the explanatory text generated by the explanatory text specialization unit 1314 to complete it as a single question (step S206).

[0087] Once a question is generated, the process is transferred to the Quality Assurance Unit 132. The Quality Assurance Unit 132 evaluates the quality of the generated question (step S207). Details of this evaluation process will be described later.

[0088] The Quality Assurance Department 132 determines, based on the evaluation results, whether the quality of the question meets the predetermined standard (step S208). If the quality does not meet the standard (No in step S208), the Quality Assurance Department 132 either modifies the question or discards the question and returns to step S202 to attempt to generate a new question. If the Quality Assurance Department 132 modifies the question, it may reuse some of the previously generated questions when returning to step S202.

[0089] If the quality of the questions meets the criteria (Yes in step S208), the control unit determines whether the required number of questions for the skill test has been generated (step S209). If the required number of questions has not yet been generated (No in step S209), the process returns to step S202, and the generation of the next questions continues. If the required number of questions has been generated (Yes in step S209), the series of question generation processes ends, and the skill test consisting of these questions is completed.

[0090] Figure 5 is a more detailed flowchart of the quality assurance unit 132 according to this embodiment. The details of the quality evaluation and correction process in the quality assurance unit 132 (corresponding to steps S207 and S208) will be explained with reference to the flowchart.

[0091] The quality control unit 1321 within the quality assurance unit 132 inputs the questions received from the generation unit 131 into the quality check unit 1322 (step S301).

[0092] The quality check unit 1322 determines whether the input question is a pass or fail based on the quality standards pre-set in the memory unit 120 (step S302). The quality standards include items such as whether the difficulty level of the question is appropriate for the level of the target job seekers, whether there are any ambiguous expressions in the question text or answer choices, and whether it is technically accurate.

[0093] If the quality check unit 1322 determines that a question is unsatisfactory (No in step S303), the quality control unit 1321 passes the unsatisfactory question to the correction unit 1323. The correction unit 1323 automatically corrects the question based on the reason for failure from the quality check unit 1322 (for example, "difficulty level is too high" or "the wording of the answer choices is inconsistent") (step S304). The corrected question is then checked again by the quality check unit 1322 or is deemed acceptable based on the prescribed rules.

[0094] On the other hand, if the quality check unit 1322 determines that the question is acceptable (Yes in step S303), or if the question is modified in step S304, the quality control unit 1321 proceeds with the question to the next process.

[0095] If the quality assurance unit 132 has a format adjustment unit 1324, the quality control unit 1321 inputs the question that has been determined to pass, or the corrected question, into the format adjustment unit 1324 (step S305).

[0096] The formatting adjustment unit 1324 formats the input question into the final format in which it will be presented as an actual skill test question (step S306). This process includes, for example, converting text into a specific markup language such as HTML or XML, adding setting information to randomize the display order of the answer choices, and embedding related images and code snippets.

[0097] Once formatting is complete, the quality control unit 1321 outputs the question after final adjustments have been made (step S307). This outputted question is then confirmed as one of the elements that make up the skill test.

[0098] Next, the question generation process in the generation unit 131 (steps S203, S204, S205) will be explained in more detail.

[0099] The basic generation unit 1312, in generating draft questions (step S203), deeply analyzes basic data such as the job description advertised by the employer, the required skills for job seekers, and the industry trends to which the employer belongs. For example, if job posting data for a "front-end engineer" is input, it generates technical questions related to "React" and "JavaScript," which are listed as required skills. Also, if the job description is "UI implementation for web applications," it incorporates specific scenarios that might be encountered in actual development environments (e.g., problems related to component state management) into the question text.

[0100] The error-specialization unit 1313 performs the error-specialization process (step S204) based on sophisticated logic. The error-specialization unit 1313 is designed to behave like a psychology or educational technology expert and accepts a draft question, its correct answer, and a measurement objective (e.g., measuring understanding of a specific coding convention) as input information.

[0101] The error-generating unit 1313 designs "high-quality" incorrect answers that are not simply wrong, but are intentionally misleading to test-takers. Its principle is to generate at least one of the following types of options: First, an option that is partially correct but lacks a crucial perspective within the context of the question. Second, an option that is generally correct but inappropriate under the specific circumstances given in the question. Third, an option that relates to typical misconceptions or bad practices common among beginners in the technical field.

[0102] In particular, the incorrect answer specialization unit 1313 generates incorrect answer candidates based on "superficial similarity," which appears correct when only the superficial features of the situation are considered. For example, in a question about a programming language, the incorrect answer specialization unit 1313 may include the name of a function from another language that has a similar syntax but completely different functionality as an incorrect answer option. This effectively measures whether the test-taker has a deep understanding of the concept, rather than simply memorizing it. The configuration information table 630 shows examples of such incorrect answer design principles and output formats.

[0103] The explanation generation process by the explanation-specialized unit 1314 (step S205) is performed with the aim of maximizing the learning effect of the test taker. The explanation-specialized unit 1314 is designed to behave like an expert in learning design in corporate training, and accepts questions, correct answers, generated incorrect answers, and key points of the questions as input information.

[0104] The explanation-specialized unit 1314 generates a structured explanation containing at least one of the following components. The first component may be a logical and detailed explanation of the essential reason for the correct answer, i.e., why that option is the correct answer.

[0105] The second component is an analysis of why each incorrect answer is inappropriate. This involves not simply pointing out a "mistake," but specifically explaining what perspectives are lacking or what misunderstandings it is based on.

[0106] The third component may be application points that show how the theme of the question is applied in practical work.

[0107] The fourth component may be an introduction to the theoretical background or framework related to the question. The fifth component may be practical advice on common failure patterns in that topic and how to avoid them. Setting information table 640 shows examples of the components and output format of the explanatory text.

[0108] Next, the quality determination process (step S302) in the quality check unit 1322 will be explained in more detail.

[0109] The quality check unit 1322 functions as the quality manager for the questions, receiving the questions, their target objectives (skills to be measured), and the target job seeker group (e.g., junior level, senior level) as input information. It then performs evaluations based on a multifaceted checklist as shown in the setting information table 650.

[0110] The quality check unit 1322 evaluates the difficulty level of the questions. This evaluation is based on at least one of the following indicators. The first evaluation is the specialization of the terminology included in the question, which the quality check unit 1322 may determine is difficult if it compares the terminology with a dictionary of specialized terms stored in the memory unit 120 and determines that the more specialized terms a question contains, the higher the difficulty level.

[0111] The second evaluation may be the number of thinking steps required to answer the question. The quality check unit 1322 may analyze the logical structure of the question and the correct answer, and determine that the more logical leaps and judgments required to reach the correct answer, the more thinking steps are needed and the higher the difficulty level.

[0112] The third evaluation may be based on past answer data. The quality check unit 1322 may refer to statistical data such as the correct answer rate and answer time when similar questions have been asked in the past, and determine that questions with a low correct answer rate are of high difficulty.

[0113] Furthermore, the quality check unit 1322 performs more detailed quality checks. The checks include at least one of the following: The first check may be "measurement accuracy." It evaluates whether the questions purely measure only the skills set as measurement targets, or whether they are influenced by other unrelated knowledge (e.g., reading comprehension in Japanese).

[0114] The second check item can be "clarity of the correct answer." This evaluates whether the correct answer to the question is uniquely determined without any room for interpretation, or whether the question is ambiguous and not one where even experts disagree.

[0115] The third check item can be "the functionality of the answer choices." Evaluate whether each incorrect answer choice reflects a different judgment axis or pattern of misunderstanding, and whether it is merely a dummy choice used to fill out the numbers.

[0116] The fourth check item can be "realism." Evaluate whether the scenario set in the question is a realistic situation that could occur in actual work, or whether it is unrealistic and purely theoretical.

[0117] The fifth check item can be "judgment criteria." This evaluates whether the judgment required in the question aligns with the criteria for good decision-making in practice and whether it deviates from practical experience.

[0118] The quality check unit 1322 comprehensively evaluates these check items and may determine "pass" only if all items meet the criteria. If even one item fails to meet the criteria, it will determine "fail" and notify the quality control unit 1321 of the reason (e.g., "Low measurement accuracy: Knowledge beyond the required skills is needed").

[0119] Next, the question correction process (step S304) in the correction unit 1323 will be explained in more detail.

[0120] The modification unit 1323 receives the questions that were deemed unacceptable from the quality check unit 1322, along with the reasons for their failure. The modification unit 1323 then modifies the questions according to the reasons for their failure.

[0121] The modification section 1323 makes at least one of the following modifications to adjust the difficulty level of the question. The first modification may be a change in expression. For example, if the difficulty level is determined to be too high, the first modification may be to replace technical terms with simpler words or to divide a complex sentence into several shorter sentences. Conversely, if the difficulty level is too low, the first modification may be to modify it with more technical expression.

[0122] The second type of revision may involve content substitution. If the scenario in the question is deemed inappropriate, it may be replaced with a more realistic and appropriate scenario. Alternatively, if the correct or incorrect answer choices are inappropriate, they may be replaced with more appropriate content.

[0123] The third modification may be the deletion of a question. If modification is deemed difficult, or if the question itself is deemed to have low value, the method may be to discard the question and encourage the generation of a new question.

[0124] Furthermore, the modification unit 1323 may perform more specific modification processes depending on the reason for failure. For example, if the reason for failure is "inconsistent writing style of the options," the modification unit 1323 may make modifications to unify the sentence-ending expressions of all options to "~is" or "~doing."

[0125] If the reason for failure is "the incorrect answer is not appealing enough," the correction unit 1323 may instruct the incorrect answer specialization unit 1313 again to regenerate a more sophisticated incorrect answer. Also, if the reason for failure is "the explanation is insufficient," the correction unit 1323 may instruct the explanation specialization unit 1314 to reinforce the content of a specific item (for example, "points of practical application"). These corrections improve the quality of the questions and increase the likelihood of meeting the standards of the quality check unit 1322.

[0126] Finally, the parameter update process in the feedback unit 133 will be explained in detail. This process forms the core of the self-learning cycle for continuously improving the quality of the generated questions.

[0127] Once a skills test is generated and taken by one or more job seekers via the job seeker terminal 200, each test taker's answer data (which option they selected for which question, whether it was correct, the time taken to answer, etc.) is transmitted to the skills test generation server 100 via the network NW.

[0128] The communication unit 110 receives this answer data and passes it to the feedback unit 133. The feedback unit 133 stores the received answer data in the storage unit 120.

[0129] Next, the statistical analysis unit 1331 within the feedback unit 133 performs a statistical analysis based on a sufficient amount of answer data stored in the memory unit 120. The statistical analysis includes statistical test theory, particularly Item Response Theory (IRT). Using this theory, the statistical analysis unit 1331 calculates parameters such as the "difficulty level" parameter (showing the relationship between the test-taker's ability and the probability of answering correctly) and the "discrimination index" parameter (showing how accurately the test-taker's ability can be identified) for each question.

[0130] Next, the effectiveness evaluation unit 1332 within the feedback unit 133 may evaluate the effectiveness of each question based on various indicators calculated by the statistical analysis unit 1331. For example, the effectiveness evaluation unit 1332 identifies questions where the difficulty level assumed during generation deviates significantly from the difficulty parameter calculated from the actual answer data, or questions with an extremely low discrimination index (so-called "bad questions" that both highly capable and less capable test-takers make the same mistakes on).

[0131] Based on the evaluation results, the effectiveness evaluation unit 1332 may provide feedback to the generation parameters used by the generation unit 131 to generate the questions. For example, if the evaluation result is obtained that "questions containing certain keywords (e.g., Async / Await) tend to be more difficult than expected," the effectiveness evaluation unit 1332 will instruct the generation unit 131 to update the generation parameters in the form of "reducing the number of thinking steps by one when using the keyword 'Async / Await'" or "using simpler synonyms."

[0132] This feedback and parameter update cycle is repeated, allowing the skill test generation system 1 to automatically improve the accuracy of question generation. This makes it possible to efficiently generate high-quality skill tests that accurately assess the skills to be measured, while minimizing adjustments by human experts.

[0133] Figure 6 shows an example of the configuration of the keyword data table 600 according to this embodiment. The keyword data table 600 is a table that manages keyword information that serves as the starting point for generating questions for the skill test. This table includes the fields "Data ID", "Keyword", and "Related Category".

[0134] The "Data ID" is an identifier used to uniquely identify each keyword. The "Keyword" stores specific technical terms or skill names related to the theme of the question (e.g., "Python," "Project Management," etc.). The "Related Category" indicates the broad category to which the keyword belongs (e.g., "Programming Language," "Business Skills," etc.).

[0135] The generation unit 131 refers to this table and determines keywords that will be the theme of the questions based on the specified categories or random selections. Alternatively, the generation unit 131 may determine keywords based on a pre-set difficulty baseline for each keyword.

[0136] Figure 7 shows an example of the configuration of the natural language text data table 610 according to this embodiment. The natural language text data table 610 is a table that manages unstructured text data that is referenced when generating the question text, the basis for the correct answer, or the explanatory text. This table includes the fields "Data ID", "Text Content", and "Source / Type".

[0137] A "data ID" is an identifier used to uniquely identify data.

[0138] "Text content" includes natural language text such as the main body of technical documents, the content of explanatory articles, or excerpts from textbooks. "Source / Type" indicates the source and type of the information. The generation unit 131 uses RAG (Retrieval-Augmented Generation) technology, etc., to extract text related to the theme of the question from this table and inputs this as context into the LLM to generate factual and accurate questions and explanations.

[0139] Figure 8 shows an example of the configuration of the job posting data table 620 according to this embodiment. The job posting data table 620 is a table that manages the requirements definition (basic data) of personnel that a company is seeking. This table includes the fields "Data ID", "Job Title", "Required Skills", and "Job Description".

[0140] A "data ID" is an identifier used to uniquely identify data.

[0141] The "Job Title" field stores the name of the job being advertised. "Required Skills" includes the specific skills and tools required for that job. "Job Description" includes the specific tasks expected after joining the company. The Quality Check Department 1322 refers to the information in this table to determine whether the difficulty level of the generated questions is appropriate for the target (job title and job description). For example, if the job posting is for a "Data Scientist," then difficult questions that test not just terminology knowledge but also the actual thought process of "model building" will be required.

[0142] Figure 9 shows an example of the configuration of the setting information table 630 referenced by the error-specialization unit 1313 according to this embodiment. The setting information table 630 is prompt configuration information that defines the behavior of the error-specialization unit 1313 (AI agent). This table includes "task definition," "input information," "principles of error design," "type of cognitive trap," "output format," and "quality criteria."

[0143] The "task definition" in the prompt may include information about the so-called persona to be set for the generating AI. The "task definition" may also include, for example, that the persona is an expert in a particular field.

[0144] "Input information" may include information about the question, the correct answer, and the measurement target.

[0145] The "Principles of Designing Incorrect Answers" define guidelines for creating high-quality incorrect answers, such as "options that are partially correct but lack a crucial perspective."

[0146] The "types of cognitive traps" define specific methods for testing test-takers, such as "superficial similarity" and "short-term perspective." By adjusting the degree to which these traps are applied, it is possible to control the difficulty level of the questions.

[0147] The "Output Format" may specify the format of the output. The format may include information on incorrect answer choices, explanations of superficial similarities, reasons why the choices are inappropriate, and partially correct elements.

[0148] Figure 10 shows an example of the configuration of the setting information table 640 referenced by the explanation specialization unit 1314 according to this embodiment. The setting information table 640 is prompt configuration information that defines the behavior of the explanation specialization unit 1314 (AI agent). This table includes "task definition," "input information," "component elements of explanation," "tone & manner," and "output format."

[0149] The "task definition" in the prompt may include information about the so-called persona to be set for the generating AI. The "task definition" may also include, for example, that the persona is an expert in a particular field.

[0150] "Input information" may include the entire question, the correct answer and its reasoning, and information about incorrect answers and their problems.

[0151] The "components of the explanation" are required to include not only the "essential reason for the correct answer," but also the "reasons why each incorrect answer is inappropriate," "points for practical application," and "related theories and frameworks." By providing detailed explanations, even difficult questions can be understood by test-takers, and the learning effect can be improved.

[0152] "Tone and manner" may include information such as being non-preachy, offering practical advice, promoting positive learning, and minimizing the use of technical jargon.

[0153] The "output format" may include explanations of why the options are inappropriate, the perspectives missing from the correct answer, and explanations of misunderstandings. It may also include important points to learn from this question and practical considerations.

[0154] Figure 11 shows an example of the configuration of the setting information table 650 referenced by the quality check unit 1322 and the correction unit 1323 according to this embodiment. The setting information table 650 is definition information for controlling the operation of the quality assurance unit 132. This table includes "task definition," "input," "check items," "output," and "correction rules."

[0155] The "task definition" in the prompt may include information about the so-called persona to be set for the generating AI. The "task definition" may also include, for example, that the persona is an expert in a particular field.

[0156] The "check items" include not only formal aspects but also "measurement accuracy," "clarity of correct answers," "function of choices," and practical quality aspects such as "realism" and "judgment criteria." In addition to these items, the quality check unit 1322 determines the appropriateness of the difficulty level based on the input information of the "target audience." For example, if the target audience is a beginner, questions that ask about "judgment criteria" that are too specialized will fail and will be corrected by the correction unit 1323.

[0157] The "output" may include reasons for the need for revision, such as deviation from the set objective, non-functional options, exceeding the word count, or problems with the question. The "output" may include revised questions, revised correct answers, and revised explanations for the revised content. The "output" may include revision scenarios, such as removal of redundant explanations or reduction of cognitive load.

[0158] The "revision rules" stipulate that formal revisions must be made, and substantive revisions should be proposed with justification, respecting the original intent while including information such as quality improvements.

[0159] Figure 12 is a functional block diagram of the skill test generation server 100, showing another example according to this embodiment. In this example, in addition to the configuration shown in Figure 2, the skill test generation server 100 further includes an acquisition unit 134 and a prompt generation unit 135 as functional units that perform pre-processing of the generation unit 131.

[0160] The acquisition unit 134 acquires basic data entered by the employer regarding questions related to the skills test, which is a test for job seekers. The acquired basic data is either stored in the storage unit 120 or passed directly to the prompt generation unit 135.

[0161] The prompt generation unit 135 generates prompts used by the generation unit 131 in the generation model (e.g., a large-scale language model) from the basic data acquired by the acquisition unit 134. The prompt generation unit 135 embeds information such as "required skills" and "job description" included in the job posting data into the format of "task definition" and "input information," and constructs specific instruction statements (system prompts and user prompts) for the generation model.

[0162] The prompt generation unit 135 may generate prompts by obtaining prompts that have been pre-generated from an external source. The prompts may also include the content of basic data.

[0163] In this case, the generation unit 131 inputs the prompt generated by the prompt generation unit 135 into the generation model, and generates questions, incorrect answer candidates, and explanatory text based on the generation results output from the generation model. The functions of the quality assurance unit 132, the feedback unit 133, and the other units are the same as described above, so a detailed explanation is omitted here. The control unit 1300 also provides overall control over the coordination of these functional units.

[0164] Figure 13 is a block diagram showing an example of the hardware configuration of a computer that functions as a skill test generation server 100 and a job seeker terminal 200 according to this embodiment. The computer 900 has a control unit 901, a storage unit 902, a communication unit 903, an input unit 904, and an output unit 905. The control unit 901, storage unit 902, communication unit 903, input unit 904, and output unit 905 are electrically connected to each other via a communication bus 910.

[0165] The control unit 901 includes a CPU (Central Processing Unit, also called a processor) and controls various parts of the computer 900, as well as reading and executing various programs stored in the storage unit 902.

[0166] The memory unit 902 includes a main memory such as DRAM (Dynamic Random Access Memory) and an auxiliary memory such as a hard disk, and is a device for storing various programs for running the operating system and various applications of the computer 900, as well as data used by these programs. The processes shown in the flowcharts described above are realized by the control unit 901 of each computer executing the programs stored in its respective memory unit 902.

[0167] The communication unit 903 is a device for communicating with external devices and sends and receives data according to instructions from the control unit 901. Each computer uses this communication unit 903 to communicate with other devices, including the network NW shown in Figure 1.

[0168] The input unit 904 is a device that receives input from an external source and supplies it to the control unit 901, and includes, for example, a keyboard, mouse, touch panel, and camera. The output unit 905 is a device that outputs the processing results of the control unit 901 to the outside, and includes, for example, a display and speaker.

[0169] <Program> Here, we will describe the programs for realizing each functional unit of the skill test generation server 100 according to this embodiment.

[0170] The skill test generation server 100 is implemented on the computer 900. The operation of each component of the skill test generation server 100 is stored in the auxiliary storage device of the storage unit 902 in the form of a program. The control unit 901 reads the program from the auxiliary storage device of the storage unit 902, expands it into the main memory of the storage unit 902, and executes the above processing according to the program. The control unit 901 also reserves a storage area in the main memory of the storage unit 902 corresponding to the storage unit 120 described above, according to the program.

[0171] Specifically, the program comprises a computer 900 comprising: a generation unit 131 that generates questions related to a skills test for job seekers based on basic data entered by the employer recruiting the job seekers; a quality assurance unit 132 that evaluates the quality of the questions generated by the generation unit 131; a feedback unit 133 that updates the parameters used by the generation unit 131 to generate the questions based on the results of the skills test; and a control unit 1300 that causes the generation unit 131, the quality assurance unit 132, and the feedback unit 133 to generate the skills test. The basic data includes keywords and raw data entered regarding the content of the skills test to be generated. The skill test generation program includes at least one of the following: natural language text relating to the content of the skill test to be made and job posting data from the recruiting party. The quality assurance unit 132 is characterized by functioning as a quality control unit 1321 that inputs the questions generated by the generation unit 131 into the quality check unit 1322 to determine whether the difficulty level of the questions is pass or fail, and if the question is failing, inputs the failing question into the correction unit 1323 to output the question with the corrected difficulty level.

[0172] The auxiliary storage device of the memory unit 902 is an example of a tangible medium that is not temporary. Other examples of tangible mediums that are not temporary include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memory connected via the input unit 904. Furthermore, if this program is distributed to the computer 900 via the network NW, the computer 900 that receives the program may load it into the main memory of the memory unit 902 and execute the above processing.

[0173] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in the auxiliary storage device of the memory unit 902.

[0174] <Note> (Note 1) A generation unit that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality control unit inputs the question generated by the generation unit and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question to the correction unit and outputs the question with the difficulty level corrected. A skill test generation system having the following features.

[0175] (Note 2) The modification unit modifies the difficulty level of the question by changing the wording of the question, replacing the content of the question, or deleting the question, as described in Appendix 1 of the skill test generation system.

[0176] (Note 3) The skill test generation system described in Appendix 1 or 2, wherein the quality check unit evaluates the difficulty level based on at least one of the specialization of the terms included in the question, the number of thinking steps required to answer the question, and past answer data.

[0177] (Note 4) The basic data includes information such as the job description offered by the recruiting party, the skills required by the recruiting party for the job seeker, and the industry trends to which the recruiting party belongs, as described in any of the appendices 1 to 3 of the skill test generation system.

[0178] (Note 5) The generating unit is A basic generation unit that generates a draft question including the question text that forms the framework of the aforementioned question and the intent behind the question, An error-specializing unit that generates candidate incorrect answers for the aforementioned draft question, A specialized explanation unit that generates an explanation text including the basis for the correct answer and the reason for the incorrect answer corresponding to the aforementioned draft question, The generation control unit generates the basic questions by inputting the basic data into the basic generation unit, generates the incorrect answer candidates by inputting the incorrect answer specialized unit, generates the explanation text by inputting the correct question and the incorrect answer candidates into the explanation text, and generates the questions based on the correct question, the incorrect answer candidates and the explanation text. A skill test generation system, including any of the systems described in Appendix 1 to 4.

[0179] (Note 6) The aforementioned Quality Assurance Department The system includes a formatting adjustment unit that formats the aforementioned questions into the final format used when presenting them in the skills test. The quality control unit, A skill test generation system according to any one of the appendices 1 to 5, wherein the format adjustment unit receives the question that the quality check unit has determined to pass, or the question after the correction unit has made corrections, and outputs the question after final adjustments.

[0180] (Note 7) The correction unit performs at least one of the following correction processes based on the reason for failure: unifying the sentence-ending expressions of the questions, regenerating incorrect answers to the questions, or reinforcing the explanatory text for the questions, as described in any of the appendices 1 to 6 of the skill test generation system.

[0181] (Note 8) The aforementioned feedback unit is A memory unit for storing answer data of the job seeker who took the aforementioned skills test, A statistical analysis unit that calculates the difficulty level of the question or the discrimination index of the question using statistical test theory based on the answer data, The effectiveness evaluation unit evaluates the effectiveness of the question based on the calculated indicators and provides feedback to the generation parameters used in generating the question in the generation unit. A skill test generation system having any of the features described in Appendix 1 to 7.

[0182] (Note 9) The aforementioned incorrect answer specialization unit accepts input information including the question, the correct answer to the question, and the measurement target of the question, and, in principle, generates the incorrect answer candidates based on superficial similarity that is likely to be selected based only on the superficial characteristics of the situation, with the incorrect answer design being based on the principle that the options include at least one of the following: an option that is partially correct, an option that is generally correct but inappropriate in the situation of the question, and an option that relates to a typical misunderstanding, and is likely to be selected based only on the superficial characteristics of the situation. (Skill test generation system as described in Appendix 5)

[0183] (Note 10) The aforementioned specialized explanation unit accepts input information including all of the aforementioned questions, the correct answers to the questions, the incorrect answers to the questions, and the problems, and generates the explanation text using at least one of the following as components: the essential reason for the correct answer to the question, the reason why the incorrect answer to the question is inappropriate, points for practical application, theories and frameworks related to the question, and frequently occurring failure patterns and methods for avoiding them, as described in Appendix 5 of the skill test generation system.

[0184] (Note 11) The quality check unit receives input information including the question, the target setting for the question, and the target group for the question, and checks at least one of the following: measurement accuracy, which indicates whether only the target skills required of the job seeker are being judged; clarity of correct answer, which indicates whether there is room for interpretation as to whether the answer to the question is correct or not; functionality of the options, which indicates whether each option of the question reflects a different judgment axis; realism, which indicates whether the situation in the question is a situation that could actually occur; and judgment criteria, which indicates whether the question is consistent with decision-making in practical work, and determines whether it is a pass or a fail, as described in any of the appendices 1 to 10.

[0185] (Note 12) Regarding the questions related to the skills test, which is a test for job seekers, the acquisition unit acquires basic data entered by the employer who is recruiting the job seekers, A prompt generation unit that generates prompts used in the generation model from the aforementioned basic data, A generation unit inputs the aforementioned prompt into the generation model and generates a question, which is the generation result, from the generation model. A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality control unit inputs the question generated by the generation unit and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question to the correction unit and outputs the question with the difficulty level corrected. A skill test generation system having the following features.

[0186] (Note 13) A generation unit that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality control unit inputs the question generated by the generation unit and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question to the correction unit and outputs the question with the difficulty level corrected. A skill test generation server with the following features.

[0187] (Note 14) A generation step that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance step which evaluates the quality of the questions generated by the generation step, A feedback step updates the parameters used by the generation step to generate the questions based on the results of the skills test, The generation step, the quality assurance step, and the feedback step are further controlled by a control step that causes the skill test to be generated. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The aforementioned quality assurance step is: A quality check step that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction step to modify the difficulty level of the aforementioned question, A quality control step which inputs the question generated in the generation step into the quality check step and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question into the correction step and outputs the question with the difficulty level corrected. A method for generating skill tests that includes the following:

[0188] (Note 15) Computers, A generation unit that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, A skill test generation program characterized in that it functions as a quality control unit, which inputs the questions generated by the generation unit into the quality check unit and determines whether the difficulty level of the questions is passable or failing, and if the questions are failing, inputs the failing questions into the correction unit and outputs the questions with corrected difficulty levels. [Explanation of symbols]

[0189] 1: Skill Test Generation System 100: Skill Test Generation Server 110: Communications Department 120: Storage section 130: Processing Unit 131: Generation part 132: Quality Assurance Department 133: Feedback Department 134: Acquisition Department 135: Prompt generation unit 200: Job seeker terminal 600: Keyword Data Table 610: Natural Language Text Data Table 620: Job posting data table 630: Configuration Information Table 640: Configuration Information Table 650: Configuration Information Table 900: Computer 901: Control Unit 902: Storage section 903: Communications Department 904: Input section 905: Output section 910: Bus 1300: Control Unit 1311: Generation Control Unit 1312 :Basic generator 1313: Incorrect Answer Specialization Department 1314: Specialized Commentary Section 1321: Quality Control Department 1322: Quality Check Department 1323: Correction section 1324: Format adjustment section 1331:Statistical Analysis Department 1332: Effectiveness Evaluation Department

Claims

1. A generation unit that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The generation unit includes a basic generation unit that generates a draft question including a question statement and the intent of the question which form the framework of the question; an incorrect answer specialized unit that generates incorrect answer candidates which are candidates for incorrect answers to the draft question; an explanation specialized unit that generates an explanation text including the basis for the correct answer and the reason for the incorrect answer corresponding to the draft question; a generation control unit that inputs the basic data to the basic generation unit to generate the draft question, inputs the draft question to the incorrect answer specialized unit to generate the incorrect answer candidates, inputs the draft question and the incorrect answer candidates to the explanation specialized unit to generate the explanation text, and integrates the draft question, the incorrect answer candidates, and the explanation text to generate the question. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality control unit inputs the question generated by the generation unit and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question to the correction unit and outputs the question with the difficulty level corrected. It has, The feedback unit includes a storage unit for storing answer data of the job seeker who took the skills test, a statistical analysis unit for calculating the difficulty level of the questions or the discrimination index of the questions using statistical test theory based on the answer data, and an effectiveness evaluation unit for evaluating the effectiveness of the questions based on the calculated indicators and providing feedback to the generation parameters used in generating the questions in the generation unit. The control unit controls the quality check unit to evaluate the difficulty level based on at least one of the specialization of the terms included in the question, the number of thinking steps required to answer the question, and past answer data. The control unit controls the modification unit to modify the difficulty level of the question by changing the wording of the question, replacing the content of the question, and deleting the question. The control unit controls the feedback unit to update the generation parameters used in generating the questions in the generation unit, in a skill test generation system.

2. The skill test generation system according to claim 1, wherein the basic data includes information that includes at least one of the job description offered by the recruiting party, the skills required by the recruiting party for the job seeker, and the industry trends to which the recruiting party belongs.

3. The aforementioned Quality Assurance Department The system includes a formatting adjustment unit that formats the aforementioned questions into the final format used when presenting them in the skills test. The quality control unit, The skill test generation system according to claim 1, wherein the format adjustment unit receives the question that the quality check unit has determined to be acceptable, or the question after the correction unit has made corrections, and outputs the question after final adjustments.

4. The skill test generation system according to claim 3, wherein the modification unit performs at least one of the following modification processes based on the reason for failure: unifying the sentence-ending expressions of the questions, regenerating the incorrect answers to the questions, or reinforcing the explanatory text to the questions.

5. The skill test generation system according to claim 1, wherein the error-specialization unit accepts input information including the question, the correct answer to the question, and the measurement target of the question, and in principle generates error-specialization units that include at least one of the following: an option that is partially correct, an option that is generally correct but inappropriate in the context of the question, and an option that relates to a typical misunderstanding, and generates error-specialization candidates based on superficial similarities that are likely to be selected based only on the superficial characteristics of the situation.

6. The skill test generation system according to claim 1, wherein the explanation-specialized unit accepts input information including all of the questions, the correct answers to the questions, the incorrect answers to the questions, and the problems, and generates the explanation text using at least one of the essential reasons for the correct answers to the questions, the reasons why the incorrect answers to the questions are inappropriate, points for practical application, theories and frameworks related to the questions, and frequently occurring failure patterns and methods for avoiding them as constituent elements.

7. The skill test generation system according to claim 1, wherein the quality check unit receives input information including the question, the target setting for the question, and the target group for the question, and checks at least one of the following: measurement accuracy indicating whether only the target skills required of the job seeker are being judged; clarity of correct answer indicating whether there is room for interpretation as to whether the answer to the question is correct or not; functionality of the options indicating whether each option of the question reflects a different judgment axis; realism indicating whether the situation in the question is a situation that could actually occur; and judgment criteria indicating whether the question is consistent with decision-making in practical work.

8. Regarding the questions related to the skills test, which is a test for job seekers, the acquisition unit acquires basic data entered by the employer who is recruiting the job seekers, A prompt generation unit that generates prompts used in the generation model from the aforementioned basic data, A generation unit inputs the prompt into the generation model and generates the question, which is the generation result, from the generation model. A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The prompt generation unit embeds the information contained in the basic data into the format of the task definition and input information, and constructs system prompts and user prompts for the generation model. The generation unit inputs the prompt to the generation model and, based on the generation results output from the generation model, integrates the draft question, the candidate incorrect answer, and the explanatory text to generate the question. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality control unit inputs the question generated by the generation unit and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question to the correction unit and outputs the question with the difficulty level corrected. It has, The feedback unit includes a storage unit for storing answer data of the job seeker who took the skills test, a statistical analysis unit for calculating the difficulty level of the questions or the discrimination index of the questions using statistical test theory based on the answer data, and an effectiveness evaluation unit for evaluating the effectiveness of the questions based on the calculated indicators and providing feedback to the generation parameters used in generating the questions in the generation unit. The control unit controls the quality check unit to evaluate the difficulty level based on at least one of the specialization of the terms included in the question, the number of thinking steps required to answer the question, and past answer data. The control unit controls the modification unit to modify the difficulty level of the question by changing the wording of the question, replacing the content of the question, and deleting the question. The control unit controls the feedback unit to update the generation parameters used in generating the questions in the generation unit, in a skill test generation system.

9. A generation unit that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. Equipped with, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The generation unit includes a basic generation unit that generates a draft question including a question statement and the intent of the question which form the framework of the question; an incorrect answer specialized unit that generates incorrect answer candidates which are candidates for incorrect answers to the draft question; an explanation specialized unit that generates an explanation text including the basis for the correct answer and the reason for the incorrect answer corresponding to the draft question; a generation control unit that inputs the basic data to the basic generation unit to generate the draft question, inputs the draft question to the incorrect answer specialized unit to generate the incorrect answer candidates, inputs the draft question and the incorrect answer candidates to the explanation specialized unit to generate the explanation text, and integrates the draft question, the incorrect answer candidates, and the explanation text to generate the question. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality control unit inputs the question generated by the generation unit and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question to the correction unit and outputs the question with the difficulty level corrected. It has, The feedback unit includes a storage unit for storing answer data of the job seeker who took the skills test, a statistical analysis unit for calculating the difficulty level of the questions or the discrimination index of the questions using statistical test theory based on the answer data, and an effectiveness evaluation unit for evaluating the effectiveness of the questions based on the calculated indicators and providing feedback to the generation parameters used in generating the questions in the generation unit. The control unit controls the quality check unit to evaluate the difficulty level based on at least one of the specialization of the terms included in the question, the number of thinking steps required to answer the question, and past answer data. The control unit controls the modification unit to modify the difficulty level of the question by changing the wording of the question, replacing the content of the question, and deleting the question. The control unit controls the feedback unit to update the generation parameters used in generating the questions in the generation unit, and is a skill test generation server.

10. A computer, A generation step that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance step which evaluates the quality of the questions generated by the generation step, A feedback step updates the parameters used by the generation step to generate the questions based on the results of the skills test, The generation step, the quality assurance step, and the feedback step are further controlled by a control step that causes the skill test to be generated. Execute, The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The generation step includes a basic generation step that generates a draft question including a question statement and the intent of the question which form the framework of the question; an incorrect answer specialization step that generates incorrect answer candidates which are candidates for incorrect answers to the draft question; an explanation specialization step that generates an explanation text including the basis for the correct answer and the reason for the incorrect answer corresponding to the draft question; a generation control step that inputs the basic data to the basic generation step to generate the draft question, inputs the draft question to the incorrect answer specialization step to generate the incorrect answer candidates, inputs the draft question and the incorrect answer candidates to the explanation specialization step to generate the explanation text, and integrates the draft question, the incorrect answer candidates, and the explanation text to generate the question. The aforementioned quality assurance step is: A quality check step that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction step to modify the difficulty level of the aforementioned question, A quality control step which inputs the question generated in the generation step into the quality check step and determines whether the difficulty level of the question is acceptable or unacceptable, and if the question is unacceptable, inputs the unacceptable question into the correction step and outputs the question with the difficulty level corrected. Includes, The feedback step includes a memory step of accumulating answer data of the job seeker who took the skills test, a statistical analysis step of calculating the difficulty level of the questions or the discrimination index of the questions using statistical test theory based on the answer data, and an effectiveness evaluation step of evaluating the effectiveness of the questions based on the calculated indicators and providing feedback to the generation parameters used in generating the questions in the generation step. The control step controls the quality check step to evaluate the difficulty level based on at least one of the specialization of the terminology included in the question, the number of thinking steps required to answer the question, and past answer data. The control step controls the modification step to modify the difficulty level of the question by changing the wording of the question, replacing the content of the question, and deleting the question. A skill test generation method, wherein the control step controls the feedback step to update the generation parameters used to generate the questions in the generation step.

11. Computers, A generation unit that generates questions for a skills test, which is a test for job seekers, based on basic data entered by the employer who is recruiting the job seekers, A quality assurance unit that evaluates the quality of the questions generated by the generation unit, A feedback unit updates the parameters used by the generation unit to generate the questions based on the results of the aforementioned skill test, The generation unit, the quality assurance unit, and the feedback unit are configured to include a control unit that generates the skill test. and make it work The basic data includes at least one of the keywords entered regarding the content of the skill test to be generated, natural language text regarding the content of the skill test to be generated, and the job posting data from the recruiting party. The generation unit functions as a generation control unit that generates a draft question including the question text and the intent of the question which form the framework of the question; an incorrect answer specialized unit that generates incorrect answer candidates which are candidates for incorrect answers to the draft question; an explanation specialized unit that generates an explanation text including the basis for the correct answer and the reason for the incorrect answer corresponding to the draft question; inputs the basic data to the basic generation unit to generate the draft question; inputs the draft question to the incorrect answer specialized unit to generate the incorrect answer candidates; inputs the draft question and the incorrect answer candidates to the explanation specialized unit to generate the explanation text; and integrates the draft question, the incorrect answer candidates, and the explanation text to generate the question. The aforementioned Quality Assurance Department A quality check unit that determines whether the difficulty level of the aforementioned question is pass or fail based on pre-set quality standards, A correction unit that modifies the difficulty level of the aforementioned question, The quality check unit is configured to input the question generated by the generation unit and determine whether the difficulty level of the question is acceptable or unacceptable. If the question is unacceptable, the correction unit is configured to input the unacceptable question and output the question with the difficulty level corrected. The feedback unit functions as a memory unit that stores answer data of the job seeker who took the skills test, a statistical analysis unit that calculates the difficulty level of the questions or the discrimination index of the questions using statistical test theory based on the answer data, and an effectiveness evaluation unit that evaluates the effectiveness of the questions based on the calculated indicators and provides feedback to the generation parameters used in generating the questions in the generation unit. The control unit controls the quality check unit to evaluate the difficulty level based on at least one of the specialization of the terms included in the question, the number of thinking steps required to answer the question, and past answer data. The control unit controls the modification unit to modify the difficulty level of the question by changing the wording of the question, replacing the content of the question, and deleting the question. The control unit is a skill test generation program that causes the computer to function such that the feedback unit updates the generation parameters used in generating the questions in the generation unit.

Images