Information processing method, program, and information processing device

Abstract

To provide a novel system that enables a user to grasp at least either usage skill of a search tool or generative AI, in an examination in which usage of at least either the search tool or the generative AI is allowed.SOLUTION: An information processing method carried out by an information processing device is configured to: obtain a reproduction request of an answer input into the processing device by the user in response to a prescribed question related to measuring the skill of an engineer; after the reproduction request is obtained, obtain first log information related to the user's answer to the prescribed question, second log information related to the request input in the search tool or the generative AI when the user answers the question, and third log information related to a response output by the search tool or the generative AI; and control a reproduction of an answer input, a request input, and a response output, based on the first log information, the second log information, and the third log information.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing apparatus.

Background Art

[0002] Conventionally, a technique for evaluating engineers' skills online has been known. For example, Patent Document 1 discloses conducting an online test for measuring engineers' programming skills.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, in order to evaluate engineers' skills, the number of companies conducting tests for skill measurement has been increasing. By conducting tests for skill measurement, it becomes possible to reduce mismatches in engineer recruitment.

[0005] Furthermore, in general engineer skill measurement tests, in order to prevent fraud, examinees are prohibited from using search tools and generative AIs during the test. In contrast, the inventors conceived that by deliberately permitting examinees to use a search tool or a generative AI, the skill of using the search tool or the generative AI can be made an evaluation target.

[0006] Therefore, the disclosed technology aims to provide a new mechanism that enables grasping the skill of using a search tool or a generative AI in a test where the use of the search tool or the generative AI is permitted.

Means for Solving the Problems

[0007] In one aspect of the disclosed technology, an information processing method includes an information processing apparatus obtaining a reproduction request for an answer input by a user to a predetermined problem related to engineer skill measurement, obtaining, after obtaining the reproduction request, first log information regarding the user's answer to the predetermined problem, second log information regarding a request input by the user to a search tool or a generative AI when answering the problem, and third log information regarding a response output by the search tool or the generative AI, and controlling reproduction of the input of the answer, the input of the request, and the output of the response based on the first log information, the second log information, and the third log information.

Advantages of the Invention

[0008] According to the present invention, a user (for example, a company conducting a test for engineer skill measurement) can grasp the usage skill of a search tool or a generative AI of a test taker in a test where the use of the search tool or the generative AI is permitted.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are examples for explaining the present invention, and are not intended to limit the present invention only to those embodiments. Further, the present invention can be variously modified without departing from its gist. Furthermore, the same reference numerals are attached to the same components in each drawing as much as possible, and duplicate explanations are omitted as much as possible.

[0011] <System Outline> FIG. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment of the present invention. In the example shown in FIG. 1, the information processing system 1 is configured to include, for example, an information processing apparatus 10, an information processing apparatus 20, a server 30, and a network N. Also, the number of the information processing apparatus 10, the information processing apparatus 20, and the server 30 may be one or more.

[0012] The information processing system 1 is used, for example, by a user (hereinafter also referred to as an operator) who creates a test related to the skill measurement of engineers, a user who takes the test (hereinafter also referred to as a test taker), and a user who conducts the test or evaluates the test results (hereinafter also referred to as an evaluator or a company). Also, the operator and the evaluator may be the same user. When a user who conducts the test or evaluates the test results is referred to as a "company", the "company" may include at least one of a company that wants to employ engineers and a company that wants to measure the skills of in-house engineers.

[0013] The information processing system 1 generates a coding test in a predetermined programming language in response to a request from, for example, an operator or an evaluator. The predetermined programming language includes at least one of known programming languages such as JavaScript (registered trademark), Python, script languages such as the C language, Go language, Swift, and Kotlin (registered trademark). Note that the test for measuring the skills of engineers is not limited to the coding test, and may also include questions that ask about knowledge related to coding, questions that ask about knowledge such as system design, development, maintenance, and requirement definition, etc.

[0014] The information processing system 1 provides the test to the examinee or the evaluator. For example, the information processing system 1 provides an environment in which the examinee takes the test online and records the content during the test as a log. The information processing system 1 reproduces the answers input, selected, or set, etc. by the examinee for the evaluator or the company based on the recorded log. As a result, the evaluator or the company can view or evaluate the answer content of the examinee. In addition, the information processing system 1 reproduces at least one of the usage histories of the search tool or the generative AI of the examinee for the evaluator or the company based on the recorded log. As a result, the evaluator or the company can view or evaluate at least one of the usage of the search tool or the generative AI of the examinee. Note that the search tool or the generative AI may be an external tool having a function of supporting the test for measuring the skills of engineers.

[0015] The information processing device 10 is an information processing device used when an examinee takes a skills measurement test. The information processing device 10 is, for example, a personal computer, a smartphone, a tablet terminal, or the like. The information processing device 10 acquires test questions via the network N and presents them to the examinee. The information processing device 10 transmits the content answered by the examinee to the test questions to the server 30. The information processing device 10 transmits the content input by the examinee to at least one of the search tool or the generative AI to the server 30. The information processing device 10 transmits the content output by at least one of the search tool or the generative AI to the server 30.

[0016] The test taker may use at least one of the search tool or generative AI displayed on the answer screen. For example, the test taker may answer the question based on the output result of at least one of the search tool or generative AI displayed on the answer screen. The search tool is, by way of example and not limitation, Google (registered trademark) of the search engine. The generative AI is, by way of example and not limitation, ChatGPT provided by OpenAI OpCo, LLC. The fact that the test taker uses at least one of the search tool or generative AI during the test is realized, for example, by the web browser or application for providing the test and the software for providing at least one of the search tool or generative AI being linked via an API (Application Programming Interface). By performing API linkage, information communication becomes possible between the mutual software, and for example, in response to a request from the software for providing the test, the search tool or generative AI can return a response.

[0017] FIG. 2 is a diagram showing an example of the answer screen of the test taker. The test taker answers the skill measurement test using the answer screen A10 displayed on the information processing device 10 such as a personal computer, a smartphone, a tablet terminal, etc.

[0018] In the screen area B10 on the answer screen A10, the test question is displayed. In the example shown in FIG. 2, in the screen area B10, the question "Please create a code to sort 7, 11, 4, 2, 9, 5, 14 in ascending order using the bubble sort." is displayed. When multiple test questions are presented, the question displayed in the screen area B10 may switch to the next question, for example, when the test taker finishes answering the question. Also, the question displayed in the screen area B10 may switch to the next question, for example, when a predetermined time has elapsed since the question was displayed.

[0019] In the screen area B20 on the answer screen A10, the answers input by the examinee are displayed. The examinee may input answers, for example, by using an input device such as a keyboard provided in the information processing apparatus 10. The examinee may also input answers, for example, by tapping keys displayed on the screen provided in the information processing apparatus 10 with a finger.

[0020] In the screen area B30 on the answer screen A10, buttons (hereinafter also referred to as search buttons) for enabling the examinee to use a search tool are displayed. By selecting the search button during the test, the examinee can use the search tool. The examinee may select the search button, for example, by moving the cursor to the screen area B30 and clicking the mouse. Also, the examinee may select the search button, for example, by tapping the screen area B30 with a finger. Note that if the operator, evaluator, or company does not permit the examinee to use the search tool during the test, the search button is not active and may not be displayed on the answer screen A10. Also, the operator, evaluator, or company can preset whether the examinee can use the search tool for one or more questions before the start of the test.

[0021] In the screen area B40 on the answer screen A10, buttons (hereinafter also referred to as AI buttons) for enabling the examinee to use a generative AI are displayed. By selecting the AI button during the test, the examinee can use the generative AI. The examinee may select the AI button, for example, by moving the cursor to the screen area B40 and clicking the mouse. Also, the examinee may select the AI button, for example, by tapping the screen area B40 with a finger. Note that if the operator, evaluator, or company does not permit the examinee to use the generative AI during the test, the AI button is not active and may not be displayed on the answer screen A10. Also, the operator, evaluator, or company can preset whether the examinee can use the generative AI for one or more questions before the start of the test.

[0022] When the test taker selects the AI button, the request input field B50 (hereinafter also referred to as the input field B50) on the answer screen A10 becomes available. That is, the test taker can input a request (e.g., a prompt) for the generative AI into the input field B50. The rightmost area including the input field B50 may be displayed when either the search button B30 or the AI button B40 is selected.

[0023] When a request is input into the input field B50, the response (e.g., an answer) output by the generative AI is displayed in the response output field B60 (hereinafter also referred to as the output field B60). Thereby, the test taker can answer the question based on the response of the generative AI.

[0024] When the test taker selects the search button, the request input field (not shown. hereinafter also referred to as the input field) on the answer screen A10 becomes available. That is, the test taker can input a request (e.g., a search query) for the search tool into the input field.

[0025] When a request is input into the input field, the response (e.g., search results) output by the search tool is displayed in the response output field (not shown. hereinafter also referred to as the output field). Thereby, the test taker can answer the question based on the response of the search tool.

[0026] An application program (app) for using the engineer skill measurement test service (hereinafter also referred to as the present service) provided by the server 30 may be installed in the information processing apparatus 10. This app causes the information processing apparatus 10 to execute at least a part of the processing related to the engineer skill measurement test disclosed in the following embodiments in the present service provided by the server 30. When this app is executed, the information processing apparatus 10 may access the server 30 and transmit and receive information used for the execution of the app.

[0027] The information processing device 20 is an information processing device used by an evaluator or a company. The information processing device 20 is, for example, a personal computer, a smartphone, a tablet terminal, or the like. The information processing device 20 acquires a request regarding a test question from an evaluator or a company by user input and transmits the request to the server 30. The information processing device 20 acquires a test question corresponding to the request or a list of selectable test questions from the server 30 via the network N. The information processing device 20 may transmit the finally determined test question to the information processing device 10 via the network N. The information processing device 20 may acquire, via the network N, the content answered by the examinee to the test question from the information processing device 10 or the server 30. The information processing device 20 may set, for each test question, whether the examinee can use an external tool including a search tool or a generation AI based on a user operation. The information processing device 20 transmits the usability information of the external tool to the server 30, and the server 30 sets the usability of the external tool for the test question based on the usability information. For example, when the external tool can be used, the server 30 sets to display a button for API cooperation with the above-described external tool on the screen of the test question.

[0028] The server 30 is an information processing device used by an operator or the like, and may be a general-purpose computer or the like. The server 30 provides the service disclosed in this embodiment. For example, the server 30 generates test questions related to the skill measurement of engineers. The server 30 transmits the generated test questions to at least one of the information processing device 10 or the information processing device 20 via the network N. The server 30 acquires, for example, via the network N, the content of the answers given by the examinee to the test questions, the requests input by the examinee to at least one of the search tool or the generation AI, and the responses output by at least one of the search tool or the generation AI. The server 30 may transmit, for example, via the network N, the content of the answers given by the examinee to the test questions or the information obtained by performing predetermined processing on the content to the information processing device 20. The server 30 may transmit, for example, via the network N, the requests input by the examinee to at least one of the search tool or the generation AI or the information obtained by performing predetermined processing on the requests to the information processing device 20. The server 30 may transmit, for example, via the network N, the responses output by at least one of the search tool or the generation AI or the information obtained by performing predetermined processing on the responses to the information processing device 20. Note that the server 30 may perform evaluations such as grading and correction on the answer content, and transmit the evaluated answer content to at least one of the information processing device 10 or the information processing device 20. The detailed processing by the server 30 will be described later.

[0029] The network N is realized by, for example, a network such as the Internet, a mobile phone network, a LAN (Local Area Network), or a combination of these.

[0030] <Configuration of the server> FIG. 3 is a block diagram showing an example of the server 30 according to an embodiment of the disclosure. The server 30 includes one or more processors (e.g., CPUs) 310, one or more network communication interfaces 320, a storage device (storage unit) 330, and one or more communication buses 350 for interconnecting these components.

[0031] The server 30 may optionally include a user interface 340. The user interface 340 includes a display and / or an input device (such as a keyboard and / or a mouse, or some other pointing device, etc.).

[0032] The storage device 330 is, for example, a high-speed random access memory (main storage device) such as DRAM, SRAM, or other random access solid-state storage devices. Further, the storage device 330 may be a non-volatile memory (auxiliary storage device) such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Further, the storage device 330 may be a computer-readable non-transitory recording medium storing programs and the like. Further, the storage device 330 may be either a main storage device (memory) or an auxiliary storage device (storage), or may include both.

[0033] The storage device 330 is a storage unit that stores data, programs, etc. used by the information processing system 1. The storage device 330 stores, in association with each log information (for example, the first log information, the second log information, and the third log information described later), information for identifying the examinee (hereinafter also referred to as the examinee ID). Further, the storage device 330 may store questions presented to the examinee, correct answers to the questions, evaluation criteria, and test results, etc.

[0034] As another example of the storage device 330, it may be one or more storage devices installed remotely from the processor 310. In an embodiment, the storage device 330 stores programs, modules, and data structures executed by the processor 310, or subsets thereof.

[0035] The processor 310 controls, for example, the processing executed by the information processing unit 312 by executing the programs stored in the storage device 330.

[0036] The information processing unit 311 includes, as an example, an acquisition unit 312, a reproduction control unit 313, a specifying unit 314, an extraction unit 315, and a determination unit 316.

[0037] The acquisition unit 312 acquires a reproduction request for an answer input by the user to a predetermined problem regarding the skill measurement of engineers. For example, the acquisition unit 312 acquires a reproduction request for an answer input by the examinee to the information processing device 10 from the information processing device 20 used by the evaluator or the company. The evaluator or the company may, for example, specify the examinee to the information processing device 20, and the information processing device 20 may send a reproduction request including the examinee ID of this examinee to the server 30.

[0038] After acquiring the reproduction request, the acquisition unit 312 acquires first log information regarding the user's answer to the predetermined problem, second log information regarding the request input by the user to the search tool or the generation AI when answering the problem, and third log information regarding the response output by the search tool or the generation AI.

[0039] For example, when the server 30 receives the examinee ID together with the reproduction request from the information processing device 20, the acquisition unit 312 acquires the first log information regarding the answer input by the examinee, which is associated with the examinee ID, from the storage device 330. Note that the first log information is stored in the storage device 330 in association with the examinee ID. The first log information may be stored as a log of key inputs by the examinee.

[0040] Further, the acquisition unit 312 may acquire, from the storage device 330, second log information regarding a request input by the examinee to at least one of the search tool or the generation AI when answering the questions, based on the examinee ID. More specifically, the acquisition unit 312 may acquire, from the storage device 330, second log information regarding a request to the generation AI input by the examinee in the input field B50 shown in FIG. 2. Also, the acquisition unit 312 may acquire, for example, second log information regarding a request input in the input field of the search tool, from the storage device 330. Note that the second log information is stored in the storage device 330 in association with the examinee ID. The second log information may be recorded as a log of key inputs by the examinee after the button for API cooperation with the external tool is operated. Note that the request input to the search tool and the request input to the generation AI may be stored as separate logs.

[0041] In addition, the acquisition unit 312 may acquire, from the storage device 330, third log information regarding a response output by at least one of the search tool or the generation AI, based on the examinee ID. More specifically, the acquisition unit 312 may acquire, from the storage device 330, third log information regarding a response of the generation AI output in the output field B60 shown in FIG. 2. That is, the third log information may store, as a log, data obtained from an external tool such as the search tool or the generation AI. Also, the acquisition unit 312 may acquire, from the storage device 330, third log information regarding a response of the search tool output in the output field of the search tool. Note that the third log information is stored in the storage device 330 in association with the examinee ID. The response (output result) output by the search tool and the response (generation result) output by the generation AI may be stored as separate logs.

[0042] The playback control unit 313 controls the playback of answer input, request input, and response output based on the first log information, the second log information, and the third log information. For example, the playback control unit 313 generates screen information regarding answer input, request input, and response output based on the first log information, the second log information, and the third log information. The generated screen information is transmitted, for example, by the server 30 to the information processing device 20 used by the evaluator or the company.

[0043] The information processing device 20 that has received the screen information displays a playback screen regarding answer input, request input, and response output based on the screen information. At this time, the playback control unit 313 performs predetermined control regarding the playback of answer input, request input, and response output, for example. The playback control unit 313 controls, for example, to playback the answer content or the usage content of the external tool in the order in which the examinee performs operations such as input, selection, and setting. The playback screen may have the same configuration as the test screen, for example, or may be played back with the answer area made larger than the answer area of the test screen so that the answer content is more understandable.

[0044] Through the above processing, the evaluator or the company can grasp the usage skills of the examinee's search tool or generative AI in a test where the use of the search tool or generative AI is permitted. For example, the information processing system 1 can present at what timing and what requests were input to the external tool, and can also present the answer content corresponding to the response, the content of the re-request, etc.

[0045] The specifying unit 314 specifies an answer based on the response output by the search tool or the generative AI. At this time, the playback control unit 313 may highlight the answer based on the specified response on the screen where the answer input, request input, and response output of the information processing device 20 are played back.

[0046] FIG. 4 is a diagram showing an example of a screen (hereinafter also referred to as a playback screen) on which an answer input, a request input, and a response output are played back. The playback screen A20 shown in FIG. 4 is displayed, for example, on an information processing apparatus 20 used by an evaluator or a company. In the example shown in FIG. 4, an examinee's answer screen (corresponding to the answer screen A10 in FIG. 2) is displayed in a screen area C10 on the playback screen A20. In the example shown in FIG. 4, since the examinee input "Teach me the code for the repetition process for the specified number of times." in the input field B50 during the test, "for variable name in range(repetition times):" is output in the output field B60. At this time, assume that the examinee entered "for i in range(len(bubble_sort_num)-1):" in the field for entering an answer (corresponding to the screen area B20 in FIG. 2) based on "for variable name in range(repetition times):" displayed in the output field B60. In this case, for example, the specifying unit 314 specifies that "for i in range(len(bubble_sort_num)-1):" is an answer based on a response output by the generation AI.

[0047] The specific part 314 may identify an answer based on, for example, the response output by a search tool or a generative AI based on the first log information and the third log information. More specifically, the specific part 314 determines, based on the first log information and the third log information, whether there is similarity between the response and the answer input by the answerer after the output of the response. If it is determined that there is similarity, the specific part 314 may identify that the answer is an answer based on the response. That is, in the example shown in FIG. 4, the specific part 314 determines that there is similarity between "for variable name in range (number of repetitions):" output to the output field B60 and "for i in range(len(bubble_sort_num)-1):" input to the field for inputting an answer (corresponding to the screen area B20 in FIG. 2). Regarding the determination of similarity, for example, the specific part 314 may calculate a similarity degree such as the mean squared error between each character string, and if the similarity degree is equal to or higher than a threshold value, determine that they are similar. Also, as long as the similarity degree between two character strings can be calculated, the method for calculating the similarity degree is not particularly limited. At this time, the specific part 314 identifies that "for i in range(len(bubble_sort_num)-1):" is an answer based on the response. Note that the determination process of similarity may be performed by the determination part 316.

[0048] The playback control unit 313 highlights, for example, an answer based on the response specified by the specifying unit 314 on the playback screen of the information processing apparatus 20. For example, on the playback screen A20 shown in FIG. 4, the playback control unit 313 highlights "for i in range(len(bubble_sort_num)-1):", which is the answer based on the response specified by the specifying unit 314. In the example shown in FIG. 4, "for i in range(len(bubble_sort_num)-1):", which is the answer based on the response, is displayed in bold, but the display form is not limited to this. For example, the answer based on the response may be displayed in a different color from other answers (for example, "bubble_sort_num=[7,11,4,2,9,5,14]" shown in FIG. 4). Further, the playback control unit 313 may display, in the vicinity of the answer based on the response on the playback screen A20, a character, symbol, illustration, etc. indicating that it is the answer based on the response.

[0049] Through the above processing, it becomes possible to highlight the answer based on the response input by the examinee on the playback screen. As a result, the evaluator or company can easily grasp which answer among the answers input by the examinee is the answer based on the response.

[0050] Further, the specifying unit 314 may specify that the input answer is an answer based on the response being copied and pasted. For example, the specifying unit 314 may use a known technique to detect a copy event key (for example, Ctrl+C) and a paste event key (for example, Ctrl+V) based on the first log information, and thereby specify that the input answer is an answer based on the response.

[0051] More specifically, the specific part 314 may detect that the examinee copied the "for variable name in range(repetition count):" displayed in the output field B60 during the test and pasted it into the field for entering the answer (corresponding to the screen area B20 in Fig. 2), and identify that "for i in range(len(bubble_sort_num)-1):" is the answer based on the response.

[0052] Through the above processing, based on the determination result of the above similarity, the answer based on the response can be identified with a smaller amount of computation than the processing of identifying the answer based on the response. Thereby, it becomes possible to reduce the processing load on the server 30.

[0053] When the user answers based on the responses output by the search tool and the generation AI, the reproduction control unit 313 may highlight the answer based on the response of the search tool and the answer based on the response of the generation AI in different display forms.

[0054] For example, when the examinee uses both the search tool and the generation AI during the test and answers based on the responses output by the search tool and the generation AI, the reproduction control unit 313 may highlight the answer based on the response of the search tool and the answer based on the response of the generation AI in different display forms.

[0055] FIG. 5 is a diagram showing an example in which, on a playback screen, answers based on the response of a search tool and answers based on the response of a generative AI are highlighted in different display forms. In the example shown in FIG. 5, the examinee answers "for i in range(len(bubble_sort_num)-1):" and "for j in range(len(bubble_sort_num)-1,i,-1):" shown in screen area D10 based on the response of the generative AI, which is "for variable name in range(number of repetitions):". Also, the examinee answers "if(bubble_sort_num[j]<bubble_sort_num[j-1]):" shown in screen area D20 based on the response of the search tool (not shown).

[0056] The specifying unit 314 specifies, for example, that the answer shown in screen area D10 is an answer based on the response of the generative AI. Also, the specifying unit 314 specifies, for example, that the answer shown in screen area D20 is an answer based on the response of the search tool. Whether it is an answer based on the response of the search tool or the generative AI tool may be determined by the similarity determination described above.

[0057] The playback control unit 313 displays, for example, the answer based on the response of the generative AI shown in screen area D10 specified by the specifying unit 314 in blue letters. The playback control unit 313 displays, for example, the answer based on the response of the search tool shown in screen area D20 specified by the specifying unit 314 in red letters. Note that the color of the answer based on the response is arbitrary. Also, the display forms of the answer based on the response of the generative AI and the answer based on the response of the search tool are not limited to this. The playback control unit 313 may, for example, display the answer based on the response of the generative AI in bold and the answer based on the search tool in italics on the playback screen A20.

[0058] The playback control unit 313 may display, for example, a character, symbol, or illustration indicating that it is an answer based on the response of the generation AI near the answer based on the response of the generation AI on the playback screen A20. At this time, the playback control unit 313 may display, for example, a character, symbol, or illustration indicating that it is an answer based on the response of the search tool near the answer based on the search tool. Note that the character, symbol, or illustration indicating that it is an answer based on the response of the generation AI is different from the character, symbol, or illustration indicating that it is an answer based on the response of the search tool.

[0059] Through the above processing, it becomes possible to highlight the answer based on the response of the search tool and the answer based on the response of the generation AI in different display forms on the playback screen. As a result, the evaluator or enterprise can easily distinguish the answer based on the response of the search tool and the answer based on the response of the generation AI on the playback screen.

[0060] When the examinee inputs an answer based on the response on the answer screen A10 and further inputs an answer based on the answer based on the response, the playback control unit 313 may display the answer in a display form different from other answers.

[0061] FIG. 6 is a diagram showing an example in which, when a test taker further inputs an answer based on a response-based answer, the answer is displayed in a display form different from other answers on a playback screen. In the example shown in FIG. 6, it is assumed that the test taker copies "for variable name in range (number of repetitions):", which is the response of the generation AI, pastes it on the answer screen, and rewrites it to "for i in range(len(bubble_sort_num)-1):" shown in screen area E10. Further, it is assumed that the test taker copies "for i in range(len(bubble_sort_num)-1):" shown in screen area E10, pastes it on the answer screen, and rewrites it to "for j in range(len(bubble_sort_num)-1,i,-1):" shown in screen area E20. In this case, the playback control unit 313 displays the answer in screen area E20 in a display form different from other answers on the playback screen A20.

[0062] The playback control unit 313, for example, displays the answer in screen area E20 in blue on the playback screen A20. The playback control unit 313, for example, displays the answer in screen area E10 in red. The playback control unit 313, for example, displays other answers (for example, "bubble_sort_num=[7,11,4,2,9,5,14]", "if(bubble_sort_num[j]<bubble_sort_num[j-1]):", and "tmp=bubble_sort_num[j]" etc.) in black. Note that the colors of the answer in screen area E10, the answer in screen area E20, and other answers are arbitrary. Also, on the playback screen A20, the display forms of the answer in screen area E10, the answer in screen area E20, and other answers are not limited to this. The playback control unit 313, for example, may display the answer in screen area E20 in bold on the playback screen A20, display the answer in screen area E10 in thin font, and display other answers in a font thinner than the answer in screen area E10.

[0063] The reproduction control unit 313 may display characters, symbols, illustrations, etc. in the vicinity of the answer in the screen area E20, for example, on the reproduction screen A20. In this case, the reproduction control unit 313 may display characters, symbols, illustrations, etc. in the vicinity of the answer in the screen area E10, for example. Note that the characters, symbols, illustrations, etc. displayed in the vicinity of the answer in the screen area E20 are different from the characters, symbols, illustrations, etc. displayed in the vicinity of the answer in the screen area E10.

[0064] Also, before the processing of the reproduction control unit 313, the specifying unit 314 may specify the first part of the answer based on the response of the search tool or the generation AI from among the answers input by the examinee. The specifying unit 314 may specify, for example, that the answer in the screen area E10 is the answer based on the response of the search tool or the generation AI. As a specific example, based on the first log information, based on the key operations of the copy event key (for example, Ctrl+C) and the paste event key (for example, Ctrl+V) by the examinee, "for i in range(len(bubble_sort_num)-1):" in the detected screen area E10 can be specified as the first part based on the response.

[0065] The specifying unit 314 may specify the first part of the answer based on the response of the search tool or the generation AI based on, for example, the detection of the key operations of the copy event key (for example, Ctrl+C) and the paste event key (for example, Ctrl+V) by the examinee based on the first log information, or the determination of the similarity between the response and the character string within a predetermined range. At this time, the specified first part is at least temporarily stored in a memory or the like and used for comparison.

[0066] Next, the extraction unit 315 may extract a predetermined answer (hereinafter also referred to as the second part) from the answers after the input of the first part. The extraction unit 315 may, for example, extract a second area that is presumed to be based on the answer in the screen area E10 specified by the specifying unit 314. For example, when the examinee uses the copy event key (e.g., Ctrl+C) and the paste event key (e.g., Ctrl+V) to copy "for i in range(len(bubble_sort_num)-1):" in the screen area E10 to the screen area E20 and rewrite it as "for j in range(len(bubble_sort_num)-1,i,-1):" and uses the line break event key (e.g., Enter), the extraction unit 315 may extract the answer in the screen area E20 by specifying "for i in range(len(bubble_sort_num)-1):" which is the copy source based on the first log information. Also, for example, when the examinee inputs "for j in range(len(bubble_sort_num)-1,i,-1):" in the screen area E20 by key input, the extraction unit 315 may detect the key inputs of "for", "in", "range", and "():", and specify, as the second part, the answer in the screen area E20 including "for", "in", "range", and "():" based on "for i in range(len(bubble_sort_num)-1):" in the screen area E10. Further, the extraction unit 315 may use a time window corresponding to the range (e.g., the length of the character string) of the first part to extract, as the second part, the character string within the time window that slides together with the key input. Note that as long as the second part can be extracted, the extraction method is not particularly limited.

[0067] At this time, the determination unit 316 may determine the similarity between the first part and the second part. For example, the determination unit 316 may determine the similarity between the answer in the screen area E20 and the answer in the screen area E10. Regarding the determination of similarity, for example, the determination unit 316 may calculate a similarity degree such as the mean squared error between each character string, and if the similarity degree is equal to or greater than a threshold value, it may be determined that they are similar. Also, as long as the similarity degree between two character strings can be calculated, the method for calculating the similarity degree is not particularly limited.

[0068] When it is determined by the determination unit 316 that there is similarity between the first part and the second part, the playback control unit 313 may display the answer in the screen area E20 in a display form different from other answers on the playback screen A20.

[0069] By the above processing, when the examinee inputs an answer based on the response on the answer screen and further inputs an answer (hereinafter, also referred to as the second answer) based on the answer based on the response, it becomes possible to display the second answer in a display form different from other answers. Thereby, for example, when the examinee copies and pastes the response of the search tool or the generation AI, inputs a predetermined answer, and further copies and pastes the predetermined answer and inputs the second answer, the second answer is displayed on the playback screen in a display form different from other answers, so that the evaluator or the company can easily grasp that the second answer is answered with reference to the response of the search tool or the generation AI.

[0070] The specifying unit 314 may specify the request input to the search tool or the generation AI. For example, the specifying unit 314 may specify the request input by the examinee to the search tool or the generation AI during the test based on the second log information. Also, the second log information may be stored in the storage device 330 in association with time information indicating at what hour, minute, and second after the start of the test the request was input.

[0071] At this time, the playback control unit 313 may make it possible to identify the playback timing of the request specified by the specifying unit 314 based on the second log information on the playback screen.

[0072] FIG. 7 is a diagram showing an example in which the playback timing of a request input to a search tool or a generation AI is visibly displayed on a playback screen. In the example shown in FIG. 7, the playback control unit 313 visibly displays the timing at which a request was input at M10 and M20 on the timeline T10 at the bottom of the playback screen. The playback control unit 313 visibly displays the timing at which a request was input, for example, based on the time information associated with the second log information. That is, in the example shown in FIG. 7, it is shown that the examinee input requests to the search tool or the generation AI twice during the test.

[0073] Further, the playback control unit 313 may make the playback timing of a request input to the search tool and the playback timing of a request input to the generation AI distinguishable in different display forms. In the example shown in FIG. 7, if M10 is set as the playback timing of a request input to the search tool and M20 is set as the playback timing of a request input to the generation AI, the playback control unit 313 may display, for example, the playback timing indicated by M10 in blue and the playback timing indicated by M20 in red. The playback control unit 313 visibly displays the timing at which a request was input, for example, based on the time information associated with the second log information. In this case, the request input to the search tool and the request input to the generation AI are stored in the storage device 330 as separate logs. Note that the color of the displayed playback timing is arbitrary.

[0074] Through the above processing, the evaluator or the company can easily grasp the timing at which a request was input to the search tool or the generation AI on the timeline.

[0075] Also, when the examinee inputs a response-based answer on the answer screen, the playback control unit 313 may make it possible to identify the playback timing of the input of the answer on the playback screen. In this case, the first log information may be stored in the storage device 330 in association with time information indicating how many hours, minutes, and seconds after the start of the test the answer was input. The playback control unit 313 may, for example, make it possible to identify the playback timing of the input of the answer specified by the specifying unit 314 based on the time information associated with the first log information. Also, when the examinee inputs a further answer (for example, the second part) based on the response-based answer (for example, the first part), the playback control unit 313 may make it possible to identify the playback timing of the input of the answer on the playback screen. The playback control unit 313 may, for example, make it possible to identify the playback timing of the input of the answer corresponding to the second part extracted by the extraction unit 315 based on the time information associated with the first log information.

[0076] More specifically, in the example shown in FIG. 7, if M10 is the playback timing of the input of the response-based answer and M20 is the playback timing of the input of the answer based on the answer, the playback control unit 313 may, for example, display the playback timing indicated by M10 in blue characters and the playback timing indicated by M20 in red characters. Note that the color of the displayed playback timing is arbitrary.

[0077] Through the above processing, the evaluator or enterprise can easily grasp the playback timing of the input of the response-based answer and the playback timing of the input of the answer based on the answer on the timeline.

[0078] <Operation> Next, the operation according to the embodiment will be described. FIG. 8 is a sequence diagram showing an example of the processing of the information processing apparatus 20 and the server 30 according to the present embodiment.

[0079] In step S11, the information processing apparatus 20 transmits a playback request for an answer input by the user to the processing apparatus (for example, the information processing apparatus 10) to the server 30 for a predetermined problem related to the skill measurement of the engineer.

[0080] In step S12, the acquisition unit 312 acquires, from the storage device 330, first log information regarding a user's (e.g., examinee's) answer to a predetermined question.

[0081] In step S13, the acquisition unit 312 acquires, from the storage device 330, second log information regarding a request input by the user (e.g., examinee) to the search tool or the generation AI when answering the question.

[0082] In step S14, the acquisition unit 312 acquires, from the storage device 330, third log information regarding a response output by the search tool or the generation AI.

[0083] In step S15, the playback control unit 313 generates screen information regarding the input of the answer, the input of the request, and the output of the response based on the first log information, the second log information, and the third log information.

[0084] In step S16, the server 30 transmits the screen information to the information processing device 20.

[0085] In step S17, the information processing device 20 displays a playback screen regarding the input of the answer, the input of the request, and the output of the response based on the received screen information.

[0086] In step S18, the playback control unit 313 performs predetermined control regarding the playback of the input of the answer, the input of the request, and the output of the response.

[0087] <Modification Example> In the above embodiment, a test regarding the measurement of engineers' skills is taken as an example for explanation, but it is not limited thereto. For example, the test may be an online test such as a test regarding a predetermined professional field, a test regarding the skills of a predetermined clerical position, a test regarding the skills in a predetermined public service, a test regarding a predetermined qualification, and a test regarding a predetermined examination.

Explanation of Reference Numerals

[0088] 1... Information processing system, 10... Information processing device, 20... Information processing device, 30... Server, 310... Processor, 311... Information processing unit, 312... Acquisition unit, 313... Reproduction control unit, 314... Identification unit, 315... Extraction unit, 316... Judgment unit, 320... Network communication interface, 330... Storage device, 340... User interface, 350... Communication bus, A10... Answer screen, A20... Reproduction screen, B10~40... Screen area, B50... Input field, B60... Output field, C10... Screen area, D10... Screen area, D20... Screen area, E10... Screen area, E20... Screen area, M10... Reproduction timing, M20... Reproduction timing, N... Network, T10... Timeline

Claims

1. An information processing apparatus acquires a reproduction request for an answer input by a user to a processing apparatus for a predetermined problem related to engineer skill measurement; after acquiring the reproduction request, acquires first log information regarding the user's answer to the predetermined problem, second log information regarding a request input by the user to a search tool or a generative AI when answering the problem, and third log information regarding a response output by the search tool or the generative AI; controls reproduction of input of the answer, input of the request, and output of the response based on the first log information, the second log information, and the third log information; An information processing method for performing the above.

2. including specifying an answer based on the response, wherein controlling the reproduction includes highlighting, on a reproduced screen, the answer based on the specified response. The information processing method according to claim 1.

3. The specifying includes specifying that the response is an answer based on the response being copy-and-pasted. The information processing method according to claim 2.

4. when the user answers based on responses output by the search tool and the generative AI, the highlighting includes highlighting, in different display forms from each other, the answer based on the response of the search tool and the answer based on the response of the generative AI. The information processing method according to claim 2 or 3.

5. specifying a first part of an answer based on the response, extracting a second part of the answer after input of the first part, including determining similarity between the first part and the second part, wherein controlling the reproduction includes, when it is determined that there is similarity, displaying, on a reproduced screen, the second part in a display form different from other answers. The information processing method according to claim 1.

6. including specifying the input request, wherein controlling the reproduction includes making it possible to identify, on a reproduced screen, the reproduction timing of the specified request based on the second log information. The information processing method according to claim 1.

7. In an information processing apparatus, acquires a reproduction request for an answer input by a user to a processing apparatus for a predetermined problem related to engineer skill measurement; After acquiring the reproduction request, acquiring first log information regarding the user's answer to the predetermined problem, second log information regarding the request input by the user to a search tool or a generative AI when the user answers the problem, and third log information regarding the response output by the search tool or the generative AI Based on the first log information, the second log information, and the third log information, controlling the reproduction of the input of the answer, the input of the request, and the output of the response A program for causing the execution

8. An information processing apparatus including a processor, wherein the processor Acquires a reproduction request for an answer input by a user to a predetermined problem related to the skill measurement of an engineer from a processing device After acquiring the reproduction request, acquiring first log information regarding the user's answer to the predetermined problem, second log information regarding the request input by the user to a search tool or a generative AI when the user answers the problem, and third log information regarding the response output by the search tool or the generative AI Based on the first log information, the second log information, and the third log information, controlling the reproduction of the input of the answer, the input of the request, and the output of the response An information processing apparatus that executes the above

Images