Recommendation device, recommendation method, and computer program

Abstract

The aim is to provide technology that enables more efficient training of security personnel. [Solution] One aspect of the present invention is a recommendation device that includes a memory unit that stores multiple second model answers obtained by classifying a first model answer, which is a model answer for a task to be judged, into predetermined classification items, and a recommended task selection unit that selects other tasks to recommend to a user related to the task to be judged, depending on the similarity between the second model answer and a third model answer, which is a model answer for another task.

Description

[Technical Field]

[0001] The present invention relates to an evaluation device, a recommendation device, an evaluation method, a recommendation method, and a computer program. [Background technology]

[0002] Security personnel are necessary to protect the safety and reliability of companies, including responding to cyberattacks and legal regulations, and protecting personal information. As cyberattacks become more sophisticated and complex, the demand for security personnel is increasing, but the supply is not keeping up, resulting in a chronic shortage of personnel. Meanwhile, technology has been proposed to provide an online learning system in which multiple instructors work together remotely to effectively and efficiently support the learning of a large number of students (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-148710 Summary of the Invention [Problem to be solved by the invention]

[0004] Dealing with increasingly sophisticated and complex cyberattacks due to advances in information processing technology requires a wide range of knowledge and skills, including not only basic knowledge of information processing technology but also infrastructure knowledge such as operating systems (OS), networks, and servers, as well as programming skills in languages ​​such as C, PHP, and Python. Given the wide range of learning topics, learners' knowledge and skill levels also vary. For efficient learning, however, it is important to select tasks with an appropriate level of difficulty to match each learner's knowledge and skill level. However, conventional technologies have not always been able to meet the needs of such security professionals. Furthermore, when it comes to learning in the field of information security, the diverse knowledge and skill levels of learners and the wide range of learning topics mean that it is sometimes difficult to select tasks appropriate for each learner's individual knowledge and skill level.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique that enables security personnel to be trained more efficiently. [Means for solving the problem]

[0006] One aspect of the present invention is an evaluation device that includes: a memory unit that stores multiple pieces of second expert operation history information obtained by classifying first expert operation history information obtained when an expert performs a specified task using a computer into predetermined classification items; an acquisition unit that acquires first user operation history information obtained when a user to be evaluated performs the specified task using a computer and acquires one or more pieces of second user operation history information that are similar to the first user operation history information for each piece of second expert operation history information; and an evaluation unit that evaluates the user's work for the classification items based on the second expert operation history information and the second user operation history information that fall into the same classification item.

[0007] One aspect of the present invention is the above-mentioned evaluation device, wherein the acquisition unit generates partial history information that is a part of the user operation history information, acquires a similarity between the partial history information and the second expert operation history information, and acquires the partial history information whose similarity exceeds a predetermined threshold as the second user operation history information.

[0008] One aspect of the present invention is the above-mentioned evaluation device, wherein the evaluation unit inputs second expert operation history information and second user operation history information corresponding to the same classification item, and instruction information regarding the evaluation of the work, into a large-scale language model, thereby causing the large-scale language model to evaluate the work.

[0009] One aspect of the present invention is a recommendation device that includes a memory unit that stores a plurality of second model answers obtained by classifying a first model answer, which is a model answer for a task to be judged, into predetermined classification items, and a recommended task selection unit that selects another task to be recommended to a user related to the task to be judged, depending on the similarity between the second model answer and a third model answer, which is a model answer for another task.

[0010] One aspect of the present invention is the above-mentioned recommendation device, wherein the recommended task selection unit selects another task corresponding to a third model answer whose similarity to the second model answer exceeds a predetermined threshold.

[0011] One aspect of the present invention is an evaluation method including the steps of: an information processing device storing a plurality of pieces of second expert operation history information obtained by classifying, into predetermined classification items, first expert operation history information obtained when an expert performed a predetermined task using a computer; acquiring first user operation history information obtained when a user to be evaluated performed the predetermined task using a computer, and acquiring, for each piece of second expert operation history information, one or more pieces of second user operation history information that are similar to the first user operation history information; and evaluating the user's work for the classification item based on the second expert operation history information and the second user operation history information that fall into the same classification item.

[0012] One aspect of the present invention is a recommendation method having an information processing device storing a plurality of second model answers obtained by classifying a first model answer, which is a model answer for a task to be judged, into predetermined classification items, and selecting another task to recommend to a user related to the task to be judged, depending on the similarity between the second model answer and a third model answer, which is a model answer for another task.

[0013] One aspect of the present invention is a computer program for causing an information processing device to execute the steps of: storing a plurality of second expert operation history information obtained by classifying, into predetermined classification items, first expert operation history information obtained when an expert performed a specified task using a computer; acquiring first user operation history information obtained when a user to be evaluated performed the specified task using a computer, and acquiring, for each of the second expert operation history information, one or more second user operation history information that is similar to the first user operation history information; and evaluating the user's work for the classification item based on the second expert operation history information and the second user operation history information that fall into the same classification item.

[0014] One aspect of the present invention is a computer program for causing an information processing device to perform the steps of: storing a plurality of second model answers obtained by classifying a first model answer, which is a model answer for a task to be judged, into predetermined classification items; and selecting another task to be recommended to a user related to the task to be judged, based on the similarity between the second model answer and a third model answer, which is a model answer for another task. [Effects of the Invention]

[0015] The present invention makes it possible to more efficiently train security personnel. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a system configuration of an online learning system 1 according to an embodiment. [Figure 2] 2 is a diagram illustrating an example of a functional configuration of a content server 100 according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of learning content that uses a practice environment. [Figure 4] FIG. 10 is a diagram illustrating expert information 122. [Figure 5] 10 is a diagram illustrating an example of a method by which a related log extraction unit 133 extracts related logs from a user's operation log. FIG. [Figure 6] 10 is a diagram illustrating an example of a method by which a related log extraction unit 133 extracts related logs from a user's operation log. FIG. [Figure 7] FIG. 10 is a diagram illustrating a skill level analysis performed by the skill evaluation unit 134. [Figure 8] This is a diagram (part 1) showing an example of instructions (instruction statements) that the skill evaluation unit 134 uses to instruct the inference model 123 to perform skill level analysis. [Figure 9] This is a diagram (part 2) showing an example of instructions (instruction statements) that the skill evaluation unit 134 uses to instruct the inference model 123 to perform skill level analysis. [Figure 10] This is a diagram (part 3) showing an example of instructions (instruction statements) that the skill evaluation unit 134 uses to instruct the inference model 123 to perform skill level analysis. [Figure 11] This is a diagram (part 4) showing an example of instructions (instruction statements) that the skill evaluation unit 134 uses to instruct the inference model 123 to perform skill level analysis. [Figure 12] This is a diagram (part 5) showing an example of instructions (instruction statements) that the skill evaluation unit 134 uses to instruct the inference model 123 to perform skill level analysis. [Figure 13] FIG. 10 is a diagram for explaining the process of recommending study content to a user by study content recommendation unit 135. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] FIG. 1 is a diagram showing an example of the system configuration of an online learning system 1 according to an embodiment. The online learning system 1 is a system that provides users with learning content related to information security. The online learning system 1 includes a user terminal 10, a practice environment server 20, and a content server 100. The user terminal 10, the practice environment server 20, and the content server 100 can communicate with each other via a network NW. The network NW may include a LAN, a cellular network, the Internet, etc.

[0019] The user terminal 10 is a terminal used by a user of the online learning system 1. The user terminal 10 is, for example, an information processing terminal such as a smartphone, tablet, or PC. An application program (hereinafter referred to as a "learning app") AP for using the learning content of the online learning system 1 is pre-installed on the user terminal 10. The learning app AP may be an application dedicated to the online learning system 1, or may be a general-purpose application such as a web browser.

[0020] The learning application AP connects to the content server 100 in response to a user's operation, and retrieves and plays learning content specified by the user from the content server 100. The learning content may be video, images, audio, or a combination of these. The learning content may also be lecture-style content in which users watch videos or the like, or test-style content in which users are asked questions and then asked to answer them. The questions may be multiple-choice questions, essay-style questions, or practical questions in which users are required to actually operate a device.

[0021] The practice environment server 20 is a server that provides a practice environment to provide practical learning content. For example, the practice environment server 20 may have the functionality of a terminal server and provide a terminal environment using a GUI (Graphical User Interface) or CUI (Character User Interface) for devices to be studied (e.g., network devices, server devices, terminal devices, etc.). The practice environment server 20 accepts input of operations for the practice environment from the user terminal 10 and shares the details of the input operations (operation log) with the content server 100.

[0022] Content server 100 provides study content to user terminal 10. Content server 100 also has the function of selecting and presenting recommended content to a user based on the user's viewing status of the study content. The content viewing status includes the user's learning progress, learning history, test answers, scoring results, and operation details for the practice environment.

[0023] 2 is a diagram showing an example of the functional configuration of the content server 100 according to the embodiment. The content server 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The communication unit 110 communicates with other devices. For example, the communication unit 110 communicates with the user terminal 10 and the practice environment server 20. The communication unit 110 receives a request for providing learning content from the user terminal 10 and transmits the requested learning content to the user terminal 10. The communication unit 110 also obtains the details of operations (operation logs) performed by the user on the practice environment from the practice environment server 20.

[0024] The storage unit 120 stores content information 121, expert information 122, an inference model 123, and user information 124. The storage unit 120 is configured using a magnetic storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), a semiconductor storage device such as a flash memory, or the like.

[0025] Content information 121 is information necessary for providing learning content. Content information 121 includes, for example, data such as videos, images, audio, and text containing the contents of lectures and tests. Content information 121 may also include information necessary for allowing users to use the practice environment, such as how to access practice environment server 20.

[0026] Expert information 122 is information that stores the operation details (operation log) of experts regarding practical learning content. Here, experts are people who have sufficient knowledge and skills regarding information security, and the operation log of experts is used in online learning system 1 as data showing learning samples and correct answers to assignments.

[0027] The inference model 123 is a model used to analyze the viewing status of learning content and is configured using a so-called large language model (LLM) such as ChatGPT (registered trademark). The inference model 123 is trained based on publicly available information on the Internet at a certain point in time and is configured to respond to natural language input with content that is consistent with the context and intent. Furthermore, the inference model 123 can provide answers based on knowledge that adds external information to previously learned information using so-called Retrieval-Augmented Generation (RAG). For example, the inference model 123 can reference content information 121, expert information 122, and user information 124 as external information.

[0028] The user information 124 is information about users of the online learning system 1. The user information includes an operation log for the user's practice environment. The user information 124 may also include, for example, the user's service registration information, course history, and test answers.

[0029] Control unit 130 is configured using a processor such as a CPU (Central Processing Unit) and memory, and executes application programs to control the entire content server 100. More specifically, control unit 130 includes, for example, content control unit 131, user log acquisition unit 132, related log extraction unit 133, skill evaluation unit 134, and study content recommendation unit 135.

[0030] Some or all of the content control unit 131, user log acquisition unit 132, related log extraction unit 133, skill evaluation unit 134, and learning content recommendation unit 135 may be realized by hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA, or by a combination of software and hardware. The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and non-transitory storage media such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0031] The content control unit 131 controls the provision of study content to the user terminal 10. In response to a content provision request from the user terminal 10, the content control unit 131 extracts the requested study content from the content information 121 and transmits it to the user terminal 10. The content control unit 131 also accepts information input by the user within the study content. The content control unit 131 saves the user's input information in the user information 124.

[0032] The user log acquisition unit 132 acquires the user's operation log for the practice environment from the practice environment server 20. The user log acquisition unit 132 stores the acquired user's operation log in the user information .

[0033] The related log extraction unit 133 extracts a log portion (related log) related to each skill item from the user's operation log. More specifically, the related log extraction unit 133 divides the user's operation log into partial logs of a predetermined size in chronological order, and identifies which skill item each partial log corresponds to by calculating the similarity of each partial log with the operation log of an expert. The related log may be stored in the user information 124. Note that partial logs may be generated so that they partially overlap with previous and subsequent partial logs. Here, a skill item is an example of a "category item." Also, operating a computer to achieve a purpose related to a category item (skill item) is an example of a "predetermined task."

[0034] The skill evaluation unit 134 analyzes the user's skill level for each skill item based on the contents of the user's related logs extracted for each skill item. More specifically, the skill evaluation unit 134 inputs the related logs to be analyzed into the inference model 123 and obtains the skill level determination result as its output. In addition to the related logs, content information and operation logs of experts may also be input to the inference model 123 as necessary.

[0035] The study content recommendation unit 135 selects study content to recommend to the user based on the skill level analysis results by the skill evaluation unit 134. The study content recommendation unit 135 presents the selected recommended study content to the user. Here, the study content recommendation unit is an example of a "recommended task selection unit."

[0036] FIG. 3 is a diagram showing an example of learning content that uses a practice environment. FIG. 3 shows an example in which a user selects to take the learning content titled "Unauthorized Access Practice." When the user performs an operation to select the learning content, for example, a first screen IM1 is displayed on the user terminal 10. For example, the first screen IM1 displays information necessary to access the practice environment and information necessary for carrying out the learning. For example, the first screen IM1 displays, as an example of information necessary for accessing the practice environment, URL information INF11 of the practice environment and login information INF12 for logging in to the practice environment. Furthermore, for example, the first screen IM1 displays, as an example of information necessary for carrying out the learning, the IP address of the practice server that will be the target of unauthorized access (the target of exploitation).

[0037] In this case, for example, a user accesses the practice environment by launching a web browser on the user terminal 10 and entering URL information INF11 displayed on the first screen IM1. This causes the first screen IM1 to transition to the second screen IM2. The second screen IM2 is a login screen for the practice environment. When the user enters login information INF12 displayed on the first screen IM1 into the login screen, the second screen IM2 transitions to the third screen IM3. The third screen IM3 is a user interface through which the user inputs operations into the practice environment. For example, in the case of learning in the form of practicing command operations, the third screen IM3 may be a virtual terminal screen connected to the practice environment server. In the example of FIG. 3, the third screen IM3 is a virtual terminal screen in which the user has logged in to the practice server from the practice environment server 20 using the IP address displayed on the first screen IM1. The operations entered on the virtual terminal screen are recorded as an operation log by the practice environment server 20 and provided to the content server 100 as appropriate.

[0038] FIG. 4 is a diagram illustrating the expert information 122. As described above, the expert information 122 is information that stores operation logs of experts related to practical learning content. More specifically, the expert information 122 includes operation logs of experts broken down by skill item. Hereinafter, the operation logs of experts broken down by skill item are referred to as "expert partial logs." The example of FIG. 4 shows that the expert operation log LG1 is included in the expert information 122 in a form broken down into expert partial logs LG1-1, LG1-2, and LG1-3 for each skill item. The expert partial logs may be generated manually or mechanically based on a predetermined extraction rule. Herein, the expert operation log is an example of "first expert operation history information," and the expert partial log is an example of "second expert operation history information."

[0039] For example, the expert partial log LG1-1 is obtained by extracting the performed portion of the operation corresponding to the skill item "port scan" from the operation log of an expert. Furthermore, for example, the expert partial log LG1-2 is obtained by extracting the performed portion of the operation corresponding to the skill item "directory scan" from the operation log of an expert. Furthermore, for example, the expert partial log LG1-3 is obtained by extracting the performed portion of the operation corresponding to the skill item "SQL injection" from the operation log of an expert. The skill items illustrated in FIG. 4 are merely examples, and expert partial logs may be generated for any skill item.

[0040] FIG. 5 is a diagram (part 1) illustrating an example of a method by which the related log extraction unit 133 extracts related logs from a user's operation log. First, the related log extraction unit 133 divides the user's operation log into chronological partial logs of a predetermined size. Hereinafter, the chronologically divided user operation log is referred to as a "user partial log." A user partial log may be configured to partially overlap with previous and subsequent user partial logs. FIG. 5 shows an example in which the operation log LG2 is divided into user partial logs of 10 lines each. FIG. 5 illustrates user partial logs LG2-1, LG2-2, LG2-3, ..., configured so that the first five lines overlap with the previous user partial log and the last five lines overlap with the subsequent user partial log. Here, the user's operation log is an example of "first user operation history information," and the user partial logs are an example of "partial history information."

[0041] Next, the related log extraction unit 133 calculates the similarity between each user partial log and the expert partial log for each skill item. FIG. 5 shows an example in which each user partial log LG2-1, LG2-2, LG2-3, ... is compared with the expert partial log for the skill item "port scan." The method for calculating the similarity is not limited to a specific method. For example, each log can be vectorized using a technique such as Word2Vec, and the Levenshtein distance, cosine similarity, Jaccard coefficient, or the like between the vectors can be calculated as the similarity. Graph G1 represents the level of similarity calculated for each user partial log LG2-1, LG2-2, LG2-3, ....

[0042] In graph G1, the horizontal axis corresponds to the time series of the operation logs, and the vertical axis represents the degree of similarity. FIG. 5 shows an example in which the user partial logs LG2-1 and LG2-2 have a relatively low degree of similarity with the expert partial log, and the user partial log LG2-3 has a relatively high degree of similarity with the expert partial log. In this case, the related log extraction unit 133 can determine whether each user partial log is an operation log for the skill item "port scan" by comparing each similarity with a predetermined similarity threshold Sth. In the example of FIG. 5, the user partial log LG2-3 is determined to be an operation log for the skill item "port scan."

[0043] FIG. 6 is a diagram (part 2) illustrating an example of a method by which the related log extraction unit 133 extracts related logs from a user's operation log. Graph G2 in FIG. 6 shows the results of calculating the similarity for all skill items for the user's operation log LG2 shown in FIG. 5. Skill item A, skill item B, skill item C, and skill item D in FIG. 6 are examples of "all skill items." That is, in the example of FIG. 6, four similarities are calculated for each user partial log. It is assumed here that the user's operation log LG2 is divided into eight user partial logs LG2-1 to LG2-8.

[0044] In the example of FIG. 6, among the similarities of the user partial log LG2-1, the similarity with skill item A exceeds the threshold Sth; among the similarities of the user partial log LG2-3, the similarity with skill item B exceeds the threshold Sth; among the similarities of the user partial log LG2-6, the similarity with skill item C exceeds the threshold Sth; and among the similarities of the user partial log LG2-8, the similarity with skill item D exceeds the threshold Sth. In this case, the related log extraction unit 133 can extract, from the user operation log LG2, the user partial log LG2-1 as the related log of skill item A, the user partial log LG2-3 as the related log of skill item B, the user partial log LG2-6 as the related log of skill item C, and the user partial log LG2-8 as the related log of skill item D. Here, the related logs are an example of "second user operation history information."

[0045] For simplicity, the case where a related log is extracted for each of all skill items A to D has been described, but which skill item a related log is extracted for depends on the content of the operation contained in the user's operation log, so it is not necessary to extract a related log for all skill items. Also, for simplicity, the case where one user partial log is extracted as a related log for one skill item has been described, but which skill item to study, when, and how many times is up to the user, so multiple user partial logs may be extracted as a related log for one skill item.

[0046] 7 is a diagram illustrating skill level analysis by the skill evaluation unit 134. As shown in FIG. 7, the skill evaluation unit 134 inputs the related logs of each skill item extracted by the related log extraction unit 133 into the inference model 123, and obtains a skill level judgment result as an output of the inference model 123. As described above, since the inference model 123 is configured using a large-scale language model, a certain degree of analysis is possible even if only the related logs are provided for analysis. However, the inference accuracy may be improved by inputting additional information to the inference model 123 in the following manner.

[0047] For example, the skill evaluation unit 134 may input content information of a skill item corresponding to the related log as additional information to the inference model 123. The content information of the corresponding skill item is expected to help the inference model 123 recognize the skill item corresponding to the related log. In addition, the content information of the corresponding skill item is expected to help the inference model 123 understand the intention of the user's operation.

[0048] Furthermore, for example, the skill evaluation unit 134 may input the expert partial log of the skill item corresponding to the related log as additional information to the inference model 123. The expert partial log of the corresponding skill item is expected to be useful for the inference model 123 in recognizing a desirable answer (operation) for the skill item. In this way, by using the content information and the expert partial log of the corresponding skill item as additional information, it is expected that the inference accuracy of the inference model 123 will be improved.

[0049] For example, Figures 8 to 12 are diagrams (No. 1 to No. 5) showing examples of instructions (instructions) that the skill evaluation unit 134 issues to the inference model 123 when instructing the inference model 123 to perform a skill level analysis. In the examples, {{}} represents a variable, and {%%} represents a so-called template tag. As shown in the examples of Figures 8 to 12, the inference accuracy of the inference model 123 is improved by inputting related logs to the inference model 123 along with instructions (prompts) containing supplementary information such as the intention and purpose of the analysis, procedures, assumptions, precautions, limitations, and variable explanations. For example, the example instructions in Figures 8 to 12 include descriptions of the evaluation criteria, judgment results, output format, and variable explanations for the skill level analysis. Furthermore, the example instructions in Figures 8 to 12 present the necessary work steps for the skill level analysis to be performed by the inference model 123 in a linked order. This is expected to improve the inference accuracy of the inference model 123 and enable stable inference results to be obtained.

[0050] FIG. 13 is a diagram illustrating the process of recommending learning content to a user by the learning content recommendation unit 135. The learning content recommendation unit 135 has a function of suggesting learning content to a user based on the results of skill level analysis by the skill evaluation unit 134. In this embodiment, a case where a user takes learning content that consists of exercises across multiple skill items (hereinafter referred to as "comprehensive practice content") is described. Hereinafter, learning content consisting of exercises for each individual skill item will be referred to as "individual practice content" in contrast to comprehensive practice content that consists of exercises across multiple skill items.

[0051] In this case, when a user takes comprehensive practice content, the study content recommendation unit 135 determines the study content to recommend to the user based on the similarity between the model answers in the comprehensive practice content and the model answers in the individual practice content. More specifically, the model answer data is document-format data that describes the content of the model answers to the practice questions, explanations of the solutions, model operating procedures (commands, setting methods, etc.), etc.

[0052] First, the study content recommendation unit 135 obtains the model answer for the comprehensive exercise content that the user is taking (hereinafter referred to as the "comprehensive exercise model answer"). Here, it is assumed that the comprehensive exercise model answer has been stored in advance in the storage unit 120. The study content recommendation unit 135 breaks down the comprehensive exercise model answer in association with multiple skill items. Hereinafter, the comprehensive exercise model answer broken down by skill item will be referred to as a "partial model answer."

[0053] For example, the partial model answers may be obtained by manually breaking down the comprehensive exercise model answers. In this case, the partial model answers may be stored in memory unit 120 in advance. Alternatively, the partial model answers may be mechanically generated using a method similar to that used for the operation log (see FIGS. 5 and 6). For example, study content recommendation unit 135 may divide the comprehensive exercise model answers into predetermined unit amounts and compare each of the divided comprehensive exercise model answers with pre-prepared comparison data for each skill item to determine which skill item each corresponds to.

[0054] The comparison data for each skill item may contain text information related to the corresponding skill item. The comparison data may be, for example, a model answer for the comprehensive exercise broken down into sections corresponding to the skill items, or individual model answers (described below) for each skill item. For example, the comparison data may include tag information for each skill item. In this case, the study content recommendation unit 135 can break down the model answer for the comprehensive exercise into skill items by associating each of the divided model answers for the comprehensive exercise with the tag information of the comparison data that has a high similarity. For example, {skill} in the examples of Figures 8 to 12 is an example of tag information.

[0055] Figure 13 shows how to obtain a partial model answer MA3-1 corresponding to the skill item "port scan," a partial model answer MA3-2 corresponding to the skill item "directory scan," and a partial model answer MA3-3 corresponding to the skill item "SQL injection" from the comprehensive exercise model answer MA3. Here, the comprehensive exercise model answer is an example of the "first model answer," and the partial model answer is an example of the "second model answer."

[0056] Next, the learning content recommendation unit 135 calculates the similarity between the comprehensive exercise model answer (partial model answer) broken down by skill item and the model answer of each individual exercise content (hereinafter referred to as the "individual model answer"). Specifically, scoring is performed based on the degree of match as text information and the similarity of the procedures and commands described, and the degree of similarity between the comprehensive exercise model answer (partial model answer) and the individual model answer corresponding to each skill item is determined. The example in FIG. 13 shows a case where the highest similarity (0.9) was obtained for the partial model answer MA3-2 with the individual model answer MA41 corresponding to "directory scan" among the individual model answers MA41 to MA44. In this case, the learning content recommendation unit 135 can recommend the individual exercise content corresponding to the individual model answer MA41 with the highest similarity. Here, the individual model answer is an example of the "third model answer."

[0057] The study content recommendation unit 135 may calculate the similarity between a partial model answer for each skill item and multiple individual model answers, and present the recommended study content in descending order of similarity. For example, as shown in FIG. 13, if the similarity threshold is set to 0.5 and multiple individual model answers MA41 and MA43 are calculated to have similarities above the threshold (0.9 and 0.5) for the same partial model answer MA3-2, the individual practice content corresponding to the individual model answers MA41 and MA43 may be recommended together. Furthermore, the threshold may be changed to a larger or smaller value as needed. For example, the threshold may be adjusted according to the priority of each skill item. The priority may be uniform for all users, or may be determined for each user depending on their learning situation.

[0058] Furthermore, based on the analysis results of the skill evaluation unit 134, the learning content recommendation unit 135 may perform filtering to prioritize skill items that the user is particularly weak at or skill items for which the user has many incorrect answers in practice problems. For example, a possible configuration would be to lower the similarity threshold for high-priority skill items and raise the threshold for low-priority skill items. This makes it possible to recommend learning content that is tailored to the proficiency level and learning situation of each individual user.

[0059] According to the embodiment described above, it is possible to more efficiently train security personnel. More specifically, according to the online learning system 1 of the embodiment, it is possible to judge the level of a user as a security personnel by taking into account not only the level of knowledge but also the level of the user's actual device operation skills, thereby training more advanced security personnel who can take necessary actions based on their knowledge.

[0060] In general, even if the purpose of a device operation is the same, the content of the operation can vary depending on the person performing it and the timing of the operation. For example, incorrect commands may be entered, commands may be entered incorrectly, commands may be entered incorrectly, commands may be entered that are unrelated to the purpose, or the order in which commands are executed may differ depending on the person. Because it is difficult to process such inconsistent information, such as operation logs, using a rule-based approach, it has traditionally been difficult to accurately determine a user's skill level from the contents of the operation logs. In contrast, the online learning system 1 of the embodiment has discovered the new insight that device operation logs are also a form of natural language and is able to recognize and determine the intent and context (including the similarity and appropriateness of operations) using a large-scale language model. This makes it possible to accurately determine a user's skill level from the user's operation log.

[0061] <Modification> In the above embodiment, a case has been described in which an expert partial log (positioned as a correct answer or model answer) is input to the inference model 123 (large-scale language model) in the instruction for skill level analysis. On the other hand, a large-scale language model may understand a given purpose or intention and spontaneously execute the necessary processing or inference, even if a clear means or method for achieving the purpose is not given. Therefore, if the instruction can be adjusted so that the inference model 123 spontaneously infers an expert partial log from given information and learned knowledge, it is not necessarily necessary to supply the expert partial log to the inference model 123.

[0062] In the above embodiment, instructions common to all skill items are exemplified for skill level analysis (see Figures 8 to 12). However, there are cases where the points to note in the operation log and the evaluation perspectives differ significantly for each skill item. In such cases, individual instructions may be used for each skill item. In this embodiment, instructions tailored to each skill item are input to the inference model 123. For example, in the examples of Figures 11 and 12, the content of the instructions is changed depending on the log_type (a parameter indicating the type of log and corresponding to the skill item). This reduces the possibility that the inference model 123 will make an incorrect inference, and is expected to improve inference accuracy.

[0063] In the above embodiment, the content server 100 has been described as having an evaluation function for evaluating the skill level of a user and a recommendation function for recommending learning content to the user, but the evaluation function and recommendation function may also be configured as separate evaluation devices and recommendation devices, respectively.

[0064] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0065] 1. Online learning system 10 User terminal 20 Practice Environment Server 100 Content Servers 110 Communications Department 120 Storage section 121 Content Information 122 Expert Information 123 Inference Model 124 User information 130 Control Unit 131 Content Control Unit 132 User log acquisition unit 133 Related Log Extraction Unit 134 Skills Evaluation Department 135 Learning Content Recommendation Department

Claims

1. a storage unit that stores a plurality of second model answers obtained by classifying a first model answer, which is a model answer for a task to be assessed, into skill items, which are predetermined skill classification items; a recommended assignment selection unit that selects another assignment to be recommended to the user taking the assignment to be evaluated according to the degree of similarity between the second model answer and a third model answer that is a model answer for another assignment; Equipped with the recommended task selection unit selects another task corresponding to a third model answer whose similarity to the second model answer exceeds a predetermined threshold, and adjusts the threshold according to the priority of each skill item. Recommendation device.

2. the recommended task selection unit determines the priority in accordance with the user's learning status; The recommendation device according to claim 1 .

3. A recommendation method executed by an information processing device, a first step of storing a plurality of second model answers obtained by classifying a first model answer, which is a model answer for a task to be judged, into skill items, which are predetermined skill classification items; a second step of selecting another assignment to be recommended to the user taking the assignment to be evaluated according to the degree of similarity between the second model answer and a third model answer that is a model answer for another assignment; A recommendation method comprising: The second step selects another task corresponding to a third model answer whose similarity to the second model answer exceeds a predetermined threshold, and adjusts the threshold according to the priority of each skill item. How to recommend.

4. In the information processing device, a first step of storing a plurality of second model answers obtained by classifying a first model answer, which is a model answer for a task to be judged, into skill items, which are predetermined skill classification items; a second step of selecting another assignment to be recommended to the user taking the assignment to be evaluated according to the degree of similarity between the second model answer and a third model answer that is a model answer for another assignment; A computer program for executing The second step selects another task corresponding to a third model answer whose similarity to the second model answer exceeds a predetermined threshold, and adjusts the threshold according to the priority of each skill item. Computer program.

Images