Information Processing System, Information Processing Method, and Program

Abstract

Provided is an information processing system or the like that enables an evaluation subject to efficiently answer security-related questions. 【Solution means】According to one aspect of the present invention, there is provided an information processing system including at least one processor configured to execute the following steps by reading a program. In a data acquisition step, answer data is acquired by combining answered questions related to the security of the evaluation subject and input answers of the evaluation subject to the answered questions. In a data arrangement step, an information processing system is provided that performs at least one of an integration process of integrating two or more overlapping answer data into one answer data and a division process of dividing answer data in which answered questions can be divided into a plurality of questions or input answers can be divided into a plurality of answers into a plurality of answer data.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes a system audit support system including a checkpoint database that stores checkpoint array information data for specifying each checkpoint when auditing an object to be audited in a regular order, and stores evidence array information data for specifying evidence necessary for evaluating each checkpoint in association with the checkpoint array information data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need for a technique that enables a respondent to efficiently answer check items.

[0005] In view of the above circumstances, the present invention aims to provide an information processing system and the like that enable an evaluation target person to efficiently answer questions related to security.

Means for Solving the Problems

[0006] According to one aspect of the present invention, there is provided an information processing system comprising at least one processor configured to execute the following steps by reading a program. In a data acquisition step, response data is acquired by combining answered questions regarding the security of an evaluation target person and input answers of the evaluation target person to the answered questions. In a data arrangement step, at least one of an integration process of integrating two or more duplicate response data into one response data and a division process of dividing response data in which the answered questions can be divided into a plurality of questions or the input answers can be divided into a plurality of answers into a plurality of response data is performed.

[0007] According to such an aspect, since the response data, which is a combination of past questions and answers, is arranged, by referring to such response data, the evaluation target person can efficiently answer questions regarding security.

Brief Description of Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the following embodiments can be combined with each other.

[0010] Incidentally, a program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium that can be read by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its function is realized on a client terminal (so-called cloud computing).

[0011] Also, in various information processes according to an embodiment, an input and an output corresponding to the input can be realized. Here, if an output is obtained as a result of the input, the mode of information (hereinafter referred to as reference information) referred to in such information processing is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, a predetermined function (including a judgment formula such as a regression formula constructed by a statistical method), a learned model in which the correlation between the input and the output is learned in advance, or a large language model capable of outputting a desired result by inputting a prompt.

[0012] Also, in one embodiment, the "part" may include, for example, a combination of hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various information is handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0013] Furthermore, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, a memory, etc. Also, the processor may be a general-purpose processor or a dedicated circuit. That is, it includes application specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0014] 1. Hardware Configuration In this section, the hardware configuration will be described.

[0015] <Information Processing System 1> FIG. 1 is a configuration diagram showing Information Processing System 1. Information Processing System 1 includes a communication line 2, a server device 10, a plurality of target terminals 20, at least one evaluator terminal 30, and a plurality of requester terminals 40. The server device 10, the target terminals 20, the evaluator terminal 30, and the requester terminals 40 are configured to be able to communicate with each other through the communication line 2. The connections of the server device 10, the target terminals 20, the evaluator terminal 30, and the requester terminals 40 may be wired or wireless.

[0016] Information Processing System 1 constitutes at least a part of a security evaluation system used by, for example, a plurality of evaluation targets (the first evaluation target U1 and the second evaluation target U2), at least one evaluator U3, and a plurality of evaluation requesters (the first evaluation requester U4 and the second evaluation requester U5). Information Processing System 1 mainly performs evaluations of the security of commercial products and the like provided by the evaluation targets, provision of security evaluations, and the like. For example, Information Processing System 1 mainly provides security evaluation services by evaluators. In one embodiment, Information Processing System 1 consists of one or more devices or components. Hereinafter, these components will be described.

[0017] <Server Device 10> FIG. 2 is a block diagram showing the hardware configurations of the server device 10 and the target terminal 20. As shown in FIG. 2A, the server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a communication bus 14. The control unit 11, the storage unit 12, and the communication unit 13 are electrically connected to each other inside the server device 10 via the communication bus 14.

[0018] <Control Unit 11> The control unit 11 processes and controls the overall operation related to the server device 10. The control unit 11 is, for example, a Central Processing Unit (CPU). The control unit 11 realizes various functions related to the server device 10 by reading a predetermined program stored in the storage unit 12. That is, the information processing by software stored in the storage unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and the server device 10 may have a plurality of control units 11 for each function. Further, the server device 10 may be configured by a combination of these.

[0019] <Storage unit 12> The storage unit 12 stores various information defined by the foregoing description. This can be implemented, for example, as a storage device such as a Solid State Drive (SSD) that stores various programs and the like related to the server device 10 executed by the control unit 11, or as a memory such as a Random Access Memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of the program. The storage unit 12 stores various programs, variables, etc. related to the server device 10 executed by the control unit 11.

[0020] <Communication unit 13> The communication unit 13 preferably uses wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may include wireless LAN network communication, mobile communication such as LTE / 5G, BLUETOOTH (registered trademark) communication, etc. as necessary. That is, it is more preferable to implement it as a collection of these plural communication means. That is, the server device 10 may communicate various information from the outside via the communication unit 13 and the network.

[0021] The server device 10 may be in an on-premises form or in a cloud form. The server device 10 in the cloud form may provide the above-described functions and processes, for example, in the form of SaaS (Software as a Service), cloud computing.

[0022] <Target person terminal 20> The target person terminal 20 is an information processing terminal used by a person whose security is to be evaluated. The "person to be evaluated" includes a trading partner organization or its person in charge having any form of trading relationship such as the provision of a commercial product (goods or services), the provision of a cloud service, or the outsourcing of business to the evaluation requester. The person to be evaluated is, for example, a business operator such as a service provider, a supplier, or an outsourcing destination organization for the person to be evaluated. Note that the "trading partner organization" includes a primary trading partner organization that directly receives a transaction from the person to be evaluated, a secondary trading partner organization that receives a secondary transaction from the primary trading partner organization, and the like.

[0023] In addition, the "organization" includes for-profit corporations (e.g., companies, etc.), non-profit corporations (e.g., cooperatives, foundations, etc.), public corporations (e.g., local public entities, etc.).

[0024] As shown in FIG. 2B, the target person terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, and a communication bus 26. The control unit 21, the storage unit 22, the communication unit 23, the input unit 24, and the output unit 25 are electrically connected via the communication bus 26 inside the target person terminal 20. The descriptions of the control unit 21, the storage unit 22, and the communication unit 23 are omitted because they are the same as the descriptions of the respective units in the server device 10.

[0025] <Input unit 24> The input unit 24 receives operation inputs made by the user. The operation inputs are transferred as command signals to the control unit 21 via the communication bus 26. The control unit 21 can execute predetermined control and calculations as necessary based on the transferred command signals. The input unit 24 may be included in the housing of the subject terminal 20, or may be externally attached. For example, the input unit 24 may be integrated with the output unit 25 and implemented as a touch panel. When the input unit 24 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 24. As the input unit 24, instead of a touch panel, a switch button, a mouse, a track pad, a QWERTY keyboard, etc. can be adopted.

[0026] <Output unit 25> The output unit 25 displays a screen of a graphical user interface (GUI) operable by the user. The output unit 25 may be included in the housing of the subject terminal 20, or may be externally attached. Specifically, the output unit 25 can be implemented as a display device such as a CRT display, a liquid crystal display, an organic EL display, a plasma display, etc. These display devices are preferably selectively implemented according to the type of the subject terminal 20.

[0027] <Evaluator terminal 30> The evaluator terminal 30 is an information processing terminal used by an evaluator who evaluates the responses of the subjects to be evaluated. The evaluator is an organization such as a company that conducts security evaluations (for example, a security evaluation company that conducts security evaluations of services such as cloud services, software, etc.) or its staff. The evaluator conducts security evaluations of the subjects to be evaluated, for example, from the evaluator terminal 30 via the information processing system 1.

[0028] FIG. 3 is a block diagram showing the hardware configuration of the evaluator terminal 30 and the requester terminal 40. As shown in FIG. 3A, the evaluator terminal 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, and a communication bus 36. The control unit 31, the storage unit 32, the communication unit 33, the input unit 34, and the output unit 35 are electrically connected inside the evaluator terminal 30 via the communication bus 36. The descriptions of the control unit 31, the storage unit 32, the communication unit 33, the input unit 34, and the output unit 35 are omitted because they are the same as the descriptions of the respective units in the subject terminal 20.

[0029] <Requester terminal 40> The requester terminal 40 is an information processing terminal used by an evaluation requester who requests an evaluation of the security of an evaluation subject. The "evaluation requester" is an individual, an organization, or a person in charge of the organization that has a transaction relationship with the evaluation subject. The evaluation requester is an organization that requests a transaction from the counterparty organization (evaluation subject), and receives and uses the commercial materials provided by the counterparty organization. For example, when the commercial material being traded is an item, the evaluation requester is a purchaser of the item (finished product or part) manufactured by the counterparty organization. Also, for example, when the commercial material being traded is a service, the evaluation requester is a user of the service provided by the counterparty organization, for example, a user of the cloud service provided by a cloud service provider. Further, for example, when the commercial material being traded is a business (operation), the evaluation requester is the contractor or the consignor of the business.

[0030] The evaluation requester includes not only individuals or organizations that conduct transactions (such as service use, business consignment, etc.) to be evaluated with the evaluation subject, but also individuals or organizations that request an evaluation of a specific service. For example, the evaluation requester may include companies that want to evaluate the security of cloud services, etc. used in their own company, companies that want to confirm such an evaluation, etc.

[0031] As shown in FIG. 4B, the requester terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a communication bus 46. The control unit 41, the storage unit 42, the communication unit 43, the input unit 44, and the output unit 45 are electrically connected inside the requester terminal 40 via the communication bus 46. The requester terminal 40 is an information processing terminal used by a user of the security evaluation service provided by the server device 10. The descriptions of the control unit 41, the storage unit 42, the communication unit 43, the input unit 44, and the output unit 45 are omitted because they are the same as the descriptions of the respective units in the subject terminal 20.

[0032] 2. Functional Configuration In this section, the functional configuration of this embodiment will be described. The information processing by software stored in the storage unit 12 is specifically realized by the control unit 11 which is an example of hardware, and thus can be executed as each functional unit included in the control unit 11 (at least one processor included in the information processing system 1).

[0033] FIG. 4 is a block diagram showing the functions realized by the server device 10 (control unit 11), the subject terminal 20 (control unit 21), the evaluator terminal 30 (control unit 31), and the requester terminal 40 (control unit 41).

[0034] As shown in FIG. 4A, the server device 10 (control unit 11) includes a basic display control unit 111, a response registration unit 112, an evaluation reception unit 113, a data acquisition unit 114, a data arrangement unit 115, a data registration unit 116, a designation reception unit 117, a response generation unit 118, a report output unit 119, and an artificial intelligence unit 120.

[0035] As shown in FIG. 4B, the subject terminal 20 (control unit 21) includes a display unit 211 and an operation acquisition unit 212. As shown in FIG. 4C, the evaluator terminal 30 (control unit 31) includes a display unit 311 and an operation acquisition unit 312. As shown in FIG. 4D, the requester terminal 40 (control unit 41) includes a display unit 411 and an operation acquisition unit 412.

[0036] <Basic Display Control Unit 111> The basic display control unit 111 is configured to display various information on the subject terminal 20, the evaluator terminal 30, and the requester terminal 40. For example, the basic display control unit 111 displays an input form for the evaluation subject to enter an answer, the answer registered by the evaluation subject, the evaluation result of the answer by the evaluator, etc. on the display unit 211 of the subject terminal 20, the display unit 311 of the evaluator terminal 30, or the display unit 411 of the requester terminal 40.

[0037] <Answer registration unit 112> The answer registration unit 112 is configured to register information related to the security of the evaluation subject for creating a report or a statement to be presented to the evaluation requester. Specifically, the answer registration unit 112 causes the subject terminal 20 to display a predetermined input form for receiving an answer to a security-related question from the evaluation subject, and registers the answer input in the input form as evaluation information.

[0038] The evaluation subject inputs an answer according to service attributes such as, for example, the type of service targeted, the service recipient, and the type of plan included in the service. That is, an evaluation subject that provides a plurality of services or plans may input an answer for each of these service attributes, and the answer registration unit 112 may register evaluation information for each service attribute.

[0039] The "service" provided by the evaluation subject includes services provided through the Internet such as SaaS, IaaS (Infrastructure as a Service), PaaS (Platform as a Service), cloud services, etc. In this case, the "evaluation subject" includes cloud service providers, etc. Also, the "service" provided by the evaluation subject may be a service provided to a trading partner, a customer, etc.

[0040] A "question" is a matter that requests an answer from the person or entity being evaluated. For example, it may include matters related to the security of the person or entity being evaluated themselves, the security of the services provided by the person or entity being evaluated, etc. Further, the "question" may include matters related to the security of other organizations included in the supply chain of the person or entity being evaluated. The "other organizations" include, for example, the trading partner organizations of the person or entity being evaluated, such as organizations to which the person or entity being evaluated has outsourced its business (including subcontractors, sub-subcontractors, sub-sub-subcontractors, etc.). The question may include what is called a security checklist.

[0041] "Evaluation information" is composed of the answers of the person or entity being evaluated to each of a plurality of questions regarding the security of the service, and is the information that is the subject of checking and evaluation by the evaluator. The evaluation information may be evaluated only once or may be evaluated multiple times. That is, the evaluation of the evaluation information may include an initial evaluation (so-called review), an intermediate evaluation, a final evaluation, etc. The initial evaluation and the intermediate evaluation are evaluations in which another evaluation is performed after the evaluation. The multiple evaluations may be performed by the same evaluator or may be performed by another evaluator (so-called reviewer, etc.). In the final evaluation, an evaluation used in a report or statement provided to the evaluation requester is created. Further, when the evaluation information is corrected, updated, etc. by the person or entity being evaluated, it may be evaluated again by the evaluator. The evaluation result of the evaluation information by the evaluator is used as part of a report or statement or as reference information.

[0042] The evaluation information may include the question corresponding to the answer together with the answer. Further, the evaluation information may include service attributes (for example, information indicating the type of service targeted, the service recipient, the types of plans and options included in the service, or a combination thereof).

[0043] The evaluation information is registered in, for example, the response database of the storage unit 12. The responses to the respective questions included in the evaluation information are the responses input by the evaluation target person to the input form for each question presented in the input form, which are registered in the response database. The response to one question may be hierarchical.

[0044] "Registration of evaluation information" includes both registration (final registration) in a state where responses have been input for all questions and registration (draft saving) in a state where responses have been input only for some questions (a state where some responses are not input). Further, "draft saving" may include registration in a state where responses have been input for all questions and before submitting the responses to the evaluator.

[0045] The questions in the evaluation information include, for example, the acquisition status or evaluation of authentication related to security, service level, scope of responsibility, security measures within the evaluation target person, handling of data related to the evaluation requester, handling of data managed by the evaluation target person, confirmation of service development, maintenance or operation policies (such as account management), etc. (those that request responses from the evaluation target person).

[0046] <Evaluation reception unit 113> The evaluation reception unit 113 is configured to receive an evaluation of the evaluation information registered by the response registration unit 112 from the evaluator and register the evaluated evaluation information together with the evaluation result in the response database or the like.

[0047] "Evaluation" is an act of checking for deficiencies or contradictions in the responses included in the evaluation information, creating pointed-out matters (comments) for the content of the responses, appraising the content of the responses, scoring the responses based on predetermined criteria, etc. As described above, the evaluation may be performed by a plurality of evaluators, and an evaluation of the registered evaluation information may be requested from a first evaluator to another second evaluator. In that case, the evaluation reception unit 113 may receive the evaluation from the second evaluator after the evaluation by the first evaluator.

[0048] For example, when the evaluation is performed in two stages, in the initial evaluation (review), confirmation of deficiencies and contradictions, creation of pointed-out items, etc. are carried out. Subsequently, for the evaluation information after the initial evaluation, based on predetermined criteria, evaluations such as grading and scoring of the content (answer), creation of comments on the entire evaluation information, etc. are performed as the final evaluation, and a report or statement (e.g., a security report) including the results is created. The report or statement is created by the final evaluator or the evaluation unit possessed by the control unit 11. The evaluation unit is configured to generate part or all of the report or statement based on the input evaluation information. The evaluation result of the evaluation information may be included in the report or statement as part of the report or statement or as reference information. The report or statement is transmitted to the requester terminal 40 of the evaluation requester who requested the report or statement.

[0049] The evaluation reception unit 113 receives input of characters, sentences, etc. used for evaluation from the evaluator terminal 30. Note that the evaluation is typically performed on evaluation information for which answers have been given to all questions. Evaluation information with unanswered questions (questions for which answers have not been registered) may not be the subject of evaluation, and the evaluation reception unit 113 may not receive the evaluation. Also, even for evaluation information including unanswered questions (questions for which answers have not been registered), only the answered questions may be partially subject to evaluation.

[0050] <Data acquisition unit 114> The data acquisition unit 114 is configured to acquire and record data for automatically generating answers to questions in the evaluation information. Specifically, the data acquisition unit 114 acquires answer data combining answered questions regarding the security of the evaluation target person and the input answers of the evaluation target person to the answered questions.

[0051] For example, the data acquisition unit 114 receives, from the subject terminal 20, answered questions regarding the security of the service provided by the subject to be evaluated, the input answers of the subject to be evaluated to the answered questions, and the designation of the category assigned to the input answers, and adds the answer data including the answered questions, the input answers, and the category to the database for generation. Note that the answer data added to the database for generation may be referred to as knowledge. One piece of answer data includes at least the data of the answered questions, the data of the input answers, and information indicating the category. Note that the data of the answered questions and the data of the input answers may each be text data, vector data to be described later, or both of these.

[0052] The category assigned to the input answers is information indicating the type of service, the destination of the service (e.g., the attributes of the evaluation requester such as the organization name of the evaluation requester), the type of plan, option, etc. included in the service, or a combination of these. Thereby, the input answers can be labeled using the types of services, plans, options, etc. provided by the subject to be evaluated. As a result, the accuracy of the answers generated by the answer generation unit 118 is improved. In addition, it becomes possible to generate answers according to the types of services, plans, options, etc. used by the evaluation requester. The category may also be information indicating business, department, etc.

[0053] "Type of service" may be read as the service name and represents the type or name of the service for which the input answer is targeted. For example, when the person to be evaluated provides Service A and Service B, at least one of "Service A" and "Service B" may be assigned to the input answer as the type of service. "Type of plan included in the service" is, for example, one or more types of contract plans set for the same service (e.g., Service A), such as "Standard Plan" and "Premium Plan". Multiple plans set for the same service differ from each other in terms of, for example, available functions, usage period, usage fees, etc. Also, even for the same service, different types of categories may be assigned depending on the service recipient, etc.

[0054] Also, the category of the input answer does not necessarily have to be entered. That is, the answer data acquired by the data acquisition unit 114 does not necessarily have to include a category.

[0055] The answered questions included in the answer data do not necessarily have to be assigned a category, or a different type of category (hereinafter referred to as a question category) from the category assigned to the input answer may be assigned. Also, a question category may be assigned to the question before answering. The question category is used, for example, as a narrowing-down condition for the answer data in the display of the answer data, generation of an answer using the answer data, etc.

[0056] The question category classifies the content of the answered question or the question before answering. For example, for an answered question or a question before answering regarding a contract, a question category of "Contract" is assigned, and for an answered question or a question before answering regarding the company's policy or rules, a question category of "Policy·Rules" is assigned. That is, the category assigned to the input answer and the question category assigned to the answered question or the question before answering differ in the viewpoints for assignment.

[0057] When a question category is assigned to an answered question or a question before an answer, for example, the data acquisition unit 114 may accept the selection of a question category on the subject terminal 20 from a predefined group of categories. This group of categories is composed of categories different from the categories assigned to the input answers. Also, the question category may be automatically assigned by the data acquisition unit 114 with reference to the questions included in the generation database and their question categories. For example, the data acquisition unit 114 may extract accumulated questions similar to the received question and assign the question category assigned to the question to the received question.

[0058] The answer data may include first answer data based on the answered questions and input answers input by the evaluation target person without using the input form provided by the answer registration unit 112, and second answer data based on the evaluation information. Further, the answer data may include third answer data based on the answers generated by the answer generation unit 118. Therefore, the data acquisition unit 114 may acquire the answer data by at least one of a first data acquisition flow for acquiring the first answer data and a second data acquisition flow for acquiring the second answer data, and a third data acquisition flow for acquiring the third answer data. Furthermore, it is preferable that all of the first data acquisition flow, the second data acquisition flow, and the third data acquisition flow are configured to be executable.

[0059] In the first data acquisition flow, the data acquisition unit 114 receives the input of data combining the answered questions and the entered answers from the subject terminal 20. For example, the data acquisition unit 114 causes the subject terminal 20 to display a UI including a question input field, an answer input field, etc., and receives the input of the answered questions and the entered answers. Further, the data acquisition unit 114 receives the input of a category for the entered answer from the subject terminal 20. The data acquisition unit 114 generates first answer data in which the received category is assigned to the entered answer. Thereby, the entered answer without a category can be registered as the categorized first answer data to be used by the answer generation unit 118. Note that the input order (receiving order) of the answered questions, the entered answers, and the category is not limited.

[0060] For example, the data acquisition unit 114 refers to the types of services and plans provided and registered in advance as user information in the information processing system 1 by the evaluation subject, and presents these to the subject terminal 20 as candidates for a type of category, "type of service" or "type of plan included in the service", and may receive the selection of a category from the subject terminal 20. Also, the data acquisition unit 114 may present the registered evaluation requester, such as an organization that has provided a report or a written report in the past in the information processing system 1, as a candidate for a type of category, "service recipient".

[0061] Also, the data acquisition unit 114 may receive the input (specification) of a plurality of categories for one entered answer. That is, the data acquisition unit 114 may assign a plurality of categories to one entered answer. For example, in the case of an answer common to a plurality of services, a plurality of service names are assigned to the answer as categories.

[0062] Furthermore, for input answers that do not rely on categories (i.e., not classified into specific categories and applicable to any service, plan, provider, etc.), the person being evaluated does not have to enter a category. Also, the data acquisition unit 114 may accept the designation (input) of an "uncategorized category" for input answers that do not rely on categories. For example, the data acquisition unit 114 may accept the designation (input) of a "common" category as the uncategorized category.

[0063] FIG. 5 is a diagram showing an example of the data input screen DD displayed on the subject terminal 20. The data input screen DD includes a registration check column CC, a question input column QC1, an answer input column AC1, a memo column MC, a service selection column SC, a past data comparison column PC, and a registration button B11.

[0064] In the registration check column CC, check boxes for selecting, as first answer data, the answered questions and input answers entered by the person being evaluated in the question input column QC1 and the answer input column AC1 are displayed. The check boxes are attached one by one in a row (i.e., one for each question).

[0065] The person to be evaluated inputs the paired answered question and the input answer into the question input field QC1 and the answer input field AC1 in the same row, respectively. The input of the question and the answer is performed, for example, by copy and paste from reference data (for example, a CSV file such as an Excel file or a Google spreadsheet) including the question and the answer prepared by the person to be evaluated on the subject terminal 20. Further, the data acquisition unit 114 may receive the upload of the reference data, extract the question and the answer from the reference data, and display them in the question input field QC1 and the answer input field AC1. Furthermore, the data acquisition unit 114 may accept operations such as repeated copying of cells on a matrix including the question input field QC1, the answer input field AC1, etc. on the data input screen DD. The first answer data is registered, for example, as a part of a file (for example, an Excel file, a Google spreadsheet, etc.) including the answers to the security check sheet created by the person to be evaluated in the past.

[0066] The memo field MC is a field where the person to be evaluated enters comments. The information entered in the memo field MC is registered, for example, as a part of the first answer data, but is not referred to when generating the answer.

[0067] The service selection column SC is a column where the person being evaluated inputs, as a category, the service or the like that is the subject of the answer entered in the answer input column AC1. In the service selection column SC, selection buttons SB for selecting a service from a list are arranged. When an input operation is performed on the selection button SB, service candidates are displayed in a form such as a pull-down menu. When the person being evaluated selects the type of service from the candidates, the service selected by the person being evaluated (corresponding to the "category to be assigned to the entered answer") is displayed in the service selection column SC. In the case of answers corresponding to multiple services, the multiple services are displayed within the frame of a single service selection column SC. In the example of FIG. 5, the service name (type of service) is displayed in the service selection column SC, but the service provider (for example, the name of the evaluation requester, etc.), the plan name (type of plan, etc.) may be displayed as options (categories). Also, the service name may be text-input by the person being evaluated. Note that what is input in the service selection column SC is not limited to the service name, and any information that can be used as a category for the entered answer is acceptable. For example, it may be information indicating the type of service, the service provider (for example, the attribute of the evaluation requester such as the organization name of the evaluation requester), the plan included in the service, the type of options, etc., or a combination thereof.

[0068] Note that if there are services or plans with different answer contents for the same question, the person being evaluated may create and input data with the same question content but different answer contents and service names. That is, for answered questions, if the contents of the entered answers are different, duplicates of the same content can be registered.

[0069] In the past data comparison column PC, the comparison result with the answered questions and the entered answers included in the answer data already registered in the generation database (only the first answer data, or any one of the first answer data, the second answer data, and the third answer data) is displayed. The comparison result includes, for example, discrepancies (differences) in the content of the entered answers in the answered questions related to each other, and duplication of the answered questions. In the display of duplicate answered questions in the past data comparison column PC, a link to display other duplicate answered questions, the number of duplicate answered questions, etc. may be included.

[0070] When an input operation is performed on the registration button B11, the combination of the answered questions and the entered answers selected (checked) in the registration check column CC is registered in the generation database as the first answer data.

[0071] In the second data acquisition flow, the data acquisition unit 114 creates the second answer data from the evaluation information registered by the answer registration unit 112 and adds it to the generation database. As a result, since the answer data can be created using the pre-created evaluation information, the labor of inputting answered questions, entered answers, etc. by the person to be evaluated can be reduced.

[0072] The first answer data and the second answer data may be added to one common generation database, or may be added to different sub-databases included in the generation database, respectively. That is, the data acquisition unit 114 may register the first answer data in the first sub-database included in the generation database and register the second answer data in the second sub-database included in the generation database.

[0073] The data acquisition unit 114 presents, for example, a list of registered evaluation information to the subject terminal 20 and accepts the selection of the evaluation information to be added to the generation database as the second answer data from the subject terminal 20. The data acquisition unit 114 accepts the selection of the evaluation information by the subject terminal 20 as the designation of a plurality of answered questions and entered answers included in the evaluation information.

[0074] When the evaluation information includes service attribute information, the data acquisition unit 114 acquires the service attribute as the category of the answered response. That is, the data acquisition unit 114 accepts the selection of the evaluation information as the designation of the category. On the other hand, when the evaluation information does not include service attribute information, the data acquisition unit 114 may accept the input of the category for the answered response (evaluation information) from the subject terminal 20.

[0075] The evaluation information used by the data acquisition unit 114 as the source information of the second response data may be registered (that is, officially registered or saved as a draft), regardless of whether it has been evaluated. That is, the data acquisition unit 114 accepts both the evaluation information before evaluation and the evaluated evaluation information as the source information of the second response data. Note that the data acquisition unit 114 may accept only the evaluated evaluation information as the source information of the second response data. In the case of the evaluation information in the draft save state, the data acquisition unit 114 uses only the questions for which responses have been input to create the response data. Note that "evaluated evaluation information" refers to the evaluation information for which evaluation (review) has been performed by the evaluator at least once, and in the case of multiple evaluations, it includes those for which the final evaluation has not been performed.

[0076] In the third data acquisition flow, the data acquisition unit 114 adds the third response data, in which the designated questions received by the designated reception unit 117 described later are regarded as answered questions, the answers to the designated questions generated by the answer generation unit 118 and edited by the evaluation subject are regarded as answered responses, to the generation database. As a result, the data generated by the answer generation unit 118 and further corrected by the evaluation subject can be newly used as the response data, so that the response accuracy of the answer generation unit 118 is improved.

[0077] In addition, the data acquisition unit 114 may add third answer data to the generation database, which sets the specified questions that the answer generation unit 118 could not generate answers for as answered questions and the answers to the specified questions input by the evaluation target as input answers. Thereby, answer data for generating answers to questions that the answer generation unit 118 could not generate can be added.

[0078] The data acquisition unit 114 may vectorize at least a part of the text data (answered questions and input answers) received as the meta-information of the answer data and add the obtained vector data to the answer data. Thereby, in the answer generation unit 118, answer generation (search for reference answers) with reference to the prepared vector data becomes possible, and thus the processing speed of answer generation is increased. Specifically, the data acquisition unit 114 may vectorize at least the answered questions.

[0079] FIG. 6 is a diagram showing an example of an answer data management screen MD displayed on the target terminal 20. The answer data management screen MD includes an evaluation information display column EF, a knowledge display column NF, a knowledge addition button B21, and an answer generation button B22.

[0080] In the evaluation information display column EF, the evaluation information received by the data acquisition unit 114 is displayed in a list. Specifically, in the evaluation information display column EF, information such as the identification name (service name in this example), the final registration date and time, and the status of the evaluation information is displayed. For the status, the ones for which the second answer data based on the evaluation information has been added to the generation database (the ones that can be searched by the answer generation unit 118 described later) are displayed as "registered". Also, the ones for which the generation of the second answer data based on the evaluation information is not completed (for example, the ones for which the vectorization is not completed) are displayed as "registration in progress".

[0081] The knowledge display column NF includes a condition setting column TF, a knowledge editing button B23, a knowledge sorting button B24, and a knowledge list NL. In the knowledge list NL, the first answer data (knowledge) added by the data acquisition unit 114 to the generation database is displayed in a list. The condition setting column TF accepts input of conditions for narrowing down the knowledge to be displayed in the knowledge list NL. In the example of FIG. 6, the condition setting column TF accepts, as narrowing conditions, input of a search keyword, input of the last update date, and selection of the category (related service, etc.) of the answered question. As shown in FIG. 6, when the category is set to "all", knowledge in all categories (service names) (including knowledge without a category) is displayed. Note that in the knowledge display column NF, knowledge may be displayed in group units grouped by category.

[0082] When an input operation is performed on the knowledge editing button B23 in the knowledge display column NF, an editing screen for knowledge conforming to the data input screen DD in FIG. 5 is displayed on the target terminal 20, and the data acquisition unit 114 accepts editing of knowledge (answered questions, input answers, or categories) from the target terminal 20.

[0083] When an input operation is performed on the knowledge sorting button B24 in the knowledge display column NF, a screen for executing sorting of knowledge by a data sorting unit 115 described later is displayed. Also, when an input operation is performed on the knowledge sorting button B24, an integration process described later may be executed.

[0084] In the answer data management screen MD, when the knowledge addition button B21 is input-operated, the data input screen DD in FIG. 5 is displayed on the target terminal 20. Note that when no knowledge is registered, the knowledge addition button B21 may be displayed in the knowledge display column NF. When the answer generation button B22 is input-operated, a screen for causing an answer generation unit 118 described later to generate an answer is displayed.

[0085] <Data sorting unit 115> The data arrangement unit 115 is configured to arrange the response data acquired (registered) by the data acquisition unit 114. Specifically, the data arrangement unit 115 performs at least one of an integration process and a division process on the response data.

[0086] <Integration process> In the integration process, the data arrangement unit 115 integrates two or more overlapping response data into one response data. Note that the integration process may target only the first response data, or one or more of the first response data, the second response data, and the third response data. For example, the first response data and the second response data may be integrated by the integration process. Also, when categories are assigned to the response data, the response data in the same category are targeted for the integration process. The integration process may be started, for example, by an input operation to the knowledge arrangement button B24 in FIG. 6.

[0087] In the integration process, the data arrangement unit 115 may determine the response data to be integrated based on the similarity based on the comparison of the feature amounts of the answered questions included in two or more response data, the similarity based on the comparison of the input responses included in two or more response data, or a combination of these similarities. Thereby, it becomes possible to integrate overlapping response data within a similar range not limited to exact matches.

[0088] As the feature amounts of the answered questions or the input responses, for example, vector data obtained by vectorizing the sentences or words included in the answered questions or the input responses may be used. The vectorization is performed, for example, by natural language processing such as morphological analysis or quantification by known methods such as encoding. The similarity between the answered questions or between the input responses is represented by the difference in feature amounts (for example, the distance between vectors).

[0089] The data sorting unit 115 may determine that only the answer data whose similarity (e.g., cosine similarity) between answered questions is equal to or greater than a threshold value is duplicate data (i.e., answer data to be integrated), or may determine that only the answer data whose similarity between input answers is equal to or greater than a threshold value is duplicate data. Further, the data sorting unit 115 may determine that answer data whose similarity of at least one of the similarity between answered questions and the similarity between input answers is equal to or greater than a threshold value is duplicate data. Additionally, the data sorting unit 115 may determine that answer data whose similarity between answered questions and the similarity between input answers are each equal to or greater than a threshold value is duplicate data. Note that when using the similarity between answered questions and the similarity between input answers for determination, these threshold values may be different from each other or the same.

[0090] Also, the data sorting unit 115 may determine the answer data to be integrated by comparing the feature amounts (vector data) of the entire combinations of the answered questions and input answers included in one answer data between answer data. That is, the data sorting unit 115 may perform the determination of duplicate data using the similarity of the entire answer data (the difference in the feature amounts of the entire answer data).

[0091] Furthermore, the data sorting unit 115 may determine which feature amount among the feature amount of the answered question, the feature amount of the input answer, and the feature amount of the entire answer data is used for similarity determination according to the question type of the answered question. The question types include, for example, single - selection form, multiple - selection form, free - description form, designated - description form, etc.

[0092] Also, for example, for a plurality of answer data that include common answered questions and different input answers, the accuracy of the integration process can be improved by using only the feature amount of the answered questions for determination. Also, for example, for answer data that include answered questions with the same intended content of the question but different question contexts, keywords used, etc., the accuracy of the integration process can be improved by using the feature amount of the answered questions and the feature amount of the input answers for determination.

[0093] The data sorting unit 115 may input two or more response data into the duplicate determination model and cause the duplicate determination model to determine the response data to be integrated. The duplicate determination model is, for example, a learning model included in the artificial intelligence unit 120 that is learned to be able to take two or more response data as input and output information regarding the duplication of the response data (presence or absence of duplication, or similarity between the response data). The duplicate determination model is, for example, a learning model that is learned using, as teacher data, data of a plurality of response data and corresponding duplicate information (information indicating presence or absence of duplication or similarity between the response data).

[0094] The duplicate determination model may be a generative AI including a large language model. In this case, the data sorting unit 115 inputs, as input, two or more response data, inputs into the duplicate determination model a prompt including an instruction to extract duplicate response data from these response data, and causes the duplicate determination model to output the duplicate data. The data sorting unit 115 may generate a prompt for giving the duplicate determination model an instruction to extract duplicate data from two or more response data and input the prompt into the duplicate determination model. Further, in addition to the instruction to extract / output the duplicate data and two or more response data, the data sorting unit 115 may input into the duplicate determination model a prompt in which, as examples, samples, or teacher data of input and output pairs, for example, samples of combinations of one or more response data and samples of one or more corresponding duplicate data are inserted.

[0095] The data sorting unit 115 may determine duplicate data by comparing the keywords included in the answered questions and / or the keywords included in the input answers. For example, the data sorting unit 115 may determine, as duplicate data, answer data with a common keyword (including similar keywords) commonly included in the answered questions and / or the input answers that is equal to or more than a predetermined number, or answer data with a degree of coincidence of the order of the common keywords that is equal to or more than a certain level. Further, the data sorting unit 115 may calculate a feature amount of a keyword based on the frequency of appearance of the keyword included in the answer data by natural language processing such as TF-IDF, Okapi BM25, etc., and obtain the similarity between the answer data using the cosine similarity or the like using the feature amount, and determine duplicate data.

[0096] The data sorting unit 115 may perform any of the above-described duplicate determinations after standardizing the keywords included in each answer data (answered question or input answer). Standardization of the keyword is performed, for example, by converting a keyword similar to a standard word registered in a keyword list in advance into the standard word.

[0097] The data sorting unit 115 integrates the answered questions and input answers included in a plurality of answer data determined to be duplicates into one answered question and one input answer, respectively. Integration of the plurality of answer data is performed, for example, by natural language processing so that there is no duplication in content in the integrated answered questions and input answers and the content included in the answered questions and input answers before integration is not omitted.

[0098] In the integration process, the data sorting unit 115 may input a plurality of answer data into an integration processing model and cause the integration processing model to output answer data obtained by integrating the duplicate answer data. The integration processing model is a learning model included in the artificial intelligence unit 120 that is learned so as to be able to input a plurality of answer data and output answer data obtained by integrating the duplicate answer data. Thereby, since the integration process is performed in parallel with the determination of duplication, the integration accuracy of the duplicate answer data is improved.

[0099] The integrated processing model is, for example, a learning model that learns a plurality of answer data and the corresponding integrated answer data as teacher data.

[0100] The integrated processing model may be generative AI including a large language model. In this case, the data arrangement unit 115 inputs a prompt including an instruction to output integrated answer data (integrated data) obtained by integrating a plurality of answer data to the integrated processing model, and causes the integrated processing model to output the integrated data. The data arrangement unit 115 may generate a prompt for giving an instruction to the integrated processing model to generate integrated data from a plurality of answer data, and input the prompt to the integrated processing model. Further, in addition to the instruction to generate and output the integrated data and the plurality of answer data, the data arrangement unit 115 may input to the integrated processing model a prompt in which, as examples, samples, or teacher data of input and output pairs, for example, samples of combinations of one or more answer data and samples of one or more integrated data corresponding thereto are inserted.

[0101] When the integrated processing model is a large language model, the data arrangement unit 115 may input to the integrated processing model an instruction to create integrated data that has no duplication in content and does not omit the content included in the answered questions and input answers before integration.

[0102] <Segmentation processing> In the segmentation processing, the data arrangement unit 115 divides answer data in which the answered question can be divided into a plurality of questions or the input answer can be divided into a plurality of answers into a plurality of answer data. Note that the segmentation processing may target only the first answer data, or may target one or more of the first answer data, the second answer data, and the third answer data. Further, when a category is assigned to the answer data, the same category as the original answer data is assigned to the divided answer data.

[0103] Whether the answered question is splittable is determined, for example, by the presence or absence of a phrase indicating a combination of a plurality of matters. Examples of the "phrase indicating a combination" include a phrase representing a conditional branch (e.g., "in the case of", "when"), a phrase representing a logical product (e.g., "and", "and then"), a phrase representing a logical sum (e.g., "or", "or"), and the like.

[0104] When the answered question is splittable into a plurality of questions, the data arrangement unit 115, for example, after splitting one answered question into a plurality of questions, creates answers corresponding to the plurality of split questions based on the input answer. The answer corresponding to the split question may be the input answer itself before splitting, or may be the content extracted from the input answer corresponding to the split question.

[0105] Whether the input answer is splittable is determined, for example, by the presence or absence of a phrase indicating a combination of a plurality of matters, similar to the determination of whether the answered question is splittable. When the input answer is splittable into a plurality of answers, the data arrangement unit 115, for example, after splitting one input answer into a plurality of answers, creates questions corresponding to the plurality of split answers based on the answered question. The question corresponding to the split answer may be the answered question itself before splitting, or may be the content extracted from the answered question corresponding to the split answer, or may be newly generated based on the split answer.

[0106] In the splitting process, the data arrangement unit 115 may input the answer data into the splitting process model and cause the splitting process model to output a plurality of answer data obtained by splitting the answer data. The splitting process model is a learning model included in the artificial intelligence unit 120 that is trained to be able to input the answer data and output a plurality of answer data obtained by splitting the answer data into a combination of a plurality of answered questions and input answers. Thereby, since the splitting process is performed together with the determination of splittability, the splitting accuracy of the answer data is improved.

[0107] The segmentation processing model is, for example, a learning model that learns using answer data and a plurality of segmented answer data corresponding thereto as teacher data.

[0108] The segmentation processing model may be a generative AI including a large language model. In this case, the data arrangement unit 115 inputs a prompt including an instruction to output segmented answer data (segmented data) obtained by segmenting the answer data to the segmentation processing model, and causes the segmentation processing model to output the segmented data. The data arrangement unit 115 may generate a prompt for giving an instruction to the segmentation processing model to generate a plurality of segmented data from the answer data, and input the prompt to the segmentation processing model. Further, in addition to the instruction to generate / output the segmented data and the answer data, the data arrangement unit 115 may input to the segmentation processing model a prompt in which, as examples, samples, or teacher data of input / output pairs, for example, one or more samples of answer data and one or more samples of corresponding segmented data are inserted.

[0109] Thus, when the segmentation processing model is a large language model, in the segmentation processing, the data arrangement unit 115 may input to the segmentation processing model an instruction to segment the answer data into a combination of a plurality of answered questions and input answers. Thereby, it becomes possible to obtain answer data in units of questions that are easy to use for automatic generation of answers based on natural language processing. For example, the data arrangement unit 115 inputs to the segmentation processing model, which is a large language model, a prompt including an instruction such as "Decompose the input answer as much as possible and make it into pairs of simple questions and answers".

[0110] In the splitting process, the data arrangement unit 115 may input a first instruction to the splitting process model, which is a large language model, to split the answered questions in the answer data including clauses representing conditional branching, logical OR, or logical AND, and further split the input answers according to the split answered questions. This can improve the splitting accuracy of both the answered questions and the input answers. Note that the split answers may include those with the same content or those containing duplicates. Also, the first instruction may be inserted into one prompt and input to the splitting process model, or may be split into multiple prompts (for example, a prompt for instructing the splitting of the answered questions and a prompt for instructing the generation of the input answers according to the split questions) and input to the splitting process model.

[0111] In the splitting process, the data arrangement unit 115 may input a second instruction to the splitting process model, which is a large language model, to split the input answers for the answer data composed of multiple options for each option, and further combine the original answered questions with each of the split input answers. This can improve the splitting accuracy of the input answers. Note that the second instruction may be inserted into one prompt and input to the splitting process model, or may be split into multiple prompts (for example, a prompt for instructing the splitting of the input answers and a prompt for instructing the combination of the answered questions with the split answers) and input to the splitting process model.

[0112] In the splitting process, the data arrangement unit 115 may input a third instruction to the splitting process model, which is a large language model, for the response data including the answered questions containing a plurality of question items and the inputted responses composed of a batch of responses to the plurality of question items. The third instruction is to split the answered questions for each question item and further combine the original inputted responses with each of the split answered questions. This can improve the splitting accuracy of the answered questions. Note that the third instruction may be inserted into one prompt and input to the splitting process model, or may be split into a plurality of prompts (for example, a prompt for instructing up to the splitting of the answered questions and a prompt for instructing the combination of the inputted responses to the split questions) and input to the splitting process model.

[0113] Furthermore, the data arrangement unit 115 may split the inputted responses and generate questions for the split responses based on the sentences or words included in the inputted responses. For example, for the response data including the answered question "Is the SLA defined?" and the inputted response "The SLA is not defined, but the service operation rate has a target value of xx% or more, and the actual operation rate in the past year was xx%", the data arrangement unit 115 splits the inputted response into a first response "No, the SLA is not defined", a second response "The service operation rate has a target value of xx% or more", and a third response "The actual operation rate in the past year was xx%". Here, the second response and the third response are unanswered question items that are not directly asked in the inputted response. Next, the data arrangement unit 115 generates a first question "Is the SLA defined?" for the first response, a second question "What is the target value of the service operation rate?" for the second response, and a third question "What was the actual operation rate in the past year?" for the third response, and creates response data including the first question and the first response, response data including the second question and the second response, and response data including the third question and the third response as split data. Here, the second question and the third question are contents not included in the answered question before splitting, and are questions generated based on the sentences or words included in the inputted response before splitting.

[0114] Therefore, in the splitting process, the data arrangement unit 115 splits the input answers including unanswered items not asked in the answered questions in the answer data, and further generates a fourth instruction for generating the answered questions corresponding to the answers including unanswered items among the split input answers based on the original input answers, and inputs the fourth instruction into the splitting process model which is a large language model. Thereby, answer data in which one question item and one answer are combined in a one-to-one manner can be obtained.

[0115] The data arrangement unit 115 may determine an instruction to be input into the splitting process model among the first instruction, the second instruction, the third instruction, the fourth instruction, and other instructions by natural language processing on the answer data (answered questions or input answers), and insert the determined instruction into the prompt for the splitting process model.

[0116] <Relationship between integration process and splitting process> The data arrangement unit 115 may perform only the integration process, only the splitting process, or both the integration process and the splitting process on the plurality of answer data acquired by the data acquisition unit 114. These processes are executed, for example, according to an instruction from the person to be evaluated. Also, these processes may be automatically performed without an instruction from the person to be evaluated at the time of acquiring the answer data.

[0117] The data arrangement unit 115 may perform the integration process and the splitting process in parallel on the answer data acquired by the data acquisition unit 114, or perform the integration process on the answer data after performing the splitting process. Thereby, for example, it is possible to suppress the repeated application of the integration process and the splitting process to the same answer data, such as avoiding performing the splitting process on the answer data created by the integration process.

[0118] <Editing of answer data> The data arrangement unit 115 may receive editing of the response data obtained by the integration process or the division process from the subject terminal 20. As a result, the evaluation subject can adjust (customize) the response data after the integration process or the division process, so that it is possible to improve the generation accuracy of the response by the response generation unit 118 described later and reduce the labor of correcting the generated response.

[0119] "Editing of response data" includes, for example, correction of sentences included in answered questions, correction of sentences included in input responses, deletion of newly generated response data, etc. Also, for a plurality of response data obtained by the division process, deletion may be accepted for each individual response data, or batch deletion may be accepted for a plurality of response data obtained from one response data.

[0120] The data arrangement unit 115 may delete the original response data in response to the registration of the response data obtained by the integration process or the division process in the generation database by the data registration unit 116 described later. Also, the data arrangement unit 115 may delete the original response data upon receiving an instruction from the evaluation subject. Further, the data arrangement unit 115 may display a list of the deleted response data or the response data to be deleted on the subject terminal 20.

[0121] FIG. 7 is a diagram showing an example of a response data arrangement screen OD displayed on the subject terminal 20. The response data arrangement screen OD is displayed, for example, when an input operation is performed on the knowledge arrangement button B24 of the response data management screen MD in FIG. 6. The response data arrangement screen OD includes a switching tab TB and a work area WA.

[0122] The switching tab TB is an object that receives selection of information (work content) to be displayed in the work area WA. In the example of FIG. 7, the switching tab TB includes a "Integrate Knowledge" tab for integrating response data and a "Decompose Knowledge" tab for dividing response data. Note that in FIG. 7, the "Integrate Knowledge" tab of the switching tab TB is shown as being selected.

[0123] When the integration of response data is selected by the switching tab TB as shown in FIG. 7, a duplicate data display column OF is displayed in the work area WA. In the duplicate data display column OF, the questions and answers of the response data determined to be duplicates, and the integrated questions UQ and integrated answers UA obtained by integrating them are displayed in a side-by-side comparison. Also, in the work area WA, a duplicate data display column OF is displayed for each set of duplicate response data. In the duplicate data display column OF, an integrated data addition button B31 and an integrated data deletion button B32 are displayed.

[0124] In the duplicate data display column OF, editing of the integrated question UQ and integrated answer UA is accepted from the subject terminal 20. When an input operation is performed on the integrated data addition button B31, response data composed of a combination of the integrated question UQ and integrated answer UA is registered in the generation database, and the response data (original response data for integration) determined to be duplicates is deleted from the generation database. Also, when an input operation is performed on the integrated data deletion button B32, the integrated question UQ and integrated answer UA are discarded, and the response data determined to be duplicates is maintained in the generation database.

[0125] FIG. 8 is a diagram showing another example of the response data sorting screen OD displayed on the subject terminal 20. The response data sorting screen OD in FIG. 8 shows a state in which the "Decompose Knowledge" tab of the switching tab TB is selected in the response data sorting screen OD of FIG. 7.

[0126] When the splitting of response data is selected by the switching tab TB as shown in FIG. 8, a splittable data display column DF is displayed in the work area WA. In the splittable data display column DF, the questions and answers of the response data determined to be splittable, and the split questions DQ and split answers DA obtained by splitting them are displayed in a side-by-side comparison. Also, in the work area WA, a splittable data display column DF is displayed for each splittable response data. In the splittable data display column DF, a split data addition button B41, a split data deletion button B42, and an original data deletion button B43 are displayed.

[0127] In the dividable data display column DF, editing of the division question DQ and the division answer DA is accepted from the subject terminal 20. The dividable data addition button B41 and the dividable data deletion button B42 are displayed for each set of a division question DQ and a division answer DA. When an input operation is performed on the dividable data addition button B41, answer data composed of the corresponding combination of the division question DQ and the division answer DA is registered in the generation database. Also, when an input operation is performed on the dividable data deletion button B42, the corresponding division question DQ and division answer DA are discarded and deleted from the dividable data display column DF.

[0128] The original data deletion button B43 is displayed for each dividable data. When an input operation is performed on the original data deletion button B43, the dividable data (the answer data before division) is deleted from the generation database.

[0129] Note that the dividable data addition button B41 and the dividable data deletion button B42 may be displayed for each dividable data. In this case, when an input operation is performed on the dividable data addition button B41, all sets of the division questions DQ and the division answers DA displayed in the dividable data display column DF are registered in the generation database as answer data. Also, in this case, when an input operation is performed on the dividable data deletion button B42, all the division questions DQ and the division answers DA displayed in the dividable data display column DF are deleted.

[0130] <Data registration unit 116> The data registration unit 116 is configured to register the answer data obtained by the integration process or the division process in the generation database. As described above, the data registration unit 116 may register the answer data edited by the subject terminal 20 in the generation database.

[0131] Note that the answer data obtained by the integration process or the division process may be registered as text data or may be registered as vector data.

[0132] <Designated reception unit 117> The designated reception unit 117 is configured to receive from the subject terminal 20 a designation of a security question. In addition to the designation of a security question, the designated reception unit 117 may receive from the subject terminal 20 a designation of a category. Questions received by the designated reception unit 117 are, for example, new questions unanswered by the evaluation subject, or questions that have been answered in another category (e.g., service name) but are unanswered in the new category (designated category).

[0133] The designated reception unit 117 receives, for example, a designation of a question by input of the question from the subject terminal 20. Further, the designated reception unit 117 may receive a designation of a question by calling an input form for registering evaluation information in the subject terminal 20 (selection of a service, plan, etc. for creating evaluation information). In this case, a plurality of questions included in the input form are designated by calling the input form.

[0134] The designated reception unit 117 may present a list of categories included in the answer data registered in the generation database and receive from the subject terminal 20 a designation of a category from the list. This can streamline the designation of a category by the evaluation subject. Note that the order of the designation of a question and the designation of a category is not limited.

[0135] When receiving a designation of a question by calling an input form, if the input form is called after input of a category, the designated reception unit 117 receives a designation of a category by input (selection) of the category (service name, plan name, etc.) at the time of calling the input form (when creating evaluation information).

[0136] FIG. 9 is a diagram showing an example of a question input screen QD displayed on the subject terminal 20. The question input screen QD includes a creation button B51, a service selection button B52, and a question specification field QI. The creation button B51 is an object that instructs the generation of an answer to the question input in the question specification field QI according to the service name (category) selected by the service selection button B52. The creation button B51 does not accept input when the service name is not selected or when no questions have been input.

[0137] When an input operation is performed on the service selection button B52, a UI for selecting a service is displayed on the subject terminal 20. FIG. 10 is a diagram showing an example of a service selection screen SD displayed on the subject terminal 20. The service selection screen SD includes a selection object SO, a cancel button B61, and a decision button B62. The selection object SO is assigned for each option of the service name. In the example of FIG. 10, radio buttons for single selection are arranged as the selection object SO. When an input operation is performed on the cancel button B61, the selection of the service name is canceled. When an input operation is performed on the decision button B62, the selected service name is accepted as the category of the answer data to be referred to when generating questions, and the service name is displayed on the question input screen QD of FIG. 9.

[0138] The question specification field QI shown in FIG. 9 accepts input of questions from the subject terminal 20. The evaluation subject can simultaneously input multiple questions for the same category (such as service name) in the question specification field QI.

[0139] FIG. 11 is a diagram showing an example of a question input screen QD in a state where a service name has been selected and questions have been input. On the question input screen QD of FIG. 11, the selected service name is displayed on the service selection button B52. As shown in FIG. 11, when an input operation is performed on the creation button B51 in a state where a service name has been selected and at least one question has been input, the designation reception unit 117 receives the designation of a question regarding security and the designation of a category, and subsequently, the answer generation unit 118 generates an answer. Also, in the question designation field QI, for example, questions may be input by copy and paste from a data file (for example, a CSV file such as an Excel file or a Google spreadsheet) containing questions prepared by the evaluation target person on the target terminal 20.

[0140] The designation reception unit 117 may further receive from the target terminal 20 the designation of the answer format of the question. The answer format of the question is the format of the answer created by the answer generation unit 118, and includes, for example, an alternative selection format in which an answer is selected from a predetermined set of options (for example, a combination of "yes" and "no"), a free description format, a designated description format in which keywords or sentences are created along a predetermined sentence pattern, and the like.

[0141] <Answer generation unit 118> The answer generation unit 118 is configured to extract, as a reference answer, the input answer for an answered question similar to the designated question (the designated question received by the designation reception unit 117) from among the input answers included in the answer data registered in the generation database, and generate an answer to the designated question based on the reference answer and the reference information for answer generation. Thereby, since the answer is generated based on the answer data obtained by integration processing or division processing, the generation accuracy of the answer is improved.

[0142] The answer generation unit 118 may extract, as a reference answer, the input answer for an answered question corresponding to the designated category (the designated category) received by the designation reception unit 117 and similar to the designated question from among the input answers included in the answer data registered in the generation database.

[0143] The "already-entered answer corresponding to the specified category" is an already-entered answer to which the same category as the specified category is assigned. The answer generation unit 118 searches for answer data including answered questions similar to the specified question from the generation database in a state where the answer data to be searched is narrowed down using the specified category.

[0144] The answer generation unit 118 may extract, as a reference answer, an already-entered answer to an answered question similar to the specified question regardless of the specified category, provided that the category is not assigned or an unclassified category indicating that it is not classified into a specific category is assigned. This can reduce the work for the evaluation target person to assign individual categories to answers common to multiple categories (such as service names) and answers regarding the evaluation target person himself / herself.

[0145] The answer generation unit 118 may extract a reference answer by referring to only arbitrary answer data among the first answer data, the second answer data, and the third answer data. Also, the answer generation unit 118 may extract a reference answer by referring to the first answer data and the second answer data. This can improve the accuracy of the answer because the answer generation unit 118 can generate an answer based on both the answer input in the input form provided by the answer registration unit 112 and other answers (for example, answers input outside the input form or answers prepared by the evaluation target person himself / herself). When the third answer data is accumulated, the answer generation unit 118 may also refer to the third answer data in addition to the first answer data and the second answer data to extract a reference answer.

[0146] The answer generation unit 118 may determine the similarity between the answered question and the specified question by comparing the vector data obtained by vectorizing the content of the answered question included in the answer data with the vector data obtained by vectorizing the content of the specified question. This can quickly and accurately extract the reference answer. Note that the "answered question similar to the specified question" also includes an answered question identical or substantially identical to the specified question.

[0147] The answer generation unit 118 may input the answered question and the designated question into a similarity determination model, cause the similarity determination model to output the similarity between the two, and determine the similarity between the answered question and the designated question based on the similarity. The similarity determination model is, for example, a learning model included in the artificial intelligence unit 120 that is learned to be able to input two questions and output the similarity between the two questions.

[0148] The similarity determination model is a learning model that is learned using, as teacher data, a combination of a set of learning questions and the similarity corresponding to the set of questions. Also, the similarity determination model may be a generative AI including a large language model. In this case, the answer generation unit 118 inputs the answered question and the designated question into the similarity determination model, and inputs a prompt including an instruction to output the similarity between the answered question and the designated question, causing the similarity determination model to output the similarity. The answer generation unit 118 may generate a prompt for giving an instruction to output the similarity between the answered question and the designated question to the similarity determination model, and input the prompt into the similarity determination model. Further, in addition to the similarity output instruction, the answered question, and the designated question, the answer generation unit 118 may input into the similarity determination model a prompt in which, as examples, samples, or teacher data of the input and output bare, for example, one or more samples of answered questions and designated questions and one or more corresponding samples of similarities are inserted.

[0149] The similarity determination model may be a learning model configured to input the designated question, refer to answer data (generation database), and output a reference answer. In this case, the similarity determination model extracts, from the answered questions included in the generation database, those with a high similarity to the designated question (for example, the cosine similarity of vector data is equal to or higher than a threshold value), and outputs the input answer for the answered question as the reference answer.

[0150] The answer generation unit 118 may extract, as a reference answer, the input answer to an answered question similar to the specified question based on the specified question category (designated question category). For example, the answer generation unit 118 extracts a reference answer from the input answers to the answered questions to which a question category identical or similar to the specified question category is assigned. Here, the specified question category may be specified based on an input by a user such as the person to be evaluated, or may be the one obtained by inputting the specified question to the above-described question category assignment model to cause the question category assignment model to output the question category corresponding to the specified question. Thereby, the accuracy of matching between the specified question and the answered question or the reference answer can be improved. Note that the extraction of the reference answer based on the question category may be performed in combination with the extraction of the reference answer based on the specified category received by the specified reception unit 117.

[0151] When the generation database includes a plurality of answered questions similar to the specified question, the answer generation unit 118 may extract, as reference answers, the input answers to each of the plurality of answered questions. Further, the answer generation unit 118 may extract, as a reference answer, only the input answer to the answered question with the highest similarity among the plurality of answered questions similar to the specified question.

[0152] The answer generation unit 118 generates, as an answer to the specified question, an arrangement of the extracted reference answer in accordance with the content of the specified question using the reference information for answer generation. The arrangement of the reference answer includes, for example, substitution of words included in the reference answer with synonyms, hypernyms, hyponyms, etc., rearrangement of the context (reordering of sentences), deletion of unnecessary information, addition of missing information, and the like.

[0153] The reference information for answer generation includes the correlation between the reference answer and the answer to the specified question. The reference information for answer generation is stored, for example, in the storage unit 12. The reference information for answer generation is an estimator constructed to be able to output an answer to the specified question with the reference answer as the input. The reference information for answer generation may include, for example, a table, a function, a simple algorithm, etc. that shows the correlation between the feature amount (vector data) extracted from the reference answer and the feature amount (vector data) extracted from the specified question. The correlation included in the reference information for answer generation can be constructed, for example, by statistically analyzing the data recording the questions to the answers generated based on the reference answer.

[0154] The reference information for answer generation may include an answer generation model included in the artificial intelligence unit 120 that is learned to be able to take the reference answer as the input and output an answer to the specified question. In this case, the answer generation unit 118 inputs the reference answer to the answer generation model and causes the answer generation model to output an answer to the specified question. Thereby, it becomes possible to generate answer generation cases for a large number of questions or generate answers based on natural language processing.

[0155] The answer generation model may be a learning model that learns, as teacher data, a combination of a learning reference answer and question set and a combination of the reference answer and the answer corresponding to the question. In the answer generation model, the parameters calculated, tuned, etc. by learning constitute the correlation of the reference information for answer generation.

[0156] Further, the answer generation model may be generative AI including a large language model. In this case, the answer generation unit 118 inputs the reference answer and the specified question, inputs an instruction to output an answer to the specified question to the answer generation model, and causes the answer generation model to output the answer. The answer generation unit 118 may generate a prompt for giving an instruction to the answer generation model to generate an answer to the specified question based on the reference answer, and input the prompt to the answer generation model. Further, in addition to the instruction to generate and output the answer, the reference answer, and the specified question, the answer generation unit 118 may input, as examples, samples, or teacher data of the input and output bare, a prompt in which, for example, samples of one or more reference answers and specified questions and samples of one or more corresponding answers are inserted, to the answer generation model. The answer generation model generates an answer referring to the reference answer according to the input prompt.

[0157] Further, for a specified question for which a reference answer has not been extracted (that is, a specified question for which a similar answered answer does not exist in the generation database), the answer generation unit 118 outputs information indicating that an answer could not be generated.

[0158] When the specification reception unit 117 receives a specification of the answer format, the answer generation unit 118 may generate an answer to the specified question according to the answer format. For example, when an alternative selection format of "yes" and "no" is specified, the answer generation unit 118 generates either "yes" or "no" as the answer. Further, for example, when a free description format is specified, the answer generation unit 118 generates a freely described sentence as the answer. The answer generation unit 118 inserts, for example, an instruction to generate an answer in the specified answer format into the prompt to the answer generation model.

[0159] The answer generation unit 118 may accept editing of the answer to the generated question. Further, when the answer to the specified question cannot be generated, the answer generation unit 118 may present to the target user terminal 20 that the answer to the specified question has not been generated, and may also accept input of the answer to the specified question. Also, the answer generation unit 118 may present to the target user terminal 20 the reason why the answer could not be generated (for example, insufficient information, etc.). For example, the answer generation unit 118 inserts an instruction to output the reason when the answer cannot be generated into the prompt to the answer generation model.

[0160] The answer generation unit 118 may display, on the target user terminal 20, in association with the generated answer, a link for displaying the content of the answer data (specifically, the input answer) on which the generated answer is based, the reason for referring to the answer data, and the like. For example, the answer generation unit 118 may display, on the target user terminal 20, the reference answer and the input answer corresponding to the reference answer (or links thereto) as the basis for answer generation. For example, the answer generation unit 118 inserts an instruction to output, in addition to the generated answer, the reference answer and the input answer corresponding to the reference answer (or links thereto) into the prompt to the answer generation model.

[0161] The answer generation unit 118 determines whether the answer generated by the answer generation model contains information that should not be made public. If it contains information that should not be made public, the fact may be displayed on the target terminal 20. "Information that should not be made public" includes, for example, personal information, non-public information, links to non-public data such as in-house data, etc. Examples of personal information include the names, phone numbers, etc. of employees of the person being evaluated. Examples of non-public information include information for internal use of the person being evaluated described in items (fields) such as "Precautions" and "Supplementary Notes". Examples of links to non-public data include links (links that enable reference to files, posts, etc.) issued in storage services, messaging services, etc. outside the information processing system 1 used by the person being evaluated. For example, the answer generation unit 118 inserts an instruction to generate an answer so that the prompt to the answer generation model does not contain information that should not be made public, such as personal information, non-public information, and links to non-public data. Also, the answer generation unit 118 determines whether the input answer used for answer generation contains "information that should not be made public", and may not use the input answer containing "information that should not be made public" for answer generation (for example, not input it to the answer generation model).

[0162] The answer generation unit 118 may generate an answer to the question included in the input form presented by the answer registration unit 112. As a result, since the answers necessary for the evaluation information can be automatically generated from the answer history held by the person being evaluated, the effort required for the person being evaluated to create the evaluation information can be reduced. In particular, the convenience is improved for a person being evaluated who uses the information processing system 1 (for example, a security evaluation system) for the first time.

[0163] When generating an answer to the input form for registering evaluation information, as described above, at the time of calling the input form or after calling the input form, a category for extracting a reference answer is specified by the person being evaluated inputting a category (service, plan, etc.), and an answer to the question included in the input form is automatically generated. Therefore, there is no need for the person being evaluated to input questions individually.

[0164] When the data acquisition unit 114 receives the input of the answered questions and the input answers from the person to be evaluated, the answer generation unit 118 may generate in advance the answers to the questions that the person to be evaluated has not answered for the input form of the category in which the answer data is stored at a predetermined timing (for example, the timing when the answer data is generated based on the information input by the data acquisition unit 114). That is, the answer generation unit 118 may create part or all of the evaluation information in the background. Thereby, since the labor for the person to be evaluated to create the evaluation information from the input form is greatly reduced, the use of the input form by the person to be evaluated can be promoted.

[0165] FIG. 12 is a diagram showing an example of an answer display screen AD displayed on the subject terminal 20. The answer display screen AD includes a designated question display column QC2, a generated answer display column AC2, a remarks column RC, and an answer output button B71. In the designated question display column QC2, the designated questions input on the question input screen QD of FIG. 11 are displayed.

[0166] In the generated answer display column AC2, the answers generated by the answer generation unit 118 are displayed in association with the designated questions (in the same row). The answer generation unit 118 may accept the editing of the answers by the person to be evaluated in the generated answer display column AC2. Also, for the designated questions for which answers could not be generated, an error message indicating that fact is displayed. In FIG. 12, an error message EM1 indicating that the answer could not be generated along with the basis, and a stereotyped error message EM2 indicating that the answer could not be generated for some reason are illustrated. Also, for the designated questions for which the error message EM2 is displayed, a regeneration instruction object RO for accepting an instruction to retry generating the answer is displayed in the generated answer display column AC2.

[0167] In the remarks column RC, objects that accept actions and the like are displayed. Such objects include, for example, a data addition acceptance object AO, a data display acceptance object DO, and the like. The data addition acceptance object AO accepts an input for displaying a screen (described later) for adding the generated response data to the generation database. The data display acceptance object DO accepts an input for displaying a screen (described later) for displaying the basis of the generated response data.

[0168] When the answer output button B71 is input-operated, the answer generation unit 118 outputs the combination data (matrix) of the questions and answers displayed in the designated question display column QC2 and the generated answer display column AC2 as CSV data. Also, the person being evaluated can individually copy the answer from the generated answer display column AC2 and paste it into an arbitrary location (for example, a file on the subject terminal 20, an input field of another service, etc.).

[0169] FIG. 13 is a diagram showing an example of an answer data registration screen RD displayed on the subject terminal 20. The answer data registration screen RD is displayed, for example, when an input operation is performed on the data addition acceptance object AO in FIG. 12. The answer data registration screen RD displays a basis message BM, a question display column QF, an answer display column AF, a memo display column MF, a category display column CF, an add button B81, and a cancel button B82.

[0170] The basis message BM is text that represents the reason why an answer could not be generated. In the specified question for which an answer has been generated, the basis message BM is not displayed. The content of the specified question is displayed in the question display field QF. Also, the question display field QF accepts editing of the specified question. The content of the generated answer is displayed in the answer display field AF. When an answer cannot be generated as shown in FIG. 13, the answer display field AF is left blank. Also, the answer display field AF accepts editing or input of an answer. The memo display field MF accepts input of comments by the person being evaluated. The selected category (service name) is displayed in the category display field CF. Also, the category display field CF accepts editing (selection) of the category.

[0171] When an input operation is performed on the add button B81, the content input in the question display field QF, the answer display field AF, the memo display field MF, and the category display field CF is registered as answer data in the generation database. When an input operation is performed on the cancel button B82, the information input on the answer data registration screen RD is discarded and the answer data registration screen RD is closed.

[0172] FIG. 14 is a diagram showing an example of the basis display screen BD displayed on the subject terminal 20. The basis display screen BD is displayed, for example, by performing an input operation on the data display reception object DO in FIG. 12. The basis display screen BD includes a basis display area BA, a reference data display area RA, and a close button B91.

[0173] In the basis display area BA, the combination of the selected specified question and the generated answer, and the basis for answer generation (reason for referring to the answer data) are displayed. In the reference data display area RA, the answer data (answered questions and input answers) referred to at the time of answer generation is displayed. When an input operation is performed on the close button B91, the basis display screen BD is closed.

[0174] <Report output unit 119> The report output unit 119 is configured to output a report or a statement created based on the evaluation received by the evaluation reception unit 113 to the requester terminal 40. The report or the statement includes at least all or part of the evaluation information input by the person to be evaluated and the evaluation results (e.g., evaluation values, comments, etc.) by the evaluator.

[0175] <Artificial intelligence unit 120> The artificial intelligence unit 120 is configured to receive inputs from each functional unit and return the instructed output. Note that the artificial intelligence used by the server device 10 in each functional unit may be common or may be individually prepared for each functional unit.

[0176] The artificial intelligence unit 120 is an AI (Artificial Intelligence) equipped with a learning model such as a language model of a Transformer (including GPT (Generative Pretrained Transformer, including GPT-1, GPT-2, GPT-3, and GPT-4), BERT (Bidirectional Encoder Representations from Transformers), BART (Bidirectional and Auto-regressive Transformer), etc.) or a Recurrent Neural Network (RNN), and may include a generative AI including a large language model. The large language model is a type of generative AI and includes models provided by services such as GPT of OpenAI, Gemini of Google, and Azure AI Studio of Microsoft. In addition, the artificial intelligence unit 120 can include any machine learning model, deep learning model, artificial intelligence model, etc.

[0177] A language model is an example of a learning model by a machine learning algorithm. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, decision trees, support vector machines, deep learning (deep neural networks) using neural networks, and the like. The artificial intelligence unit 120 can appropriately apply the above algorithms.

[0178] The artificial intelligence unit 120 may have a trained model constructed by a learning method such as supervised learning, unsupervised learning, or semi-supervised learning. In supervised learning, machine learning is performed using teacher data (learning data). The teacher data is composed of a pair of input data for learning and output data (correct data). Also, the language model may be not only trained for a specific task but also a general-purpose model that can be generally used for a wide range of tasks.

[0179] The artificial intelligence unit 120 may be a general-purpose natural language processing learning model such as a large language model (LLM) that has learned a vast amount of data as artificial intelligence. The LLM is a learning model that has previously learned a large amount of data (for example, (i) web content on the Internet or (ii) data stored in a predetermined database) composed of text data and the like, and can execute various language processing tasks by being given a task, and can perform a wide range of natural language processing tasks such as grasping the pattern and context of a sentence, answering questions, and generating sentences according to the given prompt. Such a general-purpose learning model includes a language model that can handle various tasks without fine-tuning by one-shot learning, few-shot learning, etc. Also, the general-purpose learning model may be configured to be able to handle various tasks by zero-shot learning. The artificial intelligence used in each functional unit of the control unit 11 may be a separate learning model or a common general-purpose learning model.

[0180] The learning models included in the artificial intelligence unit 120 (such as integrated processing models and segmentation processing models, which are learning models used in each functional unit) can perform additional learning as transfer learning or fine-tuning. For example, each time new data registration or the like occurs, the artificial intelligence unit 120 may use this as new teacher data to perform additional learning and be fine-tuned. As a result, the accuracy of the information output from the learning model is improved.

[0181] The learning model included in the artificial intelligence unit 120 may be a learning model (distillation model) obtained by knowledge distillation using the original learning model. In knowledge distillation, a pre-trained model such as a large language model is used as the teacher model, and the parameters of the student model (distillation model) are adjusted so that the loss (Soft Target Loss) of the output of the student model with respect to the output (Soft Target) of the teacher model is reduced, thereby learning the student model, and the student model becomes the distillation model. Also, the learning of the student model may be performed so that the loss (Hard Target Loss) of the output of the student model with respect to the correct label (Hard Target) of the teacher data (the combination of the input data and output data of the learning model) is reduced. The distillation model has a performance similar to that of the original learning model (teacher model), but has a smaller number of parameters and a smaller processing load. Therefore, by using the distillation model, the cost of the information processing system 1 can be reduced.

[0182] For example, the learning model used in each functional unit may be a distillation model learned using the combination of input data and output data in a large language model as teacher data. Also, when the information processing system 1 is introduced, a large language model is used as the learning model used in each functional unit, and when teacher data by the large language model is accumulated, the distillation model obtained by knowledge distillation using the teacher data may be used as the learning model used in each functional unit.

[0183] <Display unit> The display unit 211 of the subject terminal 20, the display unit 311 of the evaluator terminal 30, and the display unit 411 of the requester terminal 40 each display the screen indicated by the screen data transmitted from the server device 10.

[0184] <Operation acquisition unit> The operation acquisition unit 212 of the subject terminal 20 receives operations by the evaluation subject who uses the subject terminal 20. The operation acquisition unit 312 of the evaluator terminal 30 receives operations by the evaluator who uses the evaluator terminal 30. The operation acquisition unit 412 of the requester terminal 40 receives operations by the evaluation requester who uses the requester terminal 40.

[0185] 3. Information processing method In this section, the information processing method of the server device 10 will be described. This information processing method is executed by a computer for each part of the server device 10 as each step.

[0186] This information processing includes a data acquisition step, a data collation step, a data registration step, a designation reception step, and an answer generation step. In the data acquisition step, answer data obtained by combining answered questions regarding the security of the evaluation subject and the input answers of the evaluation subject to the answered questions is acquired. In the data collation step, at least one of an integration process of integrating two or more duplicate answer data into one answer data and a division process of dividing answer data in which the answered questions can be divided into a plurality of questions or the input answers can be divided into a plurality of answers is performed. In the data registration step, the answer data obtained by the integration process or the division process is registered in the database. In the designation reception step, a designation of a question regarding security is received. In the answer generation step, among the input answers included in the answer data registered in the database, the input answer to the answered question similar to the designated question is extracted as a reference answer, and an answer to the question is generated based on the reference answer and the reference information for answer generation.

[0187] FIG. 15 is an activity diagram showing an example of the flow of information processing (answer generation processing) executed by the information processing system 1. Hereinafter, the information processing will be described along each activity of this activity diagram.

[0188] The answer generation processing starts from the input of answer data by the person to be evaluated. The person to be evaluated inputs the answer data by inputting the answered questions and the input answers on the subject terminal 20 or by selecting the registered evaluation information (activity A101). The server device 10 acquires the answer data input from the subject terminal 20 (activity A102). Subsequently, the server device 10 performs at least one of an integration process and a division process on the acquired answer data (activity A103). Further, the server device 10 adds the answer data obtained by the integration process or the division process to the generation database (activity A104). Note that activities A101 to A104 are appropriately performed at an arbitrary timing.

[0189] In a state where the answer data is generated, the person to be evaluated designates a question for generating an answer on the subject terminal 20 (activity A201). The designation of the question may be performed by directly inputting the question on the subject terminal 20 or by calling up an input form for creating evaluation information. The server device 10 extracts a reference answer from the generation database based on the designated question (activity A202). After extracting the reference answer, the server device 10 generates an answer to the designated question with reference to the reference answer (activity A203). Subsequently, the server device 10 outputs the generated answer to the subject terminal 20 (activity A204). As a result, the answer generated by the server device 10 is displayed on the subject terminal 20 (activity A205).

[0190] 4. Operation Summarizing the operation of this embodiment, it is as follows. That is, since the answer data, which is a combination of past questions and answers, is organized, by referring to such answer data, the person to be evaluated can efficiently answer questions regarding security.

[0191] As described above, the embodiments of the present invention have been explained, but the present invention is not limited to this, and can be appropriately changed without departing from the technical idea of the invention.

[0192] 5. Others In the above embodiment, the server device 10 performs various storage and controls. However, instead of the server device 10, a plurality of external devices may be used. That is, various information and programs may be distributed and stored in a plurality of external devices using blockchain technology or the like. In particular, the artificial intelligence unit 120 may be an external configuration of the server device 10. In that case, the artificial intelligence unit 120, which is an external configuration, is provided by, for example, an artificial intelligence service server, receives an input from each functional unit of the server device 10, receives a request to execute an artificial intelligence service, and is configured to return an output instructed as a processing result to the server device 10. The artificial intelligence service server may be a server that provides a service using a language model as a learning model, or may be a server that executes a language processing task using a language model. The artificial intelligence service server may be constructed by an LLM. The artificial intelligence service server receives an input of a prompt by text, image, voice, etc., and generates and responds with an answer to the prompt.

[0193] The control unit 11 does not necessarily have to include the answer generation unit 118. That is, the information processing system 1 does not necessarily have to generate an answer based on the answer data obtained by integrated processing or divided processing.

[0194] Aspects of this embodiment are not limited to the information processing system 1, and may be an information processing method or a program. The information processing method includes each step executed by the information processing system 1. The program causes a computer to execute each step of the information processing system 1.

[0195] It may also be provided in each of the aspects described below.

[0196] (1) An information processing system including at least one processor, the processor being configured to execute the following steps by reading a program. In the data acquisition step, acquired is response data obtained by combining answered questions regarding the security of the person to be evaluated and the inputted responses of the person to be evaluated to the answered questions. In the data arrangement step, at least one of an integration process of integrating two or more overlapping pieces of the response data into one piece of the response data and a division process of dividing the response data in which the answered questions are divisible into a plurality of questions or the inputted responses are divisible into a plurality of responses into a plurality of pieces of the response data is performed.

[0197] (2) In the information processing system according to (1) above, the processor is further configured to execute the following steps. In the data registration step, the response data obtained by the integration process or the division process is registered in a database. In the designation reception step, a designation of a question regarding the security is received. In the response generation step, among the inputted responses included in the response data registered in the database, the inputted response to the answered question similar to the designated question is extracted as a reference response, and based on the reference response and response generation reference information, a response to the question is generated. Here, the response generation reference information includes a correlation between the reference response and the response to the question.

[0198] (3) In the information processing system according to (2) above, the reference information for answer generation includes an answer generation model learned to be able to take the reference answer as input and output an answer to the question. In the answer generation step, the reference answer is input to the answer generation model, and the answer generation model is made to output an answer to the question. An information processing system.

[0199] (4) In the information processing system according to (2) or (3) above, in the data arrangement step, editing of the answer data obtained by the integration process or the division process is accepted, and in the data registration step, the edited answer data is registered in the database. An information processing system.

[0200] (5) In the information processing system according to any one of (1) to (4) above, in the integration process, the answer data to be integrated is determined based on the similarity based on the comparison of the feature amounts of the answered questions included in the answer data, the similarity based on the comparison of the input answers included in the answer data, or a combination of these similarities. An information processing system.

[0201] (6) In the information processing system according to any one of (1) to (5) above, in the integration process, a plurality of the answer data are input to an integration processing model, and the integration processing model outputs the answer data obtained by integrating the overlapping answer data. Here, the integration processing model is a learning model learned to be able to take a plurality of the answer data as input and output the answer data obtained by integrating the overlapping answer data. An information processing system.

[0202] (7) In the information processing system according to any one of (1) to (6) above, in the splitting process, the response data is input into a splitting process model, and the splitting process model outputs a plurality of the response data obtained by splitting the response data. Here, the splitting process model is a learning model that is trained to be able to input the response data and output a plurality of the response data obtained by splitting the response data into combinations of a plurality of the answered questions and the input response. Information processing system.

[0203] (8) In the information processing system according to (7) above, the splitting process model is a large language model, and in the splitting process, an instruction to split the response data into combinations of a plurality of the answered questions and the input response is input into the splitting process model. Information processing system.

[0204] (9) In the information processing system according to (8) above, in the splitting process, for the response data including a clause representing conditional branching, logical sum, or logical product in the answered question, the answered question is split based on the clause, and an instruction to split the input response according to the further split answered question is input into the splitting process model. Information processing system.

[0205] (10) In the information processing system according to (8) or (9) above, in the splitting process, for the response data in which the input response is composed of a plurality of options, the input response is split for each option, and an instruction to combine the original answered question with each of the further split input responses is input into the splitting process model. Information processing system.

[0206] (11) In the information processing system according to any one of (8) to (10) above, in the splitting process, for the response data including the answered questions including a plurality of question items and the input answers composed of a batch of answers to the plurality of question items, the answered questions are split for each question item, and an instruction to combine the original input answers with each of the further split answered questions is input to the splitting processing model. Information processing system.

[0207] (12) In the information processing system according to any one of (1) to (11) above, in the data arrangement step, the integration process and the splitting process are performed in parallel on the obtained response data, or the integration process is performed on the response data after the splitting process. Information processing system.

[0208] (13) An information processing method, comprising each step executed by the information processing system according to any one of (1) to (12) above.

[0209] (14) A program for causing a computer to execute each step of the information processing system according to any one of (1) to (12) above. Of course, this is not all-inclusive.

[0210] Finally, although various embodiments according to the present disclosure have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and its equivalent scope.

Description of Reference Numerals

[0211] 1: Information processing system 2: Communication line 10: Server device 11: Control Unit 12: Memory Unit 13: Communication Unit 14: Communication Bus 20: Subject Terminal 21: Control Unit 22: Memory Unit 23: Communication Unit 24: Input Unit 25: Output Unit 26: Communication Bus 30: Evaluator Terminal 31: Control Unit 32: Memory Unit 33: Communication Unit 34: Input Unit 35: Output Unit 36: Communication Bus 40: Client Terminal 41: Control Unit 42: Memory Unit 43: Communication Unit 44: Input Unit 45: Output Unit 46: Communication Bus 111: Basic Display Control Unit 112: Answer Registration Unit 113: Evaluation Reception Unit 114: Data Acquisition Unit 115: Data Arrangement Unit 116: Data Registration Unit 117: Designation Reception Unit 118: Answer Generation Unit 119: Report Output Unit 120: Artificial Intelligence Unit 211: Display Unit 212: Operation Acquisition Unit 311: Display Unit 312: Operation Acquisition Unit 411: Display Unit 412: Operation Acquisition Unit AC1: Answer Input Field AC2: Generated Answer Display Field AD: Answer Display Screen AF: Answer Display Field AO: Data Addition Reception Object B11: Registration Button B21: Knowledge Addition Button B22: Answer Generation Button B23: Knowledge Editing Button B24: Knowledge Arrangement Button B31: Integrated Data Addition Button B32: Integrated Data Deletion Button B41: Split Data Addition Button B42: Split Data Deletion Button B43: Original Data Deletion Button B51: Creation Button B52: Service Selection Button B61: Cancel Button B62: Decision Button B71: Answer Output Button B81: Addition Button B82: Cancellation Button B91: Close Button BA: Grounds Display Area BD: Grounds Display Screen BM: Grounds Message CC: Registration Check Column CF: Category Display Column DA: Split Answer DD: Data Input Screen DF: Splittable Data Display Column DO: Data Display Reception Object DQ: Split Question EF: Evaluation Information Display Column EM1: Error Message EM2: Error Message MC: Memo Column MD: Answer Data Management Screen MF: Memo Display Column NF: Knowledge Display Column NL: Knowledge List OD: Answer Data Arrangement Screen OF: Duplicate Data Display Column PC: Past Data Comparison Column QC1: Question Input Column QC2: Specified Question Display Column QD: Question Input Screen QF: Question Display Column QI: Question Specified Column RA: Reference Data Display Area RC: Remarks Column RD: Answer Data Registration Screen RO: Regeneration Instruction Object SB: Selection Button SC: Service Selection Column SD: Service Selection Screen SO: Selection Object TB: Switching Tab TF: Condition Setting Column UA: Integrated Answer UQ: Integrated Question WA: Working Area

Claims

1. An information processing system, At least one processor; The processor is configured to execute the following steps by reading the program: In the data acquisition step, answer data is acquired that combines answered questions related to the security of the person to be evaluated and answers input by the person to the answered questions, The response data includes categories indicating the type of service to be subject to the security evaluation, the destination of the service, the plan or option included in the service, the business, or the department; The category is information that is assigned to the input answer and indicates a subject of the answer to the answered question, In the data organization step, the information processing system performs at least one of an integration process for integrating two or more pieces of answer data that have the same category and overlap into one piece of answer data, and a division process for dividing the answer data, in which the answered question can be divided into multiple questions or the input answer can be divided into multiple answers, into multiple pieces of answer data to which the category of the answer data before division has been assigned.

2. 2. The information processing system according to claim 1, The processor is further configured to perform the steps of: In the data registration step, the answer data obtained by the integration process or the division process is registered in a database; In the specification receiving step, a specification of the security question is received, In the answer generating step, an input answer to an answered question similar to the specified question is extracted as a reference answer from among the input answers included in the answer data registered in the database, and an answer to the question is generated based on the reference answer and answer generating reference information; Here, the reference information for generating an answer includes a correlation between the reference answer and the answer to the question.

3. 3. The information processing system according to claim 2, the reference information for answer generation includes an answer generation model that is trained to be able to output an answer to the question using the reference answer as an input, In the answer generating step, the reference answer is input to the answer generation model, and the answer generation model is caused to output an answer to the question.

4. 3. The information processing system according to claim 2, In the data reduction step, editing of the answer data obtained by the integration process or the division process is accepted, In the data registration step, the edited answer data is registered in the database.

5. 2. The information processing system according to claim 1, In the integration process, the answer data to be integrated is determined based on a similarity based on a comparison of features between the answered questions contained in the answer data, a similarity based on a comparison of features between the entered answers contained in the answer data, or a combination of these similarities.

6. 2. The information processing system according to claim 1, In the integration process, a plurality of pieces of answer data are input to an integration process model, and the integration process model is caused to output the answer data obtained by integrating the overlapping pieces of answer data; Here, the integrated processing model is a learning model that has been trained to be capable of receiving multiple pieces of answer data as input and outputting answer data that integrates overlapping pieces of answer data, in this information processing system.

7. 2. The information processing system according to claim 1, In the division process, the answer data is input to a division process model, and the division process model is caused to output a plurality of pieces of answer data obtained by dividing the answer data; Here, the splitting processing model is a learning model that has been trained to be capable of inputting the answer data and outputting a plurality of answer data obtained by splitting the answer data into a plurality of combinations of the answered questions and the input answers, in this information processing system.

8. 8. The information processing system according to claim 7, The segmentation processing model is a large-scale language model, In the division process, an instruction to divide the answer data into a plurality of combinations of the answered questions and the input answers is input to the division process model.

9. 9. The information processing system according to claim 8, In the splitting process, for answer data in which the answered question includes a phrase representing a conditional branch, a logical sum, or a logical product, the answered question is split based on the phrase, and an instruction to further split the input answer according to the split answered question is input to the splitting process model, the information processing system.

10. 9. The information processing system according to claim 8, In the splitting process, for the answer data in which the input answer is composed of a plurality of options, the input answer is split into each option, and an instruction to combine each of the split input answers with the original answered question is input to the splitting process model, the information processing system.

11. 9. The information processing system according to claim 8, In the splitting process, for the answer data including the answered questions each containing a plurality of questions and the input answers each consisting of a collective answer to the plurality of questions, the answered questions are split into the respective questions, and an instruction to combine each of the split answered questions with the original input answers is input to the splitting process model, the information processing system.

12. 2. The information processing system according to claim 1, An information processing system, in which, in the data organization step, the integration process and the division process are performed in parallel on the acquired response data, or the integration process is performed on the response data after the division process has been performed.

13. 2. The information processing system according to claim 1, The information processing system, in the integration process, determines the answer data to be integrated based on a similarity based on a comparison of features between the answered questions included in the answer data.

14. 2. The information processing system according to claim 1, The information processing system determines the answer data to be integrated based on a similarity between the answer data of the features of the entire combination of the answered question and the input answer contained in the answer data, in the integration process.

15. 2. The information processing system according to claim 1, In the integration process, depending on the answer format of the answered question, the information processing system selects one of the features of the answered question contained in the answer data, the features of the entered answer contained in the answer data, or the features of the entire combination of the answered question and the entered answer contained in the answer data, and determines the answer data to be integrated based on the similarity based on a comparison of the selected features.

16. 2. The information processing system according to claim 1, In the splitting process, the answered question is split into a plurality of split questions, and then split answers corresponding to the split questions are created by extracting content corresponding to each of the plurality of split questions from the input answer, in an information processing system.

17. 2. The information processing system according to claim 1, In the splitting process, the information processing system splits the input answer into multiple split answers, and then creates split questions corresponding to the split answers by extracting content corresponding to each of the multiple split answers from the answered question.

18. 1. An information processing method, comprising: An information processing method comprising the steps executed by the information processing system according to any one of claims 1 to 17.

19. A program, A program for causing a computer to execute each step of the information processing system according to any one of claims 1 to 17.

Images