Information processing systems, information processing methods, and programs

Abstract

We provide an information processing system that enables those being evaluated to efficiently answer security-related questions. [Solution] According to one aspect of the present invention, an information processing system is provided, comprising at least one processor, wherein the processor is configured to perform the following steps by reading a program, the data acquisition step involves acquiring answer data that combines answered questions about the security of the person being evaluated with the answers the person being evaluated has entered to those answered questions, and the data organization step involves performing at least one of the following: an integration process that combines two or more duplicate answer data into one answer data, and a splitting process that splits answer data into multiple answer data, where the answered questions can be divided into multiple questions, or the entered answers can be divided into multiple answers.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes a system audit support system including a checkpoint database that stores checkpoint array information data for identifying each checkpoint when auditing an audit target in an order according to rules, and stores evidence array information data for identifying evidence necessary for evaluating each checkpoint in association with the checkpoint array information data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] [[ID=3,4]] There is a need for a technology that allows respondents to answer check items efficiently.

[0005] In view of the above circumstances, the present invention aims to provide an information processing system and the like that enable an evaluation target person to efficiently answer security-related questions.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an information processing system is provided, comprising at least one processor, wherein the processor is configured to perform the following steps by reading a program: in the data acquisition step, answer data is acquired by combining answered questions about the security of the person being evaluated with the answers the person being evaluated has entered to those answered questions; and in the data organization step, at least one of the following is performed: an integration process that combines two or more duplicate answer data into one answer data, and a splitting process that splits answer data into multiple answer data, where the answered questions can be split into multiple questions, or the entered answers can be split into multiple answers.

[0007] In this configuration, the response data, which consists of combinations of past questions and answers, is organized, allowing the person being evaluated to efficiently answer security-related questions by referring to this response data. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the configuration of Information Processing System 1. [Figure 2] This block diagram shows the hardware configuration of the server device 10 and the target terminal 20. [Figure 3] This block diagram shows the hardware configuration of the evaluator terminal 30 and the client terminal 40. [Figure 4] This is a block diagram showing the functions realized by the server device 10 (control unit 11), the subject terminal 20 (control unit 21), the evaluator terminal 30 (control unit 31), and the requester terminal 40 (control unit 41). [Figure 5] This figure shows an example of the data entry screen DD displayed on the target terminal 20. [Figure 6] This figure shows an example of the response data management screen MD displayed on the target user's terminal 20. [Figure 7] This figure shows an example of the response data organization screen OD displayed on the target user's terminal 20. [Figure 8]This figure shows another example of the response data organization screen OD displayed on the target terminal 20. [Figure 9] This figure shows an example of a question input screen (QD) displayed on the user's terminal 20. [Figure 10] This figure shows an example of the service selection screen SD displayed on the target terminal 20. [Figure 11] This figure shows an example of a question input screen (QD) where the service name has been selected and the question has been entered. [Figure 12] This figure shows an example of the response display screen AD that is displayed on the target user's terminal 20. [Figure 13] This figure shows an example of the response data registration screen RD displayed on the target user's terminal 20. [Figure 14] This figure shows an example of the evidence display screen BD that is displayed on the target user's terminal 20. [Figure 15] This is an activity diagram showing an example of the flow of information processing (answer generation process) performed by Information Processing System 1. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.

[0010] Incidentally, the program for implementing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or it may be provided as a downloadable medium from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Also, in various information processing according to an embodiment, an input and an output corresponding to the input can be realized. Here, if an output is obtained as a result of the input, the mode of information (hereinafter referred to as reference information) referred to in such information processing is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, a predetermined function (including a judgment formula such as a regression formula constructed by a statistical method), a learned model in which the correlation between the input and the output is learned in advance, or a large language model capable of outputting a desired result by inputting a prompt.

[0012] Also, in one embodiment, the "part" may include, for example, hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various information is handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the level of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0013] Furthermore, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, and a memory, etc. Also, the processor may be a general-purpose processor or a dedicated circuit. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] 1. Hardware Configuration In this section, the hardware configuration will be described.

[0015] <Information Processing System 1> FIG. 1 is a configuration diagram showing an information processing system 1. The information processing system 1 includes a communication line 2, a server device 10, a plurality of target terminals 20, at least one evaluator terminal 30, and a plurality of requester terminals 40. The server device 10, the target terminals 20, the evaluator terminal 30, and the requester terminals 40 are configured to be mutually communicable through the communication line 2. The connections of the server device 10, the target terminals 20, the evaluator terminal 30, and the requester terminals 40 may be wired or wireless.

[0016] The information processing system 1 constitutes at least a part of a security evaluation system used by, for example, a plurality of evaluation targets (the first evaluation target U1 and the second evaluation target U2), at least one evaluator U3, and a plurality of evaluation requesters (the first evaluation requester U4 and the second evaluation requester U5). The information processing system 1 mainly performs evaluations of security regarding products and the like provided by the evaluation targets, provision of security evaluations, and the like. For example, the information processing system 1 mainly provides security evaluation services by evaluators. In one embodiment, the information processing system 1 consists of one or more devices or components. Hereinafter, these components will be described.

[0017] <Server Device 10> FIG. 2 is a block diagram showing the hardware configurations of the server device 10 and the target terminal 20. As shown in FIG. 2A, the server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a communication bus 14. The control unit 11, the storage unit 12, and the communication unit 13 are electrically connected inside the server device 10 via the communication bus 14.

[0018] <Control Unit 11> The control unit 11 performs processing and control of the overall operation related to the server device 10. The control unit 11 is, for example, a Central Processing Unit (CPU). The control unit 11 realizes various functions related to the server device 10 by reading predetermined programs stored in the memory unit 12. That is, information processing by software stored in the memory unit 12 is concretely realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being a single unit, and the server device 10 may have multiple control units 11 for each function. The server device 10 may also be composed of a combination of these.

[0019] <Storage section 12> The storage unit 12 stores various types of information as defined above. This can be done, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the server device 10 executed by the control unit 11, or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 12 stores various programs, variables, etc. related to the server device 10 executed by the control unit 11.

[0020] <Communications Department 13> The communication unit 13 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 5G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the server device 10 may communicate various information from the outside via the communication unit 13 and the network.

[0021] The server device 10 may be on-premises or in a cloud environment. A cloud-based server device 10 may provide the above-mentioned functions and processing in the form of, for example, SaaS (Software as a Service) or cloud computing.

[0022] <Target user device 20> The target terminal 20 is an information processing terminal used by the person being evaluated for security. The "person being evaluated" includes business partners or their representatives who have any form of business relationship with the evaluation requester, such as the provision of goods or services, the provision of cloud services, or the outsourcing of work. The person being evaluated is, for example, a business operator such as a service provider, supplier, or outsourcing organization to the person being evaluated. The "business partner organization" includes primary business partners that receive transactions directly from the person being evaluated, and secondary business partners that receive secondary transactions from primary business partners.

[0023] Furthermore, "organizations" include for-profit corporations (e.g., companies), non-profit corporations (e.g., cooperatives, foundations, etc.), and public corporations (e.g., local governments, etc.).

[0024] As shown in Figure 2B, the target terminal 20 comprises a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, and a communication bus 26. The control unit 21, storage unit 22, communication unit 23, input unit 24, and output unit 25 are electrically connected within the target terminal 20 via the communication bus 26. The descriptions of the control unit 21, storage unit 22, and communication unit 23 are the same as the descriptions of each part in the server device 10 and are therefore omitted.

[0025] <Input section 24> The input unit 24 receives operation inputs made by the user. The operation inputs are transmitted as command signals to the control unit 21 via the communication bus 26. The control unit 21 can perform predetermined controls or calculations based on the transmitted command signals as needed. The input unit 24 may be included in the housing of the user terminal 20 or it may be an external component. For example, the input unit 24 may be implemented as a touch panel integrated with the output unit 25. When the input unit 24 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 24. Instead of a touch panel, the input unit 24 can be a switch button, mouse, trackpad, QWERTY keyboard, etc.

[0026] <Output section 25> The output unit 25 displays a graphical user interface (GUI) screen that can be operated by the user. The output unit 25 may be included in the casing of the user terminal 20 or it may be an external component. Specifically, the output unit 25 can be implemented as a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display. It is preferable that these display devices be used in accordance with the type of user terminal 20.

[0027] <Evaluator terminal 30> The evaluator terminal 30 is an information processing terminal used by evaluators to evaluate the responses of the person being evaluated. The evaluator is an organization such as a company that conducts security evaluations (for example, a security evaluation company that conducts security evaluations of services such as cloud services, software, etc.) or a person in charge of such an organization. The evaluator conducts the security evaluation of the person being evaluated via the information processing system 1 from the evaluator terminal 30, for example.

[0028] Figure 3 is a block diagram showing the hardware configuration of the evaluator terminal 30 and the client terminal 40. As shown in Figure 3A, the evaluator terminal 30 comprises a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, and a communication bus 36. The control unit 31, storage unit 32, communication unit 33, input unit 34, and output unit 35 are electrically connected within the evaluator terminal 30 via the communication bus 36. The explanation of the control unit 31, storage unit 32, communication unit 33, input unit 34, and output unit 35 is the same as the explanation of each part in the subject terminal 20, so it is omitted.

[0029] <Client terminal 40> The client terminal 40 is an information processing terminal used by the evaluation requester who requests an evaluation of the security of the entity being evaluated. The "evaluation requester" is an individual, organization, or person in charge of such organization that has a business relationship with the entity being evaluated. The evaluation requester is an organization that requests a transaction from the trading partner organization (the entity being evaluated) and receives and uses the goods provided by the trading partner organization. For example, if the goods being traded are physical goods, the evaluation requester is the purchaser of the goods (finished products or parts) manufactured by the trading partner organization. Also, for example, if the goods being traded are services, the evaluation requester is a user of the services provided by the trading partner organization, for example, a user of cloud services provided by a cloud service provider. Furthermore, for example, if the goods being traded are operations (work), the evaluation requester is the client or contractor of the operations.

[0030] Applicants for evaluation include not only individuals or organizations that conduct transactions subject to evaluation (such as service use or outsourcing) with the person being evaluated, but also individuals or organizations that request evaluations of specific services. For example, applicants for evaluation may include companies that want to have the security of their cloud services, etc., evaluated, or companies that want to verify such evaluations.

[0031] As shown in Figure 4B, the client terminal 40 comprises a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, an output unit 45, and a communication bus 46. The control unit 41, storage unit 42, communication unit 43, input unit 44, and output unit 45 are electrically connected within the client terminal 40 via the communication bus 46. The client terminal 40 is an information processing terminal used by users of the security assessment service provided by the server device 10. The descriptions of the control unit 41, storage unit 42, communication unit 43, input unit 44, and output unit 45 are the same as the descriptions of each part in the target terminal 20 and are therefore omitted.

[0032] 2. Functional Configuration This section describes the functional configuration of this embodiment. Information processing by software stored in the memory unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11 (at least one processor provided by the information processing system 1).

[0033] Figure 4 is a block diagram showing the functions realized by the server device 10 (control unit 11), the subject terminal 20 (control unit 21), the evaluator terminal 30 (control unit 31), and the requester terminal 40 (control unit 41).

[0034] As shown in Figure 4A, the server device 10 (control unit 11) comprises a basic display control unit 111, a response registration unit 112, an evaluation reception unit 113, a data acquisition unit 114, a data organization unit 115, a data registration unit 116, a designation reception unit 117, a response generation unit 118, a report output unit 119, and an artificial intelligence unit 120.

[0035] As shown in Figure 4B, the subject terminal 20 (control unit 21) includes a display unit 211 and an operation acquisition unit 212. As shown in Figure 4C, the evaluator terminal 30 (control unit 31) includes a display unit 311 and an operation acquisition unit 312. As shown in Figure 4D, the client terminal 40 (control unit 41) includes a display unit 411 and an operation acquisition unit 412.

[0036] <Basic display control unit 111> The basic display control unit 111 is configured to display various information on the subject terminal 20, the evaluator terminal 30, and the requester terminal 40. For example, the basic display control unit 111 displays the input form in which the subject enters their answers, the answers registered by the subject, and the evaluator's evaluation results of the answers on the display unit 211 of the subject terminal 20, the display unit 311 of the evaluator terminal 30, or the display unit 411 of the requester terminal 40.

[0037] <Response Registration Section 112> The response registration unit 112 is configured to register information about the security of the person being evaluated in order to create a report or document to be presented to the requester of the evaluation. Specifically, the response registration unit 112 displays a predetermined input form on the person being evaluated's terminal 20 for receiving answers to security-related questions from the person being evaluated, and registers the answers entered in the input form as evaluation information.

[0038] The person being evaluated enters their answers according to service attributes, such as the type of service being evaluated, the service provider, and the types of plans included in the service. In other words, a person being evaluated who provides multiple services or plans may enter answers for each of these service attributes, and the answer registration unit 112 may register evaluation information for each service attribute.

[0039] The "services" provided by the evaluated party include services delivered via the internet, such as SaaS, IaaS (Infrastructure as a Service), PaaS (Platform as a Service), and cloud services. In this case, the "evaluated party" includes cloud service providers, etc. Furthermore, the "services" provided by the evaluated party may also be services provided to business partners, customers, etc.

[0040] "Questions" are matters that require the evaluated party to answer, and may include, for example, matters concerning the evaluated party's own security, the security of the services provided by the evaluated party, etc. "Questions" may also include matters concerning the security of other organizations included in the evaluated party's supply chain. "Other organizations" include, for example, organizations to which the evaluated party outsources its operations (including contractors, subcontractors, sub-subcontractors, etc.), and other organizations that the evaluated party deals with. Questions may also include so-called security checklists.

[0041] "Evaluation information" consists of the evaluator's answers to multiple questions regarding the service's security, and is the information that the evaluator checks and evaluates. Evaluation information may be evaluated only once or multiple times. In other words, the evaluation of evaluation information may include an initial evaluation (so-called review), an intermediate evaluation, a final evaluation, etc. The initial evaluation and intermediate evaluation are evaluations in which further evaluations are conducted after the initial evaluation. Multiple evaluations may be performed by the same evaluator or by different evaluators (so-called reviewers, etc.). The final evaluation creates an evaluation that will be used in the report or document provided to the evaluation requester. In addition, evaluation information may be evaluated again by the evaluator when it is modified or updated by the evaluator. The evaluator's evaluation results of the evaluation information will be used as part of the report or document or as reference information.

[0042] The evaluation information may include the answers along with the corresponding questions. The evaluation information may also include service attributes (for example, information indicating the type of service, the service provider, the types of plans and options included in the service, or a combination thereof).

[0043] The evaluation information is registered, for example, in the answer database of the memory unit 12. The answers to each question included in the evaluation information are the answers that the person being evaluated entered into the input form for each question presented in the input form, and these answers are registered in the answer database. The answers to a single question may be hierarchical.

[0044] "Registering evaluation information" includes both registration with all questions answered (final registration) and registration with some questions answered (some answers left blank) (save as draft). Furthermore, "save as draft" may also include registration with all questions answered, but before submitting the answers to the evaluators.

[0045] Questions in the evaluation information include, for example, those that confirm the status or evaluation of security certifications, service levels, scope of responsibility, security measures within the evaluated entity, handling of data related to the evaluation requester, handling of data managed by the evaluated entity, and service development, maintenance, or operational policies (such as account management) (questions that require answers from the evaluated entity).

[0046] <Evaluation Reception Department 113> The evaluation reception unit 113 is configured to receive evaluations from evaluators regarding the evaluation information registered by the response registration unit 112, and to register the evaluated evaluation information along with the evaluation results in a response database or the like.

[0047] "Evaluation" refers to the act of checking for deficiencies or inconsistencies in the answers included in the evaluation information, creating comments on the content of the answers, assessing the content of the answers, and scoring the answers based on predetermined criteria. As mentioned above, evaluation may be carried out by multiple evaluators, and the first evaluator may request the second evaluator to evaluate the registered evaluation information. In that case, the evaluation reception unit 113 may accept the evaluation from the second evaluator after the evaluation by the first evaluator.

[0048] For example, if the evaluation is conducted in two stages, the initial evaluation (review) involves checking for deficiencies and inconsistencies, and creating points of concern. Subsequently, the evaluation of the evaluation information after the initial evaluation is performed as a final evaluation based on predetermined criteria, including assessment and scoring of the content (responses) and creation of comments on the evaluation information as a whole. A report or document (e.g., a security report) containing the results is then created. The report or document is created by the final evaluator or by the evaluation unit of the control unit 11. The evaluation unit is configured to generate part or all of the report or document based on the input evaluation information. The evaluation results of the evaluation information may be included in the report or document as part of the report or as reference information. The report or document is transmitted to the requester terminal 40 of the evaluation requester who requested the report or document.

[0049] The evaluation reception unit 113 receives input such as characters and sentences used for evaluation from the evaluator terminal 30. Typically, evaluations are performed on evaluation information for which all questions have been answered. Evaluation information that contains unanswered questions (questions for which answers have not been registered) is not subject to evaluation, and the evaluation reception unit 113 does not need to accept the evaluation. Furthermore, even if evaluation information contains unanswered questions (questions for which answers have not been registered), only the questions that have been answered may be partially subject to evaluation.

[0050] <Data acquisition unit 114> The data acquisition unit 114 is configured to acquire and record data for automatically generating answers to questions in the evaluation information. Specifically, the data acquisition unit 114 acquires answer data that combines the security-related questions already answered by the person being evaluated with the answers the person being evaluated has entered to those questions.

[0051] For example, the data acquisition unit 114 receives from the subject terminal 20 the answered questions regarding the security of the services provided by the subject, the subject's entered answers to those answered questions, and the designation of the category assigned to the entered answers. The unit then adds the answer data, including the answered questions, entered answers, and categories, to the generation database. The answer data added to the generation database may also be called knowledge. Each answer data set includes at least the data of the answered questions, the data of the entered answers, and information indicating the category. The data of the answered questions and the data of the entered answers may be text data, vector data as described later, or both.

[0052] The categories assigned to the entered responses are information indicating the type of service, the recipient of the service (for example, the organization name of the evaluation requestor, or other attributes of the evaluation requestor), the types of plans and options included in the service, or combinations thereof. This allows the entered responses to be labeled using the types of services, plans, and options provided by the person being evaluated. As a result, the accuracy of the responses generated by the response generation unit 118 is improved. It also becomes possible to generate responses that correspond to the types of services, plans, and options used by the evaluation requestor. The categories may also be information indicating a business, department, etc.

[0053] "Service type" may be interpreted as "service name" and represents the type or name of the service targeted by the entered response. For example, if the person being evaluated provides Service A and Service B, at least one of "Service A" and "Service B" may be assigned to the entered response as the service type. "Type of plan included in the service" refers to one or more types of contract plans set for the same service (e.g., Service A), such as "Standard Plan" and "Premium Plan." Multiple plans set for the same service may differ from each other in terms of available features, usage period, usage fees, etc. Furthermore, even for the same service, different categories may be assigned depending on the service provider, etc.

[0054] Furthermore, the category of the entered response does not necessarily have to be entered. In other words, the response data acquired by the data acquisition unit 114 does not necessarily have to include a category.

[0055] Answered questions included in the response data may not be assigned a category, or they may be assigned a category of a different type than the category assigned to the entered answers (hereinafter referred to as "question category"). Furthermore, questions before they are answered may also be assigned a question category. The question category is used, for example, as a filtering condition for the response data when displaying the response data or generating answers using the response data.

[0056] Question categories classify the content of answered or pending questions. For example, answered or pending questions about contracts are assigned the "Contract" category, while answered or pending questions about company policies or rules are assigned the "Policies / Rules" category. In other words, the categories assigned to submitted answers and the categories assigned to answered or pending questions differ in their perspective.

[0057] If a question category is assigned to an answered or pending question, the data acquisition unit 114 may, for example, accept the selection of a question category from a predefined group of categories at the user terminal 20. This group of categories consists of different categories from those assigned to the entered answers. Alternatively, the data acquisition unit 114 may automatically assign the question category by referring to the questions and their corresponding question categories in the generation database. For example, the data acquisition unit 114 may extract similar stored questions to the received question and assign the question category assigned to those questions to the received question.

[0058] The response data may include first response data based on the questions and answers entered by the person being evaluated without using the input form provided by the response registration unit 112, and second response data based on evaluation information. Furthermore, the response data may include third response data based on the answers generated by the response generation unit 118. Therefore, the data acquisition unit 114 may acquire the response data by at least one of a first data acquisition flow for acquiring the first response data and a second data acquisition flow for acquiring the second response data, and a third data acquisition flow for acquiring the third response data. Furthermore, it may be configured to be able to execute all of the first, second, and third data acquisition flows.

[0059] In the first data acquisition flow, the data acquisition unit 114 receives input data from the user terminal 20, which combines answered questions and entered answers. For example, the data acquisition unit 114 displays a UI including question input fields and answer input fields on the user terminal 20 and accepts input of answered questions and entered answers. Furthermore, the data acquisition unit 114 accepts input of categories for the entered answers from the user terminal 20. The data acquisition unit 114 generates first answer data by assigning the accepted categories to the entered answers. This allows entered answers that do not have categories assigned to them to be registered as categorized first answer data for use by the answer generation unit 118. The order of input (reception order) of answered questions, entered answers, and categories does not matter.

[0060] The data acquisition unit 114 may, for example, refer to the types of services and plans that the person being evaluated has previously registered as user information in the information processing system 1, and present these to the person's terminal 20 as candidates for a category called "Type of Service" or "Type of Plan Included in the Service," and accept the selection of a category from the person's terminal 20. Alternatively, the data acquisition unit 114 may present registered evaluation requesters, such as organizations that have previously received reports or documents from the information processing system 1, as candidates for a category called "Destination of Service."

[0061] Furthermore, the data acquisition unit 114 may accept input (specification) of multiple categories for a single entered response. In other words, the data acquisition unit 114 may assign multiple categories to a single entered response. For example, if a response is common to multiple services, multiple service names may be assigned to the response as categories.

[0062] Furthermore, for submitted responses that are not category-dependent (i.e., not classified into a specific category and applicable to any service, plan, provider, etc.), the evaluator does not need to enter a category. The data acquisition unit 114 may also accept the designation (input) of an "uncategorized category" for submitted responses that are not category-dependent. For example, the data acquisition unit 114 may accept the designation (input) of a "common" category as an uncategorized category.

[0063] Figure 5 shows an example of the data entry screen DD displayed on the user terminal 20. The data entry screen DD includes a registration check field CC, a question input field QC1, an answer input field AC1, a memo field MC, a service selection field SC, a past data comparison field PC, and a registration button B11.

[0064] The registration check box CC displays checkboxes to select which of the questions and answers entered by the person being evaluated in the question input field QC1 and the answer input field AC1 should be registered as the first answer data. There is one checkbox per row (i.e., one checkbox per question).

[0065] The person being evaluated enters a pair of answered questions and their entered answers into the question input field QC1 and the answer input field AC1, respectively, on the same row. The questions and answers are entered, for example, by copying and pasting from reference data (e.g., an Excel file, a CSV file such as a Google Spreadsheet) containing the questions and answers, which the person being evaluated has prepared on the person's terminal 20. The data acquisition unit 114 may also accept the upload of reference data, extract the questions and answers from the reference data, and display them in the question input field QC1 and the answer input field AC1. Furthermore, the data acquisition unit 114 may accept operations such as repeatedly copying cells on the matrix containing the question input field QC1, the answer input field AC1, etc., on the data input screen DD. The first answer data is registered, for example, as part of a file (e.g., an Excel file, a Google Spreadsheet, etc.) containing answers to a security check sheet that the person being evaluated has created in the past.

[0066] The Memo field (MC) is a field where the person being evaluated can write comments. Information entered in the Memo field (MC) is registered as part of the first response data, for example, but is not referenced when generating the response.

[0067] The Service Selection Field SC is where the person being evaluated enters the services, etc., that are the subject of the answer entered in the Answer Input Field AC1 as a category. The Service Selection Field SC has a selection button SB for selecting a service from a list. When an input operation is performed on the selection button SB, a list of service candidates is displayed, for example, in the form of a pull-down menu. When the person being evaluated selects a type of service from the candidates, the service selected by the person being evaluated (corresponding to the "category to be assigned to the entered answer") is displayed in the Service Selection Field SC. If the answer corresponds to multiple services, multiple services will be displayed within the frame of one Service Selection Field SC. In the example in Figure 5, the Service Selection Field SC displays the service name (type of service), but the service provider (for example, the name of the evaluation requestor) or plan name (type of plan) may also be displayed as options (categories). The service name may also be entered as text by the person being evaluated. Furthermore, the information entered in the Service Selection field (SC) is not limited to the service name; any information that can be used as a category for the entered response is acceptable. For example, it may include the type of service, the service provider (e.g., the organization name of the evaluation requestor, or other attributes of the evaluation requestor), the types of plans and options included in the service, or combinations thereof.

[0068] Furthermore, if there are services or plans with different answers to the same question, the evaluator may create and input data with the same question but different answers and service names. In other words, for questions that have already been answered, duplicate entries with the same content are possible as long as the entered answers are different.

[0069] The past data comparison section on the PC displays the results of a comparison between the answered questions and entered answers included in the answer data already registered in the generation database (either only the first answer data, or any of the first, second, and third answer data). The comparison results include, for example, discrepancies (differences) in the content of entered answers for related answered questions, and duplicate answered questions. The display of duplicate answered questions in the past data comparison section on the PC may include links to display other duplicate answered questions, the number of duplicate answered questions, etc.

[0070] When an input operation is performed on the registration button B11, the combination of the answered question and the entered answer selected (checked) in the registration check field CC is registered in the generation database as the first answer data.

[0071] In the second data acquisition flow, the data acquisition unit 114 creates second response data from the evaluation information registered by the response registration unit 112 and adds it to the generation database. This allows response data to be created using pre-created evaluation information, thus reducing the effort required for the evaluators to input previously answered questions, entered responses, etc.

[0072] The first response data and the second response data may be added to a single common generation database, or they may be added to different sub-databases included in the generation database. That is, the data acquisition unit 114 may register the first data in a first sub-database included in the generation database, and register the second response data in a second sub-database included in the generation database.

[0073] The data acquisition unit 114, for example, presents a list of registered evaluation information to the subject terminal 20 and accepts the selection of evaluation information to be added to the generation database as second response data from the subject terminal 20. The data acquisition unit 114 accepts the selection of evaluation information by the subject terminal 20 as a specification of multiple answered questions and entered answers included in the evaluation information.

[0074] If the evaluation information includes service attribute information, the data acquisition unit 114 acquires the service attribute as a category of the entered response. In other words, the data acquisition unit 114 accepts the selection of evaluation information as a category specification. On the other hand, if the evaluation information does not include service attribute information, the data acquisition unit 114 may accept the input of a category for the entered response (evaluation information) from the subject terminal 20.

[0075] The evaluation information that the data acquisition unit 114 uses as source information for the second response data is registered (i.e., officially registered or saved as a draft), regardless of whether it has been evaluated or not. In other words, the data acquisition unit 114 accepts both evaluation information before evaluation and evaluation information after evaluation as source information for the second response data. The data acquisition unit 114 may also accept only evaluation information after evaluation as source information for the second response data. Furthermore, in the case of evaluation information saved as a draft, the data acquisition unit 114 uses only questions in which answers have been entered to create the response data. Note that "evaluated evaluation information" refers to evaluation information that has been evaluated (reviewed) by an evaluator at least once, and if multiple evaluations are performed, it includes information for which a final evaluation has not yet been performed.

[0076] In the third data acquisition flow, the data acquisition unit 114 adds the third response data, which consists of the designated questions received by the designated reception unit 117 (described later) as answered questions, and the answers to the designated questions generated by the response generation unit 118 and edited by the person being evaluated, to the generation database. This improves the accuracy of the responses generated by the response generation unit 118, as the data generated by the response generation unit 118 and further modified by the person being evaluated can be used as new response data.

[0077] Furthermore, the data acquisition unit 114 may add third response data to the generation database, in which designated questions for which the response generation unit 118 was unable to generate an answer are designated as answered questions, and the answers to designated questions entered by the person being evaluated are designated as entered answers. This allows for the addition of response data to generate answers to questions for which the response generation unit 118 was unable to generate an answer.

[0078] The data acquisition unit 114 may convert at least a portion of the text data (answered questions and entered answers) received as source information for the answer data into vector data and add the obtained vector data to the answer data. This allows the answer generation unit 118 to generate answers by referring to pre-prepared vector data (searching for reference answers), thereby increasing the processing speed of answer generation. Specifically, the data acquisition unit 114 should convert at least the answered questions into vector data.

[0079] Figure 6 shows an example of the response data management screen MD displayed on the participant terminal 20. The response data management screen MD includes an evaluation information display area EF, a knowledge display area NF, a knowledge addition button B21, and a response generation button B22.

[0080] The evaluation information display field EF displays a list of evaluation information received by the data acquisition unit 114. Specifically, the evaluation information display field EF displays information such as the identification name of the evaluation information (service name in this example), the last registration date and time, and the status. The status displays "Registered" for evaluation information for which the second response data based on the evaluation information has been added to the generation database (making it searchable by the response generation unit 118, described later). In addition, evaluation information for which the generation of the second response data based on the evaluation information has not been completed (for example, for which vector data conversion has not been completed) is displayed as "Registering".

[0081] The Knowledge Display area NF includes the Condition Setting area TF, the Knowledge Edit button B23, the Knowledge Organization button B24, and the Knowledge List NL. The Knowledge List NL displays a list of the first response data (knowledge) that the Data Acquisition Unit 114 has added to the generation database. The Condition Setting area TF accepts input of conditions to narrow down the knowledge displayed in the Knowledge List NL. In the example in Figure 6, the Condition Setting area TF accepts input of search keywords, input of the last updated date, and selection of the category of the entered response (related services, etc.) as filtering conditions. As shown in Figure 6, if the category is set to "All," knowledge from all categories (service names) (including knowledge without a category assigned) will be displayed. Note that in the Knowledge Display area NF, knowledge may also be displayed in groups, grouped by category.

[0082] When an input operation is performed on the knowledge editing button B23 in the knowledge display field NF, a knowledge editing screen conforming to the data input screen DD in Figure 5 is displayed on the user terminal 20, and the data acquisition unit 114 accepts the editing of knowledge (answered questions, entered answers, or categories) from the user terminal 20.

[0083] When an input operation is performed on the Knowledge Organization button B24 in the Knowledge display field NF, a screen for executing the knowledge organization by the data organization unit 115, described later, is displayed. Alternatively, when an input operation is performed on the Knowledge Organization button B24, the integration process described later may also be executed.

[0084] When the "Add Knowledge" button B21 is pressed on the response data management screen MD, the data input screen DD shown in Figure 5 is displayed on the user's terminal 20. Note that if no knowledge is registered, the "Add Knowledge" button B21 may be displayed in the knowledge display area NF. When the "Generate Response" button B22 is pressed, a screen is displayed to allow the response generation unit 118 (described later) to generate a response.

[0085] <Data Organization Section 115> The data organization unit 115 is configured to organize the response data acquired (registered) by the data acquisition unit 114. Specifically, the data organization unit 115 performs at least one of the following on the response data: integration processing and splitting processing.

[0086] <Integration Process> In the integration process, the data organization unit 115 integrates two or more duplicate response data into a single response data. The integration process may target only the first response data, or it may target one or more response data from the first, second, and third response data. For example, the integration process may merge the first response data and the second response data. If the response data has a category assigned to it, response data of the same category will be subject to the integration process. The integration process may be started, for example, by an input operation to the knowledge organization button B24 in Figure 6.

[0087] In the integration process, the data organization unit 115 may determine which response data to integrate based on similarity derived from comparing the features of answered questions included in two or more response data sets, similarity derived from comparing the features of entered answers included in two or more response data sets, or a combination of these similarities. This makes it possible to integrate duplicate response data within a range of similarity that is not limited to exact matches.

[0088] As features of answered questions or input answers, for example, vector data obtained by vectorizing sentences or words contained in the answered questions or input answers may be used. Vectorization is performed by quantification using known methods such as natural language processing by morphological analysis or encoding. The similarity between answered questions or between input answers is represented by the difference in features (e.g., the distance between vectors).

[0089] The data organization unit 115 may, for example, determine only the response data whose similarity (e.g., cosine similarity) between answered questions is above a threshold as duplicate data (i.e., response data to be merged), or it may determine only the response data whose similarity between entered responses is above a threshold as duplicate data. The data organization unit 115 may also determine the response data whose similarity between answered questions and the similarity between at least one of the entered responses is above a threshold as duplicate data. Furthermore, the data organization unit 115 may determine the response data whose similarity between answered questions and the similarity between entered responses are above a threshold, respectively as duplicate data. When using the similarity between answered questions and the similarity between entered responses for determination, these thresholds may be different or the same.

[0090] Furthermore, the data organization unit 115 may determine which response data to integrate by comparing the feature quantities (vector data) of the entire combination of answered questions and entered answers contained in a single response data with the other response data. In other words, the data organization unit 115 may determine duplicate data using the similarity of the entire response data (the difference in the feature quantities of the entire response data).

[0091] Furthermore, the data organization unit 115 may decide which features to use for similarity judgment from among the features of the answered questions, the features of the input answers, and the features of the entire response data, depending on the question type of the answered questions. Question types include, for example, single-choice, multiple-choice, free-response, and specified-response formats.

[0092] Furthermore, for example, with multiple response data sets that share a common answered question but contain different input answers, the accuracy of the integration process can be improved by using only the features of the answered question for determination. Also, for example, with response data sets that contain answered questions with the same intent but different contexts, keywords, etc., the accuracy of the integration process can be improved by using both the features of the answered question and the features of the input answer for determination.

[0093] The data organization unit 115 may input two or more response data into the duplicate detection model and have the duplicate detection model determine which response data to integrate. The duplicate detection model is, for example, a learning model included in the artificial intelligence unit 120 that has been trained to take two or more response data as input and output information about duplicates of response data (whether or not there are duplicates, or the similarity between the response data). The duplicate detection model is, for example, a learning model that has been trained using data of multiple response data and corresponding duplicate information (information indicating whether or not there are duplicates, or the similarity between the response data) as training data.

[0094] The duplicate detection model may be a generative AI that includes a large-scale language model. In this case, the data processing unit 115 takes two or more response data as input, inputs a prompt to the duplicate detection model that includes an instruction to extract duplicate response data from these response data, and causes the duplicate detection model to output the duplicate data. The data processing unit 115 may also generate a prompt that gives the duplicate detection model an instruction to extract duplicate data from two or more response data, and input this prompt to the duplicate detection model. In addition to the instruction to extract and output duplicate data and two or more response data, the data processing unit 115 may also input a prompt to the duplicate detection model that includes, for example, a sample of one or more combinations of response data and a sample of one or more corresponding duplicate data as an example, sample, or training data of input and output pairs.

[0095] The data organization unit 115 may determine duplicate data by comparing keywords included in answered questions and / or keywords included in entered answers. For example, the data organization unit 115 may determine duplicate data if the number of common keywords (including similar keywords) included in both answered questions and / or entered answers exceeds a predetermined number, or if the degree of agreement in the order of the common keywords exceeds a certain level. Furthermore, the data organization unit 115 may calculate keyword features based on the frequency of occurrence of keywords included in the answer data using natural language processing such as TF-IDF or Okapi BM25, and determine the similarity between the answer data using cosine similarity or the like with these features to determine duplicate data.

[0096] The data processing unit 115 may standardize the keywords included in each response data (answered questions or entered answers) before performing any of the duplicate determinations described above. Keyword standardization can be performed, for example, by converting keywords similar to standard words registered in a keyword list into those standard words.

[0097] The data organization unit 115 integrates the answered questions and entered answers contained in multiple response data sets that have been determined to be duplicates into a single answered question and entered answer set. The integration of multiple response data sets is performed, for example, by natural language processing, so that the integrated answered question and entered answer set does not contain any duplicate content and does not omit any content that was included in the original answered question and entered answer set.

[0098] In the integration process, the data organization unit 115 may input multiple response data into the integration processing model and have the integration processing model output response data that integrates the duplicate response data. The integration processing model is a learning model included in the artificial intelligence unit 120 that has been trained to take multiple response data as input and output response data that integrates the duplicate response data. As a result, the integration process is performed in parallel with the determination of duplicates, thereby improving the accuracy of integrating duplicate response data.

[0099] An integrated processing model is, for example, a learning model that has learned using multiple response data and their corresponding integrated response data as training data.

[0100] The integrated processing model may be a generative AI including a large-scale language model. In this case, the data organization unit 115 takes multiple response data as input and inputs a prompt to the integrated processing model that includes an instruction to output response data (integrated data) which is an integrated version of these response data, causing the integrated processing model to output the integrated data. The data organization unit 115 may also generate a prompt that gives the integrated processing model an instruction to generate integrated data from multiple response data and input this prompt to the integrated processing model. In addition to the instruction to generate and output integrated data and multiple response data, the data organization unit 115 may also input a prompt to the integrated processing model that includes, for example, a sample of one or more combinations of response data and a sample of one or more corresponding integrated data as an example, sample, or training data of input and output pairs.

[0101] If the integrated processing model is a large-scale language model, the data organization unit 115 may input instructions to the integrated processing model to create integrated data that does not have any duplicate content and does not omit any content that was included in the previously answered questions and entered answers before integration.

[0102] <Splitting process> In the splitting process, the data organization unit 115 splits answer data into multiple answer data if the answered question can be split into multiple questions, or if the entered answer can be split into multiple answers. The splitting process may target only the first answer data, or it may target one or more of the first, second, and third answer data. If the answer data has a category assigned to it, the split answer data will be assigned the same category as the original answer data.

[0103] Whether an answered question is divisible or not is determined, for example, by the presence or absence of phrases indicating a combination of multiple items. Examples of "phrases indicating a combination" include phrases that indicate conditional branching (e.g., "if...", "when...", etc.), phrases that indicate logical conjunction (e.g., "and", "and", etc.), and phrases that indicate logical disjunction (e.g., "or", "or else", etc.).

[0104] If an answered question can be divided into multiple questions, the data organization unit 115, for example, divides one answered question into multiple questions, and then creates corresponding answers for each of the divided questions based on the input answers. The answers corresponding to the divided questions may be the input answers themselves before division, or they may be extracted from the input answers to correspond to the divided questions.

[0105] Whether an entered response is divisible is determined, for example, by the presence or absence of a phrase indicating a combination of multiple items, similar to the determination of whether an answered question is divisible. If an entered response is divisible into multiple responses, the data organization unit 115, for example, divides one entered response into multiple responses, and then creates questions corresponding to each of the divided responses based on the answered question. The questions corresponding to the divided responses may be the answered question itself before division, or they may be content extracted from the answered question corresponding to the divided responses, or they may be newly generated based on the divided responses.

[0106] In the splitting process, the data organization unit 115 may input the response data into the splitting model and have the splitting model output multiple response data obtained by splitting the response data. The splitting model is a learning model included in the artificial intelligence unit 120 that has been trained to take the response data as input and output multiple response data obtained by splitting the response data into combinations of multiple answered questions and input answers. As a result, the splitting process is performed together with the determination of whether or not splitting is possible, thereby improving the accuracy of splitting the response data.

[0107] A segmentation model is, for example, a learning model that has been trained using the response data and the corresponding segmented response data as training data.

[0108] The segmentation processing model may be a generative AI that includes a large-scale language model. In this case, the data organization unit 115 takes the response data as input and inputs a prompt to the segmentation processing model that includes an instruction to output response data obtained by segmenting this response data (segmented data), causing the segmentation processing model to output the segmented data. The data organization unit 115 may also generate a prompt that gives the segmentation processing model an instruction to generate multiple segmented data from the response data and input the prompt to the segmentation processing model. In addition to the instruction to generate and output segmented data and the response data, the data organization unit 115 may also input a prompt to the segmentation processing model that includes, for example, one or more sample response data and one or more corresponding sample segmented data as an example, sample, or training data of input and output pairs.

[0109] Thus, when the partitioning model is a large-scale language model, the data organization unit 115 may input instructions to the partitioning model to divide the response data into multiple combinations of answered questions and entered answers. This makes it possible to obtain question-based response data that is easy to use for automatic response generation based on natural language processing. For example, the data organization unit 115 may input a prompt to the partitioning model, which is a large-scale language model, that includes instructions such as, "Decompose the entered answers as much as possible into simple question-and-answer pairs."

[0110] In the splitting process, the data organization unit 115 may input a first instruction to the splitting processing model, which is a large-scale language model, that splits the answered questions based on the phrases representing conditional branching, logical OR, or logical AND in the answered questions, and further splits the input answers according to the split answered questions. This improves the splitting accuracy of both the answered questions and the input answers. Note that the answers obtained by splitting the input answers may include the same content or duplicates. The first instruction may be inserted into a single prompt and input to the splitting processing model, or it may be split into multiple prompts (for example, a prompt that instructs up to the splitting of the answered questions and a prompt that instructs the generation of input answers according to the split questions) and input to the splitting processing model.

[0111] In the splitting process, the data organization unit 115 may input a second instruction to the splitting processing model, which is a large-scale language model, which instructs the response data, where the input response consists of multiple choices, to split the input response according to each choice, and then to combine each of the split input responses with the original answered question. This improves the accuracy of splitting the input response. The second instruction may be inserted into a single prompt and input to the splitting processing model, or it may be divided into multiple prompts (for example, a prompt that instructs up to the splitting of the input response and a prompt that instructs the combination of the split responses with the answered question) and input to the splitting processing model.

[0112] In the splitting process, the data organization unit 115 may input a third instruction to the splitting processing model, which is a large-scale language model, to split the answered questions into individual questions and then combine each of the split answered questions with the original input answers. This improves the accuracy of splitting the answered questions. The third instruction may be inserted into a single prompt and input to the splitting processing model, or it may be split into multiple prompts (for example, a prompt instructing the splitting of the answered questions and a prompt instructing the combination of the input answers to the split questions) and input to the splitting processing model.

[0113] Furthermore, the data processing unit 115 may divide the entered responses and generate questions for the divided responses based on sentences or words contained in the entered responses. For example, for response data that includes the answered question "Do you have an SLA?" and the entered response "We do not have an SLA, but our target service availability is xx% or higher, and our actual availability over the past year is xx%", the data processing unit 115 divides the entered response into a first response "No, we do not have an SLA", a second response "Our target service availability is xx% or higher", and a third response "Our actual availability over the past year is xx%". Here, the second and third responses are unasked questions that were not directly asked in the entered response. Next, the data processing unit 115 generates the first question, "Do you have an SLA?" in response to the first answer, the second question, "What is the target value for service availability?" in response to the second answer, and the third question, "What is the actual availability rate for the past year?" in response to the third answer, and creates response data containing the first question and first answer, response data containing the second question and second answer, and response data containing the third question and third answer as split data. Here, the second and third questions are not included in the answered questions before splitting, and are questions generated based on sentences or words included in the input answers before splitting.

[0114] Therefore, in the splitting process, the data organization unit 115 may input a fourth instruction to the splitting processing model, which is a large-scale language model, to split the input responses into items, and then generate the input questions corresponding to the answers containing unasked items among the split input responses, based on the original input responses. This results in answer data in which one question item and one answer are combined on a one-to-one basis.

[0115] The data processing unit 115 may determine which of the first, second, third, fourth, and other instructions should be input to the split processing model by performing natural language processing on the response data (answered questions or entered answers), and may insert the determined instructions as prompts to the split processing model.

[0116] <Relationship between integration and partitioning processes> The data processing unit 115 may perform only integration processing, only splitting processing, or both integration and splitting processing on the multiple response data acquired by the data acquisition unit 114. These processes may be performed, for example, based on instructions from the person being evaluated. Alternatively, these processes may be performed automatically when the response data is acquired, without instructions from the person being evaluated.

[0117] The data processing unit 115 may perform integration processing and splitting processing in parallel on the response data acquired by the data acquisition unit 114, or it may perform integration processing on the response data after splitting processing. This makes it possible to suppress the repeated application of integration processing and splitting processing to the same response data, for example, by avoiding the application of splitting processing to response data created by integration processing.

[0118] <Editing response data> The data organization unit 115 may accept edits to the response data obtained through integration or splitting processing from the subject terminal 20. This allows the subject to make adjustments (customizations) to the response data after integration or splitting processing, thereby improving the accuracy of response generation by the response generation unit 118 (described later) and reducing the effort required to correct the generated responses.

[0119] "Editing response data" includes, for example, modifying text included in answered questions, modifying text included in entered responses, and deleting newly generated response data. Furthermore, for multiple response data obtained through the splitting process, deletion may be accepted for each individual response data, or a batch deletion of multiple response data obtained from a single response data may be accepted.

[0120] The data organization unit 115 may delete the original response data in response to the registration of the response data obtained by the integration or splitting process by the data registration unit 116 (described later) into the generation database. The data organization unit 115 may also delete the original response data upon receiving instructions from the person being evaluated. Furthermore, the data organization unit 115 may display a list of deleted response data or response data scheduled to be deleted on the person's terminal 20.

[0121] Figure 7 shows an example of the response data organization screen OD displayed on the target terminal 20. The response data organization screen OD is displayed, for example, when an input operation is performed on the knowledge organization button B24 on the response data management screen MD in Figure 6. The response data organization screen OD includes a switching tab TB and a work area WA.

[0122] The toggle tab TB is an object that accepts the selection of information (work content) to be displayed in the work area WA. In the example in Figure 7, the toggle tab TB includes the "Integrate Knowledge" tab for integrating response data and the "Decompose Knowledge" tab for splitting response data. Note that Figure 7 shows the state where the "Integrate Knowledge" tab of the toggle tab TB is selected.

[0123] As shown in Figure 7, when the integration of response data is selected using the switch tab TB, the Duplicate Data Display Field OF is displayed in the Work Area WA. The Duplicate Data Display Field OF displays the questions and answers of the response data that were determined to be duplicates, along with the integrated question UQ and integrated answer UA, which are the result of integrating them, side by side. In addition, the Duplicate Data Display Field OF is displayed in the Work Area WA for each set of duplicate response data. The Duplicate Data Display Field OF displays the Add Integrated Data button B31 and the Delete Integrated Data button B32.

[0124] In the duplicate data display area OF, edits to the integrated question UQ and integrated answer UA are accepted from the user terminal 20. When an input operation is performed on the integrated data add button B31, the answer data consisting of the combination of integrated question UQ and integrated answer UA is registered in the generation database, and the answer data determined to be duplicate (the answer data from the source of integration) is deleted from the generation database. Also, when an input operation is performed on the integrated data delete button B32, the integrated question UQ and integrated answer UA are discarded, and the answer data determined to be duplicate is retained in the generation database.

[0125] Figure 8 shows another example of the response data organization screen OD displayed on the participant terminal 20. The response data organization screen OD in Figure 8 shows the state in which the "Decompose Knowledge" tab of the switching tab TB is selected in the response data organization screen OD of Figure 7.

[0126] As shown in Figure 8, when the option to split the response data is selected using the switch tab TB, the divisible data display area DF is displayed in the work area WA. The divisible data display area DF displays the question and answer of the response data that has been determined to be divisible, and the divided question DQ and divided answer DA, respectively, side by side for comparison. In addition, the divisible data display area DF is displayed in the work area WA for each divisible response data. The divisible data display area DF displays the add divided data button B41, the delete divided data button B42, and the delete original data button B43.

[0127] In the divisible data display area DF, editing of the divisible question DQ and divisible answer DA is accepted from the user terminal 20. The add divisible data button B41 and delete divisible data button B42 are displayed for each set of divisible question DQ and divisible answer DA. When input is performed on the add divisible data button B41, the answer data consisting of the corresponding combination of divisible question DQ and divisible answer DA is registered in the generation database. When input is performed on the delete divisible data button B42, the corresponding divisible question DQ and divisible answer DA are discarded and removed from the divisible data display area DF.

[0128] The "Delete Original Data" button B43 is displayed for each divisible data point. When an input operation is performed on the "Delete Original Data" button B43, the divisible data (the original response data from which it was divided) is deleted from the generation database.

[0129] The Add Split Data button B41 and the Delete Split Data button B42 may be displayed for each divisible data item. In this case, when an input operation is performed on the Add Split Data button B41, all pairs of split questions DQ and split answers DA displayed in the divisible data display field DF are registered as answer data in the generation database. Also, in this case, when an input operation is performed on the Delete Split Data button B42, all split questions DQ and split answers DA displayed in the divisible data display field DF are deleted.

[0130] <Data Registration Section 116> The data registration unit 116 is configured to register the response data obtained through integration or splitting into a generation database. The data registration unit 116 may also register the response data edited by the subject terminal 20 into the generation database, as described above.

[0131] The response data obtained through integration or splitting may be registered as text data or as vector data.

[0132] <Designated reception desk 117> The designated reception unit 117 is configured to receive designations of security questions from the target terminal 20. In addition to designations of security questions, the designated reception unit 117 may also receive designations of categories from the target terminal 20. The questions that the designated reception unit 117 receives include, for example, new questions that the person being evaluated has not yet answered, or questions that have been answered in another category (e.g., service name) but have not been answered in the new category (designated category).

[0133] The designated reception unit 117 accepts the designation of questions, for example, by inputting questions from the target terminal 20. Alternatively, the designated reception unit 117 may accept the designation of questions by calling an input form for registering evaluation information on the target terminal 20 (selection of services, plans, etc., for which evaluation information will be created). In this case, calling the input form will specify multiple questions included in the input form.

[0134] The designated reception unit 117 may present a list of categories included in the response data registered in the generation database and accept category selections from the subject terminal 20. This reduces the effort required for the evaluated person to select a category. The order in which the question is selected and the category is selected does not matter.

[0135] When a question is specified by calling an input form, if the input form is called after the category has been entered, the specification reception unit 117 accepts the category specification based on the input (selection) of the category (service name, plan name, etc.) at the time the input form is called (when the evaluation information is created).

[0136] Figure 9 shows an example of a question input screen QD displayed on the user terminal 20. The question input screen QD includes a create button B51, a service selection button B52, and a question specification field QI. The create button B51 is an object that instructs the system to generate an answer to the question entered in the question specification field QI, according to the service name (category) selected by the service selection button B52. The create button B51 does not accept input when no service name is selected or when no questions have been entered.

[0137] When an input operation is performed on the service selection button B52, a UI for selecting a service is displayed on the user terminal 20. Figure 10 shows an example of the service selection screen SD displayed on the user terminal 20. The service selection screen SD includes a selection object SO, a cancel button B61, and a confirm button B62. A selection object SO is assigned to each service name selection. In the example in Figure 10, a single-choice radio button is placed as the selection object SO. When an input operation is performed on the cancel button B61, the selection of the service name is canceled. When an input operation is performed on the confirm button B62, the selected service name is accepted as a category of answer data to be referenced when generating the question, and the service name is displayed on the question input screen QD in Figure 9.

[0138] The question designation field QI shown in Figure 9 accepts questions from the participant terminal 20. Participants can simultaneously enter multiple questions for the same category (e.g., service name) into the question designation field QI.

[0139] Figure 11 shows an example of a question input screen QD in which a service name has been selected and a question has been entered. In the question input screen QD of Figure 11, the selected service name is displayed in the service selection button B52. As shown in Figure 11, when an input operation is performed on the create button B51 with a service name selected and at least one question entered, the specification reception unit 117 accepts the specification of the security question and the category specification, and then the answer generation unit 118 generates the answer. In addition, the question specification field QI may be entered by copying and pasting from a data file containing questions (for example, an Excel file, a CSV file such as a Google Spreadsheet) prepared by the person being evaluated on the target terminal 20.

[0140] The designated reception unit 117 may further accept the designation of the question's answer format from the target terminal 20. The question's answer format is the format of the answer created by the answer generation unit 118, and includes, for example, a multiple-choice format in which the answer is selected from predetermined options (for example, a combination of "yes" and "no"), a free-response format, and a designated description format in which keywords or sentences are created according to a predetermined sentence pattern.

[0141] <Answer generation unit 118> The answer generation unit 118 is configured to extract input answers from the answer data registered in the generation database that are similar to the specified question (the specified question for which the specification reception unit 117 has received the specification) as reference answers, and to generate an answer to the specified question based on these reference answers and the answer generation reference information. As a result, the answer is generated based on the answer data obtained through integration processing or splitting processing, thus improving the accuracy of answer generation.

[0142] The response generation unit 118 may extract, as reference responses, input responses from the input responses included in the response data registered in the generation database that correspond to the specified category (specified category) that the specification reception unit 117 has received a specification for, and that are similar to the specified question.

[0143] "Pre-entered responses corresponding to the specified category" are pre-entered responses that have been assigned the same category as the specified category. The response generation unit 118 searches the generation database for response data that includes pre-entered questions similar to the specified question, after narrowing down the response data to be searched using the specified category.

[0144] The response generation unit 118 may extract input responses to similar questions that have been answered but have not been assigned a category, or that are assigned an "uncategorized" category indicating they do not belong to a specific category, as reference responses regardless of the specified category. This reduces the effort required for the evaluator to assign individual categories to responses that are common to multiple categories (such as service names), responses related to the evaluator themselves, etc.

[0145] The response generation unit 118 may extract reference responses by referring to any of the response data from the first response data, second response data, and third response data. Alternatively, the response generation unit 118 may extract reference responses by referring to the first response data and second response data. This allows the response generation unit 118 to generate responses based on both the responses entered in the input form provided by the response registration unit 112 and other responses (for example, responses entered outside of the input form or responses prepared by the person being evaluated), thereby improving the accuracy of the responses. Furthermore, if third response data is accumulated, the response generation unit 118 may also refer to the third response data in addition to the first and second response data to extract reference responses.

[0146] The response generation unit 118 may determine the similarity between the answered questions and the specified question by comparing vector data obtained by vectorizing the content of the answered questions included in the response data with vector data obtained by vectorizing the content of the specified question. This allows for the rapid and accurate extraction of reference answers. Note that "answered questions similar to the specified question" also include answered questions that are identical or substantially identical to the specified question.

[0147] The answer generation unit 118 may input the answered question and the specified question into a similarity judgment model, have the similarity judgment model output the similarity score between the two, and then determine the similarity between the answered question and the specified question based on that similarity score. The similarity judgment model is, for example, a learning model included in the artificial intelligence unit 120 that has been trained to take two questions as input and output the similarity score between the two questions.

[0148] The similarity judgment model is a learning model that has learned using training data consisting of sets of training questions and the similarity scores corresponding to those sets of questions. The similarity judgment model may also be a generative AI that includes a large-scale language model. In this case, the answer generation unit 118 takes the answered questions and the specified question as input, inputs a prompt to the similarity judgment model that includes an instruction to output the similarity score between the answered questions and the specified question, and causes the similarity judgment model to output the similarity score. The answer generation unit 118 may also generate a prompt that gives the similarity judgment model an instruction to output the similarity score between the answered questions and the specified question, and input this prompt to the similarity judgment model. In addition to the similarity score output instruction and the answered questions and specified questions, the answer generation unit 118 may also input a prompt to the similarity judgment model that includes, for example, one or more samples of answered questions and specified questions and one or more samples of their corresponding similarity scores as bare examples, samples, or training data for input and output.

[0149] The similarity classification model may be a learning model configured to take a specified question as input, refer to the answer data (generation database), and output a reference answer. In this case, the similarity classification model extracts questions from the generation database that have a high similarity to the specified question (for example, the cosine similarity of the vector data is above a threshold), and outputs the input answer for that question as a reference answer.

[0150] The answer generation unit 118 may extract input answers to similar questions as reference answers based on the specified question category (specified question category). For example, the answer generation unit 118 extracts reference answers from input answers to questions that have been assigned the same or similar question category as the specified question category. Here, the specified question category may be specified based on input by a user such as the person being evaluated, or the question category assignment model may be output by inputting the specified question into the question category assignment model described above, causing the question category assignment model to output a question category corresponding to the specified question. This improves the accuracy of matching the specified question with the answered question or reference answer. Note that the extraction of reference answers based on the question category may be performed in conjunction with the extraction of reference answers based on the specified category that the specification reception unit 117 has received.

[0151] The response generation unit 118 may extract the input answers for each of the multiple answered questions as reference answers if the generation database contains multiple answered questions similar to the specified question. Alternatively, the response generation unit 118 may extract only the input answer for the answered question with the highest similarity among multiple answered questions similar to the specified question as a reference answer.

[0152] The answer generation unit 118 generates an answer to the specified question by arranging the extracted reference answer to match the content of the specified question using the reference information for answer generation. Arrangements to the reference answer include, for example, replacing words in the reference answer with synonyms, superordinate terms, subordinate terms, etc., organizing the context (such as rearranging the order of sentences), deleting unnecessary information, and adding missing information.

[0153] The reference information for generating answers includes the correlation between the reference answers and the answers to the specified questions. The reference information for generating answers is stored, for example, in the memory unit 12. The reference information for generating answers is an estimator constructed to take the reference answers as input and output answers to the specified questions. The reference information for generating answers may also include, for example, a table, a function, a simple algorithm, etc., that shows the correlation between the features (vector data) extracted from the reference answers and the features (vector data) extracted from the specified questions. The correlations included in the reference information for generating answers can be constructed, for example, by statistically analyzing data that records the questions related to the answers generated based on the reference answers.

[0154] The reference information for answer generation may include an answer generation model included in the artificial intelligence unit 120, which has been trained to take a reference answer as input and output an answer to a specified question. In this case, the answer generation unit 118 inputs the reference answer into the answer generation model and causes the answer generation model to output an answer to the specified question. This makes it possible to generate examples of answers to a large number of questions, or to generate answers based on natural language processing.

[0155] The answer generation model may be a learning model that has learned using training data consisting of sets of reference answers and questions, and combinations of the reference answers and the answers corresponding to those questions. In the answer generation model, the parameters calculated and tuned through learning constitute the correlation of the reference information for answer generation.

[0156] Furthermore, the answer generation model may be a generative AI that includes a large-scale language model. In this case, the answer generation unit 118 takes the reference answer and the specified question as input, inputs an instruction to the answer generation model to output an answer to the specified question, and causes the answer generation model to output the answer. The answer generation unit 118 may also generate a prompt that gives the answer generation model an instruction to generate an answer to the specified question based on the reference answer, and input this prompt to the answer generation model. In addition to the answer generation and output instruction, the reference answer and the specified question, the answer generation unit 118 may also input a prompt to the answer generation model that includes, for example, one or more sample reference answers and specified questions and one or more sample answers corresponding to them, as bare examples, samples, or training data for input and output. The answer generation model generates an answer that references the reference answer according to the input prompt.

[0157] Furthermore, the answer generation unit 118 outputs information indicating that it was not possible to generate an answer for specified questions from which no reference answers were extracted (i.e., specified questions for which no similar answered answers existed in the generation database).

[0158] When the designated reception unit 117 receives a designation of an answer format, the answer generation unit 118 may generate an answer to the designated question according to that answer format. For example, if a single-choice format of "yes" and "no" is specified, the answer generation unit 118 will generate either "yes" or "no" as the answer. Also, for example, if a free-response format is specified, the answer generation unit 118 will generate the freely written text as the answer. The answer generation unit 118 may, for example, insert an instruction to generate an answer in the specified answer format into the prompt for the answer generation model.

[0159] The answer generation unit 118 may accept edits to the answers it has generated for the questions. Furthermore, if the answer generation unit 118 is unable to generate an answer to a specified question, it may inform the user terminal 20 that an answer to that specified question could not be generated and may accept input for an answer to that specified question. The answer generation unit 118 may also inform the user terminal 20 of the reason why an answer could not be generated (e.g., missing information). For example, the answer generation unit 118 may insert an instruction to output the reason why an answer could not be generated in the prompts to the answer generation model.

[0160] The response generation unit 118 may display links on the user terminal 20, associated with the generated response, to show the content of the response data (specifically, the input response) that served as the basis for the generated response, the reason for referring to the said response data, etc. For example, the response generation unit 118 may display the reference response and the input response corresponding to the reference response (or links to them) on the user terminal 20 as the basis for generating the response. For example, the response generation unit 118 may insert instructions to output the reference response and the input response corresponding to the reference response (or links to them) in addition to the generated response in the prompt to the response generation model.

[0161] The response generation unit 118 determines whether the response generated by the response generation model contains information that should not be made public, and if it does, it may display a message to that effect on the target user's terminal 20. "Information that should not be made public" includes, for example, personal information, confidential information, and links to confidential data such as company data. Examples of personal information include the names and telephone numbers of employees of the person being evaluated. Examples of confidential information include, for example, internal information of the person being evaluated that is written in items (fields) such as "Notes" and "Supplementary Information". Examples of links to confidential data include links issued by external storage services, messaging services, etc., used by the information processing system 1 (links that allow access to files, posts, etc.). For example, the response generation unit 118 inserts an instruction into the prompt to the response generation model to generate a response that does not include personal information, confidential information, links to confidential data, or other information that should not be made public. Furthermore, the response generation unit 118 may determine whether the input responses used for response generation contain "information that should not be made public," and may choose not to use input responses containing "information that should not be made public" for response generation (for example, not to input them into the response generation model).

[0162] The response generation unit 118 may generate answers to the questions included in the input form presented by the response registration unit 112. This allows for the automatic generation of answers necessary for evaluation information from the response history held by the person being evaluated, thereby reducing the effort required for the person being evaluated to create the evaluation information. In particular, convenience is improved for people being evaluated who are using the information processing system 1 (for example, the security evaluation system) for the first time.

[0163] When generating responses to an input form for registering evaluation information, as described above, when the input form is called, or after the input form is called, the category (service, plan, etc.) is entered by the person being evaluated. This specifies the category for extracting reference responses, and the answers to the questions included in the input form are automatically generated. Therefore, the person being evaluated does not need to enter the questions individually.

[0164] When the data acquisition unit 114 receives input of answered questions and entered answers from the person being evaluated, the response generation unit 118 may, at a predetermined timing (for example, when the data acquisition unit 114 generates response data based on the entered information), pre-generate answers to questions that the person being evaluated has not yet answered for the input form of the category where the response data is stored. In other words, the response generation unit 118 may create some or all of the evaluation information in the background. This significantly reduces the effort required of the person being evaluated to create evaluation information from the input form, thereby encouraging the person being evaluated to use the input form.

[0165] Figure 12 shows an example of the answer display screen AD displayed on the user terminal 20. The answer display screen AD includes a designated question display field QC2, a generated answer display field AC2, a remarks field RC, and an answer output button B71. The designated question entered in the question input screen QD in Figure 11 is displayed in the designated question display field QC2.

[0166] In the generated answer display area AC2, the answers generated by the answer generation unit 118 are displayed, associated with the specified questions (on the same line). The answer generation unit 118 may accept edits to the answers by the person being evaluated in the generated answer display area AC2. In addition, for specified questions for which an answer could not be generated, an error message indicating this is displayed. Figure 12 illustrates error message EM1, which indicates that an answer could not be generated along with the reason, and error message EM2, which is a standard phrase indicating that an answer could not be generated for some reason. Furthermore, for specified questions for which error message EM2 is displayed, a regeneration instruction object RO, which accepts instructions to retry generating the answer, is displayed in the generated answer display area AC2.

[0167] The remarks column RC displays the objects that accept actions. These objects include, for example, the data addition acceptance object AO and the data display acceptance object DO. The data addition acceptance object AO accepts input to display a screen (described later) for adding the generated response data to the generation database. The data display acceptance object DO accepts input to display a screen (described later) for displaying the basis for the generated response data.

[0168] When the answer output button B71 is pressed, the answer generation unit 118 outputs the question and answer combination data (matrix) displayed in the specified question display field QC2 and the generated answer display field AC2 as CSV data. The person being evaluated can also copy individual answers from the generated answer display field AC2 and paste them to any location (for example, a file on the person's terminal 20 or an input field in another service).

[0169] Figure 13 shows an example of the response data registration screen RD displayed on the user terminal 20. The response data registration screen RD is displayed, for example, when an input operation is performed on the data addition acceptance object AO in Figure 12. The response data registration screen RD displays the basis message BM, the question display field QF, the answer display field AF, the memo display field MF, the category display field CF, the add button B81, and the cancel button B82.

[0170] The rationale message BM is text that indicates the reason why an answer could not be generated. For designated questions for which an answer was generated, the rationale message BM is not displayed. The question display field QF displays the content of the designated question. The question display field QF also accepts editing of the designated question. The answer display field AF displays the content of the generated answer. If an answer could not be generated as shown in Figure 13, the answer display field AF is left blank. The answer display field AF also accepts editing or input of the answer. The memo display field MF accepts comments from the person being evaluated. The category display field CF displays the selected category (service name). The category display field CF also accepts editing (selection) of the category.

[0171] When an input operation is performed on the Add button B81, the contents entered in the Question display field QF, Answer display field AF, Memo display field MF, and Category display field CF are registered as answer data in the generation database. When an input operation is performed on the Cancel button B82, the information entered in the Answer Data Registration screen RD is discarded, and the Answer Data Registration screen RD is closed.

[0172] Figure 14 shows an example of the evidence display screen BD displayed on the user terminal 20. The evidence display screen BD is displayed, for example, when an input operation is performed on the data display reception object DO in Figure 12. The evidence display screen BD includes an evidence display area BA, a reference data display area RA, and a close button B91.

[0173] The evidence display area BA shows the combination of the selected specified question and the generated answer, as well as the rationale for generating the answer (the reason for referencing the answer data). The reference data display area RA shows the answer data (answered questions and entered answers) that were referenced when generating the answer. When an input operation is performed on the close button B91, the evidence display screen BD is closed.

[0174] <Report Output Section 119> The report output unit 119 is configured to output a report or document created based on the evaluation received by the evaluation reception unit 113 to the requester terminal 40. The report or document includes at least all or part of the evaluation information entered by the person being evaluated, and the evaluation results by the evaluator (e.g., evaluation values, comments, etc.).

[0175] <Artificial Intelligence Department 120> The artificial intelligence unit 120 is configured to receive input from each functional unit and return the instructed output. The artificial intelligence used by each functional unit of the server device 10 may be common to all units, or it may be prepared individually for each functional unit.

[0176] The artificial intelligence unit 120 is an AI (Artificial Intelligence) equipped with learning models such as transformers including GPT (Generative Pretrained Transformer, including GPT-1, GPT-2, GPT-3, and GPT-4), BERT (Bidirectional Encoder Representations from Transformers), BART (Bidirectional and Auto-regressive Transformer), and language models such as recurrent neural networks (RNNs), and may include generative AI including large-scale language models. Large-scale language models are a type of generative AI and include models provided by services such as OpenAI's GPT, Google's Gemini, and Microsoft's Azure AI Studio. In addition, the artificial intelligence unit 120 can include any machine learning model, deep learning model, artificial intelligence model, etc.

[0177] The language model is an example of a learning model using a machine learning algorithm. Specific machine learning algorithms include nearest neighbors, naive Bayes, decision trees, support vector machines, and deep learning using neural networks. The artificial intelligence unit 120 can apply the above algorithms as appropriate.

[0178] The artificial intelligence unit 120 may have a trained model constructed by a learning method such as supervised learning, unsupervised learning, or self-supervised learning. In supervised learning, machine learning is performed using training data. Training data consists of pairs of input data and output data (correct answer data) for training. Furthermore, the language model may not only be one trained for a specific task, but also a general-purpose model that can be used universally for a wide range of tasks.

[0179] The artificial intelligence unit 120 may be a general-purpose natural language processing learning model, such as a Large Language Model (LLM), which has learned from a vast amount of data. An LLM is a learning model that has been pre-trained on a large amount of data consisting of text data, etc. (for example, (i) web content on the internet, or (ii) data stored in a predetermined database), and can perform various language processing tasks by being given a task. It can perform a wide range of natural language processing tasks, such as understanding sentence patterns and context, responding to questions, and generating sentences, according to the given prompts. Such a general-purpose learning model includes language models that can handle various tasks without fine-tuning using One-shot Learning or Few-shot Learning. Furthermore, the general-purpose learning model may also be configured to handle various tasks using Zero-shot Learning. The artificial intelligence used in each functional unit of the control unit 11 may be a separate learning model, or it may be a common general-purpose learning model.

[0180] The learning models included in the artificial intelligence unit 120 (such as integrated processing models and partitioned processing models used in each functional unit) can undergo additional learning through transfer learning or fine-tuning. For example, the artificial intelligence unit 120 may perform additional learning and fine-tuning each time new data is registered, using it as new training data. This improves the accuracy of the information output from the learning model.

[0181] The learning model included in the artificial intelligence unit 120 may be a learning model (distilled model) obtained by knowledge distillation using the original learning model. In knowledge distillation, a pre-trained model, such as a large-scale language model, is used as the teacher model, and the parameters of the student model are adjusted so that the output loss of the student model (distilled model) relative to the output (Soft Target Loss) of the teacher model is small. The student model is then trained, and this student model becomes the distilled model. Alternatively, the student model may be trained so that the output loss of the student model relative to the correct labels (Hard Target) of the teacher data (combinations of input and output data of the learning model) is small. Compared to the original learning model (teacher model), the distilled model has similar performance to the original learning model, but with fewer parameters and a lower processing load. Therefore, using a distilled model can reduce the cost of the information processing system 1.

[0182] For example, the learning model used in each functional unit may be a distilled model that has been trained using combinations of input and output data from a large-scale language model as training data. Alternatively, when the information processing system 1 is introduced, a large-scale language model may be used as the learning model in each functional unit, and once training data from the large-scale language model has been accumulated, the distilled model obtained by knowledge distillation using that training data may be used as the learning model in each functional unit.

[0183] <Display section> The display unit 211 of the subject terminal 20, the display unit 311 of the evaluator terminal 30, and the display unit 411 of the requester terminal 40 each display the screen indicated by the screen data transmitted from the server device 10.

[0184] <Operation acquisition section> The operation acquisition unit 212 of the target terminal 20 receives operations from the person being evaluated using the target terminal 20. The operation acquisition unit 312 of the evaluator terminal 30 receives operations from the evaluator using the evaluator terminal 30. The operation acquisition unit 412 of the requester terminal 40 receives operations from the requester using the requester terminal 40.

[0185] 3. Information Processing Methods This section describes the information processing method of the server device 10. In this information processing method, each part of the server device 10 is executed by a computer as a step.

[0186] This information processing comprises a data acquisition step, a data organization step, a data registration step, a designation acceptance step, and an answer generation step. In the data acquisition step, answer data is acquired by combining the security-related questions answered by the person being evaluated and the answers entered by the person being evaluated to those questions. In the data organization step, at least one of the following is performed: an integration process that combines two or more duplicate answer data into one answer data, or a splitting process that splits answer data into multiple answer data if the answered question can be divided into multiple questions, or if the entered answer can be divided into multiple answers. In the data registration step, the answer data obtained by the integration or splitting process is registered in the database. In the designation acceptance step, the designation of security-related questions is accepted. In the answer generation step, from the entered answers included in the answer data registered in the database, entered answers to answered questions similar to the designated question are extracted as reference answers, and answers to the questions are generated based on these reference answers and reference information for answer generation.

[0187] Figure 15 is an activity diagram showing an example of the flow of information processing (answer generation processing) performed by information processing system 1. The information processing will be explained below in accordance with each activity in this activity diagram.

[0188] The response generation process begins with the input of response data by the person being evaluated. The person being evaluated inputs response data on the person terminal 20 by either entering previously answered questions and answers, or by selecting registered evaluation information (Activity A101). The server device 10 acquires the response data entered from the person terminal 20 (Activity A102). Subsequently, the server device 10 performs at least one of the integration process and / or splitting process on the acquired response data (Activity A103). Furthermore, the server device 10 adds the response data obtained through the integration or splitting process to the generation database (Activity A104). Activities A101 through A104 can be performed at any time as appropriate.

[0189] With the response data already generated, the person being evaluated specifies a question to generate the response on the participant terminal 20 (Activity A201). The question may be specified by directly entering the question on the participant terminal 20, or by calling an input form for creating evaluation information. Based on the specified question, the server device 10 extracts reference answers from the generation database (Activity A202). After extracting the reference answers, the server device 10 generates the response to the specified question by referring to the reference answers (Activity A203). Subsequently, the server device 10 outputs the generated response to the participant terminal 20 (Activity A204). As a result, the response generated by the server device 10 is displayed on the participant terminal 20 (Activity A205).

[0190] 4. Effect The operation of this embodiment can be summarized as follows: Since the response data, which is a combination of past questions and answers, is organized, the person being evaluated can efficiently answer security-related questions by referring to this response data.

[0191] Although embodiments of the present invention have been described above, the present invention is not limited thereto and can be modified as appropriate without departing from the technical spirit of the invention.

[0192] 5. Others In the above embodiment, the server device 10 performed various storage and control functions, but instead of the server device 10, multiple external devices may be used. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like. In particular, the artificial intelligence unit 120 may be an external configuration of the server device 10. In that case, the external artificial intelligence unit 120 may be provided by, for example, an artificial intelligence service server, and is configured to receive input from each functional unit of the server device 10, receive requests to execute artificial intelligence services, and return the instructed output as a processing result to the server device 10. The artificial intelligence service server may be a server that provides services using a language model as a learning model, or a server that executes language processing tasks using a language model. The artificial intelligence service server may be constructed using an LLM. The artificial intelligence service server receives prompt input in the form of text, images, audio, etc., and generates and responds with answers to the prompts.

[0193] The control unit 11 does not necessarily have to include an answer generation unit 118. In other words, the information processing system 1 does not necessarily have to generate answers based on answer data obtained by integrated processing or partitioning processing.

[0194] The embodiments of this model are not limited to the information processing system 1, but may also be an information processing method or a program. The information processing method comprises each step executed by the information processing system 1. The program causes a computer to execute each step of the information processing system 1.

[0195] The product may be provided in any of the following embodiments.

[0196] (1) An information processing system comprising at least one processor, wherein the processor is configured to perform the following steps by reading a program, the data acquisition step of acquiring response data which is a combination of the security questions of the person being evaluated and the answers that the person being evaluated has entered to the answered questions, and the data organization step of performing at least one of the following: an integration process which combines two or more duplicate response data into one response data, and a splitting process which splits the response data into multiple response data if the answered questions can be divided into multiple questions, or if the entered answers can be divided into multiple answers.

[0197] (2) Information processing system as described in (1) above, wherein the processor is configured to further perform the following steps: in the data registration step, register the answer data obtained by the integration process or the division process in a database; in the designation acceptance step, accept the designation of a security question; and in the answer generation step, extract from the input answers included in the answer data registered in the database that are similar to the designated question and are input answers for the answered questions, and generate an answer to the question based on the reference answers and the answer generation reference information, wherein the answer generation reference information includes the correlation between the reference answers and the answers to the questions.

[0198] (3) An information processing system as described in (2) above, wherein the reference information for generating answers includes an answer generation model that has been trained to take the reference answers as input and output an answer to the question, and in the answer generation step, the reference answers are input to the answer generation model and the answer generation model is made to output an answer to the question.

[0199] (4) An information processing system as described in (2) or (3) above, wherein in the data organization step, the system accepts editing of the response data obtained by the integration process or the division process, and in the data registration step, the system registers the edited response data in the database.

[0200] (5) An information processing system according to any one of (1) to (4) above, wherein in the integration process, the information processing system determines the response data to be integrated based on the similarity based on the comparison of feature quantities of the answered questions included in the response data, the similarity based on the comparison of feature quantities of the input answers included in the response data, or a combination of these similarities.

[0201] (6) An information processing system according to any one of (1) to (5) above, wherein in the integration process, multiple response data are input to an integration processing model, and the integration processing model outputs response data that integrates the duplicate response data, wherein the integration processing model is a learning model that has been trained to take multiple response data as input and output response data that integrates the duplicate response data.

[0202] (7) An information processing system according to any one of (1) to (6) above, wherein in the division process, the answer data is input to a division process model, and the division process model is made to output a plurality of the answer data obtained by dividing the answer data, wherein the division process model is a learning model that has been trained to take the answer data as input and output a plurality of the answer data obtained by dividing the answer data into a plurality of combinations of the answered questions and the input answers.

[0203] (8) An information processing system as described in (7) above, wherein the partitioning processing model is a large-scale language model, and in the partitioning processing, an instruction is input to the partitioning processing model to partition the response data into a plurality of combinations of the answered questions and the entered answers.

[0204] (9) An information processing system as described in (8) above, wherein in the division process, with respect to the answer data in which the answered question contains a phrase representing a conditional branch, logical OR, or logical AND, the system inputs an instruction to the division process model to divide the answered question based on the phrase, and further divide the input answer according to the divided answered question.

[0205] (10) An information processing system as described in (8) or (9) above, wherein in the division process, the system inputs an instruction to the division process model to divide the input response into each of the options, and further combine each of the divided input responses with the original answer question.

[0206] (11) An information processing system according to any one of (8) to (10) above, wherein in the division process, the answer data, which includes an answered question containing multiple question items and an input answer consisting of a single answer to the multiple question items, is input to the division process model, which in turn inputs an instruction to divide the answered question into question items and to combine each of the divided answered questions with the original input answer.

[0207] (12) An information processing system according to any one of (1) to (11) above, wherein in the data organization step, the integration process and the division process are performed in parallel on the acquired response data, or the integration process is performed on the response data after the division process has been performed.

[0208] (13) An information processing method comprising each step performed by the information processing system described in any one of (1) to (12) above.

[0209] (14) A program that causes a computer to perform each step of the information processing system described in any one of (1) to (12) above. Of course, this is not always the case.

[0210] Finally, while various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0211] 1: Information Processing System 2: Communication lines 10: Server device 11: Control Unit 12: Storage section 13: Communications Department 14: Communications bus 20: Target user's device 21: Control Unit 22: Storage section 23: Communications Department 24: Input section 25: Output section 26: Communications bus 30: Evaluator terminal 31: Control Unit 32: Storage section 33: Communications Department 34: Input section 35: Output section 36: Communications bus 40: Client terminal 41: Control Unit 42: Storage section 43: Communications Department 44: Input section 45: Output section 46: Communications bus 111: Basic Display Control Unit 112: Answer Registration Section 113: Evaluation Reception Department 114: Data Acquisition Unit 115: Data Organization Department 116: Data Registration Department 117: Designated Reception Department 118:Answer generation section 119: Report Output Section 120: Artificial Intelligence Department 211:Display section 212: Operation acquisition section 311: Display section 312: Operation acquisition section 411: Display section 412: Operation acquisition section AC1: Answer input field AC2: Generated answer display field AD:Answer display screen AF: Answer display field AO: Data Addition Acceptance Object B11: Register button B21: Add Knowledge Button B22: Answer generation button B23: Knowledge Edit Button B24: Knowledge Organization Button B31: Add Integrated Data Button B32: Delete Integrated Data Button B41: Add Split Data Button B42: Delete Split Data Button B43: Delete original data button B51: Create button B52: Service selection button B61: Cancel button B62: Select button B71: Answer Output Button B81: Add button B82: Cancel button B91: Close button BA: Supporting information area BD: Evidence display screen BM: Rational message CC: Registration check box CF: Category display field DA: split answer DD: Data entry screen DF: Display field for divisible data DO: Data display receiving object DQ: split question EF: Evaluation information display field EM1: Error message EM2: Error message MC: Memo section MD: Response Data Management Screen MF: Memo display field NF: Knowledge display field NL: Knowledge List OD: Response Data Organization Screen OF: Duplicate data display field PC: Past data comparison section QC1: Question input field QC2:Specified question display field QD: Question input screen QF:Question display field QI:Question specification field RA: Reference data display area RC:Remarks column RD: Response data registration screen RO: Regeneration instruction object SB: Select button SC: Service selection field SD: Service selection screen SO: Selected Objects TB: Switch tab TF: Condition setting field UA: Integrated answer UQ: Integrated Questions WA: Work Area

Claims

1. An information processing system, Equipped with at least one processor, The aforementioned processor is configured to perform the following steps by reading a program: In the data acquisition step, response data is acquired by combining the security-related questions already answered by the person being evaluated and the answers entered by the person being evaluated to those questions. An information processing system that, in the data organization step, performs at least one of the following: an integration process that combines two or more duplicate response data into one response data; and a splitting process that splits response data into multiple response data if the answered question can be divided into multiple questions, or if the entered answer can be divided into multiple answers.

2. In the information processing system described in claim 1, The aforementioned processor is configured to perform the following steps: In the data registration step, the response data obtained by the integration process or the division process is registered in the database. In the designated reception step, the designated questions regarding the aforementioned security are accepted. In the response generation step, the input responses included in the response data registered in the database that are similar to the specified question are extracted as reference responses, and a response to the question is generated based on these reference responses and the reference information for response generation. Here, the reference information for generating the answer includes the correlation between the reference answer and the answer to the question, in an information processing system.

3. In the information processing system described in claim 2, The aforementioned reference information for generating answers includes an answer generation model that has been trained to take the aforementioned reference answers as input and output an answer to the aforementioned question. An information processing system that, in the aforementioned answer generation step, inputs the reference answer into the answer generation model and causes the answer generation model to output an answer to the question.

4. In the information processing system described in claim 2, In the data organization step, the editing of the response data obtained by the integration process or the division process is accepted. The data registration step involves an information processing system that registers the edited response data into the database.

5. In the information processing system described in claim 1, An information processing system that determines the response data to be integrated in the integration process based on the similarity of the feature quantities of the answered questions included in the response data, the similarity of the feature quantities of the input answers included in the response data, or a combination of these similarities.

6. In the information processing system described in claim 1, In the aforementioned integration process, multiple response data are input to the integration processing model, and the integration processing model outputs response data that integrates the redundant response data. Here, the integrated processing model is an information processing system that is a learning model trained to take multiple response data as input and output response data that integrates the overlapping response data.

7. In the information processing system described in claim 1, In the aforementioned splitting process, the response data is input to the splitting process model, and the splitting process model outputs multiple response data obtained by splitting the response data. Herein, the partitioning processing model is an information processing system that is a learning model trained to take the answer data as input and output multiple sets of answer data obtained by partitioning the answer data into multiple combinations of the answered questions and the input answers.

8. In the information processing system described in claim 7, The aforementioned partitioning model is a large-scale language model, An information processing system that, in the division process, inputs an instruction to the division process model to divide the response data into a plurality of combinations of the answered questions and the entered answers.

9. In the information processing system described in claim 8, An information processing system that, in the division process, inputs instructions to the division process model to divide the answer data, which includes phrases representing conditional branching, logical OR, or logical AND in the answered questions, based on the phrases, and further divide the input answers according to the divided answered questions.

10. In the information processing system described in claim 8, The information processing system provides the following instructions to the division processing model for the division processing, which involves dividing the input response data, in which the input response is composed of multiple choices, into individual choices, and then combining each of the divided input responses with the original answer question.

11. In the information processing system described in claim 8, In the division process, the information processing system inputs instructions to the division process model to divide the answer data, which includes the answered questions containing multiple question items and the input answers consisting of a single answer to the multiple question items, into individual question items, and then combine each of the divided answered questions with the original input answers.

12. In the information processing system described in claim 1, An information processing system that, in the data organization step, performs the integration process and the splitting process on the acquired response data in parallel, or performs the integration process on the response data after the splitting process has been performed.

13. Information processing method, An information processing method comprising each step performed by the information processing system according to any one of claims 1 to 12.

14. It is a program, A program for causing a computer to perform each step of the information processing system described in any one of claims 1 to 12.

Images