Information processing system, information processing method and program

Abstract

An information processing system, method, and program for constructing an information search tool that can obtain high-quality, highly reliable answers is provided. [Solution] A processing method executed by an information processing system includes an acquisition step of acquiring a specific document, and a first generation step of generating evaluation data based on the specific document and preset first reference information. The evaluation data includes a question about specific content contained in the specific document and a model answer to the question obtained based on the specific content. Furthermore, a second generation step generates a practical answer to the question based on the question, a group of information, and preset second reference information, and an output step outputs an evaluation result based on the model answer, the practical answer, and preset third reference information. The evaluation result includes a result of evaluating the practical answer obtained using the model answer as a standard.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an information search system that searches for known information, and if specific information is not found, uses AI-based inference to retrieve the information.

[0003] This information retrieval system includes a means for searching and narrowing down known information, and a means for using AI inference to retrieve information when specific information cannot be found. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2025-049356 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is still room for improvement in the functions of the above-mentioned known techniques.

[0006] In view of the above circumstances, the present invention provides a technology that enables the construction of an information search tool that can obtain high-quality and highly reliable answers. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided an information processing system comprising at least one processor, the processor being configured to execute the following steps by reading a program: in an acquisition step, a specific document is acquired, the specific document being a document selected based on arbitrary conditions from a group of information including a plurality of documents; in a first generation step, evaluation data is generated based on the specific document and predetermined first reference information, the evaluation data including a question regarding specific content contained in the specific document and a model answer to the question obtained based on the specific content; in a second generation step, a practical answer to the question is generated based on the question, the group of information, and predetermined second reference information; and in an output step, an evaluation result is output based on the model answer, the practical answer, and predetermined third reference information, the evaluation result including a result of evaluating the practical answer obtained based on the model answer.

[0008] According to the present disclosure, an information search tool capable of obtaining high-quality and reliable answers can be constructed. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram illustrating an information processing system 1 according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of an information processing device 2. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of a terminal 3. [Figure 4] 1 is a functional block diagram showing functions of an information processing system 1 according to the present embodiment. [Figure 5] 3 is a flowchart showing an outline of processing executed by the information processing system 1. [Figure 6] 2 is an activity diagram showing a specific example of processing executed by the information processing system 1. FIG. [Figure 7] FIG. 2 is a schematic diagram showing an example of a setting screen 4a, which is the top screen of the setting screens 4. [Figure 8]10 is a schematic diagram showing an example of a setting screen 4b of the setting screen 4, which shows the result ER1 of evaluating the practical answer PA. FIG. [Figure 9] 10 is a schematic diagram showing an example of a setting screen 4c of the setting screen 4, which shows the result ER1 of evaluating the practical answer PA. FIG. [Figure 10] 10 is a schematic diagram showing an example of a setting screen 4d of the setting screen 4, which shows the result ER1 of evaluating the practical answer PA before and after the optimization process of the setting information IFs is executed. FIG. [Figure 11] FIG. 10 is a schematic diagram showing an example of a setting screen 4e of the setting screen 4, which shows the result ER1 of evaluating the practical answer PA after the optimization process of the setting information IFs of the information search application IR is executed. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Various features shown in the following embodiments can be combined with each other.

[0011] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0012] Furthermore, various information processing according to an embodiment may realize input and output corresponding to the input. Here, the form of information referenced in such information processing (hereinafter referred to as reference information) is not limited as long as an output is obtained as a result of the input. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression formula constructed using a statistical method), a trained model that has previously learned the correlation between input and output, or a generative AI such as a large-scale language model (these models include parameters that establish the correlation between input and output) or a visual language model that can output a desired result in response to a prompt.

[0013] In one embodiment, a "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In one embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations can be performed on a circuit in the broad sense.

[0014] Furthermore, a circuit in the broad sense is a circuit realized by at least an appropriate combination of a circuit, circuitry, processor, memory, etc. The processor may be a general-purpose processor or a dedicated circuit. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0015] 1. Hardware Configuration This section explains the hardware configuration.

[0016] 1.1 Information Processing System 1 FIG. 1 is a configuration diagram showing an information processing system 1 according to this embodiment. The information processing system 1 includes an information processing device 2 and a terminal 3, which are connected via a general-purpose or dedicated communication network 11. Here, a system exemplified as the information processing system 1 is made up of one or more devices or components. Therefore, even the information processing device 2 alone is included in the system exemplified as the information processing system 1. Each component included in the information processing system 1 will be further described below.

[0017] 1.2 Information processing device 2 2 is a block diagram showing the hardware configuration of the information processing device 2. The information processing device 2 is preferably configured as a server. The information processing device 2 has a communication unit 21, a storage unit 22, and a control unit 23, and these components are electrically connected via a communication bus 20 inside the information processing device 2. Each component will be further described below.

[0018] The communication unit 21 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, Bluetooth (registered trademark) communication, etc. as necessary. In other words, it is more preferable to implement it as a collection of multiple communication means. In other words, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the communication network 11.

[0019] The memory unit 22 stores various pieces of information defined above. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the information processing device 2 executed by the control unit 23, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program operations. The memory unit 22 stores various programs, variables, etc. related to the information processing device 2 executed by the control unit 23.

[0020] The control unit 23 processes and controls the overall operations related to the information processing device 2. The control unit 23 is, for example, a central processing unit (CPU) not shown. The control unit 23 realizes various functions related to the information processing device 2 by reading out predetermined programs stored in the storage unit 22. That is, information processing by software stored in the storage unit 22 is specifically realized by the control unit 23, which is an example of hardware, and can be executed as each functional unit included in the control unit 23. These will be described in more detail in the next section. Note that the control unit 23 is not limited to being single, and multiple control units 23 may be provided for each function. A combination of these may also be used. The information processing system 1 includes the control unit 23 as at least one processor. The control unit 23, which is a processor, is configured to execute each step described below by reading out a program.

[0021] 1.3 Terminal 3 3 is a block diagram showing the hardware configuration of terminal 3. Terminal 3 may be operated by a user and may take any form, such as a smartphone, tablet terminal, computer, or any other device that can access information processing device 2 via an electric communication line. Specifically, terminal 3 includes a communication unit 31, a storage unit 32, a control unit 33, a display unit 34, and an input unit 35, and these components are electrically connected via a communication bus 30 inside terminal 3. Description of communication unit 31, storage unit 32, and control unit 33 will be omitted as they are the same as the descriptions of the respective units in information processing device 2.

[0022] The display unit 34 may be, for example, included in the housing of the terminal 3 or may be externally attached. The display unit 34 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably implemented by selectively using display devices such as a CRT display, a liquid crystal display, an organic EL display, and a plasma display depending on the type of terminal 3.

[0023] The input unit 35 may be included in the housing of the terminal 3 or may be externally attached. For example, the input unit 35 may be implemented as a touch panel integrated with the display unit 34. A touch panel allows the user to input tapping, swiping, and the like. Of course, a switch button, a mouse, a QWERTY keyboard, or the like may be used instead of a touch panel. That is, the input unit 35 accepts an operation input made by the user. The input is transferred as a command signal to the control unit 33 via the communication bus 30, and the control unit 33 can execute predetermined control or calculation as necessary.

[0024] 2. Functional configuration This section describes the functional configuration of this embodiment. As described above, information processing by software stored in the storage unit 22 is specifically realized by the control unit 23, which is an example of hardware, and each functional unit included in the control unit 23 can be executed.

[0025] 4 is a functional block diagram showing functions of the information processing system 1 according to this embodiment. Specifically, the information processing device 2, which is an example of the information processing system 1, includes a reception unit 231, an acquisition unit 232, a calculation unit 233, a generation unit 234, a change unit 235, an optimization unit 236, an output unit 237, and a display control unit 238.

[0026] The reception unit 231 is configured to receive various pieces of information as a reception step. For example, the reception unit 231 receives data, instructions input by a user, and the like from the storage unit 22, or from the input unit 35 of the terminal 3 or other external devices via the communication network 11. In one embodiment, the description will be given assuming that the various pieces of information received by the reception unit 231 are stored in the storage unit 22.

[0027] The acquisition unit 232 is configured to acquire various pieces of information as an acquisition step. For example, the acquisition unit 232 acquires any information from the storage unit 22, or from the input unit 35 of the terminal 3 or another external device via the communication network 11. In one embodiment, the description will be given assuming that the various pieces of information acquired by the acquisition unit 232 are stored in the storage unit 22.

[0028] The calculation unit 233 is configured to execute various information processing calculations related to the information processing device 2. Details will be described later.

[0029] The generating unit 234 is configured to generate arbitrary information based on various information as a first generating step or a second generating step, as will be described in detail later.

[0030] The change unit 235 is configured to change the setting information IFs by an arbitrary algorithm as a change step, as will be described in detail later.

[0031] The optimization unit 236 is configured to execute an optimization process in response to a predetermined instruction received as an optimization step, which will be described in detail later.

[0032] The output unit 237 is configured to output various information as an output step, which will be described in detail later.

[0033] The display control unit 238 is configured to execute various display processes. For example, the display control unit 238 controls the display unit 34 of the terminal 3 to display visually recognizable information such as screens, images including still images or moving images, icons, messages, etc. The display control unit 238 may generate only rendering information for displaying visually recognizable information on the display unit 34 of the terminal 3. Details will be described later.

[0034] 3. Operation of Information Processing System 1 In this section, the information processing method of the information processing system 1 described above will be described with reference to the drawings. The order of the processes can be changed as appropriate, multiple processes can be executed simultaneously, or some processes can be omitted. The results of the processes described below can be output in a manner that can be recognized by the user, using the display unit 34 of the terminal 3, as appropriate. The information processing can include any exception processing not shown. Exception processing includes the interruption of the information processing or the omission of each process. The selection or input performed in the information processing can be based on a user operation or can be performed automatically without relying on a user operation.

[0035] 3.1 Overview FIG. 5 is a flowchart illustrating an outline of a process executed by the information processing system 1. In this process, the acquisition unit 232 first acquires a specific document Dc1 (step S001). The specific document Dc1 is a document Dc selected based on an arbitrary condition from an information group IG containing multiple documents Dc. Next, the generation unit 234 generates evaluation data ED based on the specific document Dc1 and preset first reference information RI1 (step S002). The evaluation data ED includes a question ED1 related to specific content Cn1 contained in the specific document Dc1 and a model answer ED2 to the question ED1 obtained based on the specific content Cn1. Next, the generation unit 234 generates a practical answer PA to the question ED1 based on the question ED1, the information group IG, and preset second reference information RI2 (step S003). Then, the output unit 237 outputs the evaluation result ER based on the model answer ED2, the practical answer PA, and the third reference information RI3 (step S004). Here, the evaluation result ER includes a result ER1 of evaluating the practical answer PA based on the model answer ED2.

[0036] To summarize, the information processing system 1 according to one embodiment includes at least one processor. The processor is configured to execute the following steps by reading a program. The acquisition unit 232 acquires a specific document Dc1. The specific document Dc1 is a document Dc selected based on an arbitrary condition from an information group IG including multiple documents Dc. In a first generation step, the generation unit 234 generates evaluation data ED based on the specific document Dc1 and preset first reference information RI1. The evaluation data ED includes a question ED1 related to specific content Cn1 included in the specific document Dc1 and a model answer ED2 to the question ED1 obtained based on the specific content Cn1. In a second generation step, the generation unit 234 generates a practical answer PA to the question ED1 based on the question ED1, the information group IG, and preset second reference information RI2. The output unit 237 outputs an evaluation result ER based on the model answer ED2, the practical answer PA, and preset third reference information RI3. The evaluation result ER includes a result ER1 of evaluating the practical answer PA, obtained based on the model answer ED2. According to this embodiment, it is possible to improve the accuracy of the practical answer PA using the obtained evaluation result ER, and as a result, it is possible to build an information retrieval tool that can obtain high-quality and highly reliable answers.

[0037] 3.2 Specific examples Next, a specific example of processing according to an embodiment will be described with reference to an activity diagram. The specific example may fall within the scope defined in the above-described overview. FIG. 6 is an activity diagram showing a specific example of processing executed by the information processing system 1. Below, the flow of information processing will be outlined with reference to this activity diagram. As an example, the following describes a situation in which a system administrator (hereinafter simply referred to as "administrator") tunes the setting information IFs of an information retrieval application IR (an example of an information retrieval tool) used in organization ABC. Specifically, the following describes a situation in which the information retrieval application IR is tuned by executing processing to optimize the setting information IFs from a state in which the information retrieval application IR is not tuned.

[0038] In this case, the information search application IR is an application that allows an employee, etc. belonging to organization ABC to input information such as text, voice, image, etc., and obtain a response from reference information RI based on information stored in organization ABC's database (an example of information group IG). This allows the employee, etc., to obtain a response based on the information in the database. Note that the reference information RI in this case may be a trained model that has previously learned the correlation between input and output, and may be a generation AI such as a large-scale language model or visual language model that can output a desired result by inputting a prompt.

[0039] First, the administrator uses a browser on the terminal 3 to access the information retrieval application IR. The information retrieval application IR may be provided as a dedicated desktop or mobile application, in which case the administrator launches the application. The information retrieval application IR may also be provided as a web application or SaaS format that does not require downloading to the terminal 3. In response to this, the display control unit 238 controls the display unit 34 of the terminal 3 to display a login screen (not shown).

[0040] Then, the reception unit 231 receives input of login information of the administrator through operations by the administrator via the login screen (for example, a click operation, a tap operation, a swipe operation, a selection operation, etc.). When the login to the information search application IR is completed, the display control unit 238 controls the display unit 34 of the terminal 3 to display the setting screen 4 (see FIGS. 7 to 11) (activity A001).

[0041] Next, the receiving unit 231 receives an instruction Ins to perform optimization based on an operation by the administrator via the setting screen 4 (activity A002). Specifically, for example, the receiving unit 231 receives the instruction Ins as an instruction to perform tuning of the setting information IFs of the information retrieval application IR, through an operation by the administrator via the setting screen 4. Upon receiving the instruction Ins, the optimization unit 236 executes optimization processing. In this case, the optimization processing sequentially executes processes for preparing the optimization processing (activities A003 to A004) and processes for executing the optimization processing (activities A005 to A008), and when the optimization processing is completed, the results of the optimization processing are displayed (activity A009). Details of these processes will be described below.

[0042] First, the acquiring unit 232 acquires a specific document Dc1 (activity A003). In this case, the specific document Dc1 is a document Dc selected based on an arbitrary condition from an information group IG that includes a plurality of documents Dc.

[0043] Here, the information group IG is a collection of information including multiple documents Dc that is systematically stored in the memory unit 22 of the information processing device 2 or other external device, for example, a collection of information that is stored in the database of organization ABC and includes multiple documents Dc.

[0044] Furthermore, the arbitrary conditions are not particularly limited as long as they are conditions that allow an arbitrary document Dc to be selected from multiple documents Dc, and may be, for example, conditions that are frequently viewed, that the creation date or update date is an arbitrary day, that the content is arbitrary, that the document is selected by random sampling, or other combinations of these.

[0045] When the acquiring unit 232 acquires the specific document Dc1, the generating unit 234 generates evaluation data ED based on the specific document Dc1 and preset first reference information RI1 as a first generating step (activity A004).

[0046] Here, the first reference information RI1 is an example of reference information RI, and includes at least a correlation between a specific document Dc1 and evaluation data ED, and may be, for example, a trained model or generation AI that has been trained to be able to take an arbitrary document Dc as input and output a question ED1 and a model answer ED2 based on the document Dc.

[0047] Furthermore, the specific content Cn1 contained in the specific document Dc1 is the content of all or part of the specific document Dc1, and is not particularly limited as long as it is information that can be used to generate a question ED1 and a model answer ED2 as evaluation data ED, such as content contained in the company rules of organization ABC or content contained in documents related to the business of organization ABC. The format of the specific content Cn1 is also not particularly limited, and may be, for example, text information, image information, or text and image information.

[0048] The generated evaluation data ED includes a question ED1 regarding specific content Cn1 contained in the specific document Dc1, and a model answer ED2 to the question ED1 obtained based on the specific content Cn1. Specifically, for example, the evaluation data ED includes a question (an example of question ED1) regarding holidays, etc., and a model answer (an example of model answer ED2) to the question, based on the content contained in the work rules of organization ABC (an example of specific content Cn1 contained in the specific document Dc1).

[0049] Next, as a second generation step, the generation unit 234 generates a practical answer PA to the question ED1 based on the question ED1, the information group IG, and the second reference information RI2 that has been set in advance (activity A005). In this case, the practical answer PA includes information that can identify the information searched from the information group IG that was used when generating the practical answer PA.

[0050] Here, the second reference information RI2 is an example of reference information RI, and includes at least the correlation between the question ED1, the information group IG, and the practical answer PA, and may be, for example, a trained model or generation AI that has been trained to be able to take the question ED1 as input, extract the supporting document Dc from the information group IG, and output the practical answer PA.

[0051] The second reference information RI2 also includes arbitrarily configurable setting information IFs. The setting information IFs is information on settings related to the generation of various types of information, and has multiple parameters P used when extracting arbitrary information from the information group IG. The parameters P include, for example, chunk length and the amount of overlap between chunks, and adjusting each parameter P adjusts the accuracy with which arbitrary information is extracted from the information group IG. In other words, adjusting each parameter P and tuning the setting information IFs improves the accuracy with which arbitrary information is extracted from the information group IG.

[0052] Furthermore, because the practical answer PA is an answer to the question ED1, it is desirable that the content be similar to the model answer ED2 obtained based on the specific content Cn1 of the specific document Dc1. However, by including a process for extracting the specific content Cn1 of the specific document Dc1 from the information group IG, there is a possibility that the information used to generate the practical answer PA will differ depending on the accuracy of the process. If a difference occurs in the information on which the answer is based, the content of the practical answer PA may differ from the content of the model answer ED2.

[0053] Next, the output unit 237 outputs the evaluation result ER based on the model answer ED2, the practical answer PA, and the third reference information RI3 (activity A006). In this case, the evaluation result ER includes the result ER1 of evaluating the practical answer PA based on the model answer ED2, and may also include the setting information IFs used when the model answer ED2 was generated, the evaluation data ED, etc.

[0054] Here, the third reference information RI3 is an example of reference information RI, and includes at least a correlation between the model answer ED2, the practical answer PA, and the evaluation result ER, and may be, for example, a trained model or generation AI that has been trained to be able to input the model answer ED2 and the practical answer PA and output the evaluation result ER.

[0055] Furthermore, the evaluation result ER1 of the practical answer PA may be any result that allows evaluation of the accuracy of the generated practical answer PA, and may be, for example, a result using a preset level of accuracy class, and may include a score based on the model answer ED2. Specifically, for example, the score based on the model answer ED2 may be calculated based on the proportion of specific content Cn1 contained in the practical answer PA, the degree of agreement between the model answer ED2 and the content of the practical answer PA, etc. According to this aspect, the practical answer PA can be evaluated using the score based on the model answer ED2.

[0056] The evaluation result ER1 of the practical answer PA may include the results of evaluation using multiple evaluation items EI, and may further include a score for each evaluation item EI based on the model answer ED2. Specifically, for example, the multiple evaluation items EI may include an evaluation of the validity and comprehensiveness of the information retrieved from the information group IG, an evaluation of the consistency of the content between the model answer ED2 and the practical answer PA, and an evaluation of the consistency between the intent of the answer in the model answer ED2 and the intent of the answer in the practical answer PA. Furthermore, a score based on the model answer ED2 may be displayed for each of these evaluation items EI. According to this embodiment, the practical answer PA can be evaluated from multiple perspectives using multiple evaluation items EI.

[0057] The calculation unit 233 determines whether the evaluation result ER satisfies a predetermined criterion (activity A007). The predetermined criterion in this case is a criterion corresponding to the accuracy of the generated practical answer PA, such as whether the practical answer PA falls into any of a predetermined range of accuracy classes, or whether the score based on the model answer ED2 is equal to or greater than a predetermined value.

[0058] If the evaluation result ER does not satisfy the predetermined criteria, the process proceeds to activity A008, and if the evaluation result ER satisfies the predetermined criteria, the process proceeds to activity A009. The process of activity A008 will be described below first.

[0059] When the process proceeds to activity A008, the modification unit 235 modifies the setting information IFs based on the reference information RI (activity A008). Specifically, for example, a mathematical optimization algorithm may be employed for the reference information RI, and the reference information RI may be defined so as to obtain a high score using an evaluation result ER indicated by a score based on the model answer ED2 as an objective function. That is, the modification unit 235 may modify the setting information IFs using a mathematical optimization algorithm so as to obtain a high score using the evaluation result ER indicated by a score based on the model answer ED2 as an objective function. In addition, in this case, an importance level may be set for each parameter P in the setting information IFs, and the modification unit 235 may modify each parameter P according to the importance level. Note that the reference information RI in this case may be rule-based information such as a predetermined function, or may be a trained model in which the correlation between input and output has been trained in advance.

[0060] The process then returns to activity A005. In the process of activity A005, the generation unit 234 generates a practical answer PA based on the question ED1, the information group IG, and the second reference information RI2 in which the setting information IFs has been changed. In the process of activity A006, the output unit 237 outputs the evaluation result ER based on the model answer ED2, the generated practical answer PA, and the third reference information RI3. That is, the output unit 237 outputs multiple evaluation results ER based on multiple practical answers PA obtained by repeating the processes of activities A005 and A008 (the processes of the change unit 235 and the generation unit 234). In the process of activity A007, the calculation unit 233 again determines whether the output evaluation result ER satisfies a predetermined criterion. In this manner, the processes of activities A005 to A008 are repeated until the evaluation result ER satisfies the predetermined criterion. According to this embodiment, the optimization of the setting information IFs can be efficiently performed using the reference information RI, such as a mathematical optimization algorithm.

[0061] Furthermore, when the reference information RI is a mathematical optimization algorithm, an optimization process is executed using a score based on the model answer ED2 as the objective function. In other words, the optimization process in this case is a process of executing the processes of activities A005 to A008 (the processes of the modification unit 235, the generation unit 234, and the output unit 237) until the objective function (the score based on the model answer ED2) satisfies a predetermined standard. Specifically, for example, the optimization process repeats changing the setting information IFs, generating the practical answer PA, and outputting the evaluation result ER until the score based on the model answer ED2, which is the objective function, becomes 80% or more. In this case, the setting information IFs is changed by deriving an optimal solution using the mathematical optimization algorithm. The setting information IFs may be changed according to the output evaluation result ER, or may be changed regardless of the output evaluation result ER.

[0062] Then, when the evaluation result ER satisfies a predetermined criterion and the processing proceeds to activity A009, the display control unit 238 displays the evaluation result ER on the display unit 34 of the terminal 3 (activity A009). That is, the optimization unit 236 executes the optimization process in response to receiving the instruction Ins, and then, when the optimization process is completed, the display control unit 238 displays the evaluation result ER. According to this embodiment, the optimization can be executed automatically.

[0063] The above is the flow of processing according to the specific example. According to this embodiment, it is possible to improve the accuracy of the practical answers PA using the obtained evaluation results ER, and as a result, it is possible to construct an information retrieval tool that can obtain high-quality and highly reliable answers.

[0064] 3.3 Details Next, the above-mentioned information processing will be further explained using another diagram.

[0065] FIG. 7 is a schematic diagram showing an example of the setting screen 4a, which is the top screen of the setting screens 4.

[0066] 7, the setting screen 4a has at least buttons 401, 410, and 420. The button 401, which says "Confirm Settings," is a button for checking the current setting information IFs, and pressing it transitions from the setting screen 4a to a screen (not shown) that displays the setting information IFs.

[0067] The button 410 labeled "Execute Evaluation" is a button for evaluating the practical answer PA generated under the current settings. Pressing this button executes the process of evaluating the practical answer PA, and then transitions to the setting screens 4b and 4c, which show the evaluation results. In other words, the receiving unit 231 receives an instruction to execute the process of evaluating the practical answer PA through the setting screen 4a by pressing the button 410. In this case, the generating unit 234 generates a practical answer PA for the question ED1, the output unit 237 outputs the evaluation result ER, and then the display control unit 238 displays the evaluation result ER. In other words, the process of determining whether the evaluation result ER satisfies a predetermined criterion by the calculation unit 233 and the process of changing the setting information IFs based on the reference information RI by the change unit 235 are not executed.

[0068] Furthermore, button 420 labeled "Perform automatic tuning" is a button for optimizing each parameter P in the setting information IFs, and when pressed, the optimization process is executed, and then the screen transitions to setting screens 4d and 4e showing the evaluation results. In other words, the receiving unit 231 receives an instruction Ins as an instruction to perform tuning of the setting information IFs of the information retrieval application IR by an input operation of pressing button 420 as an operation via the setting screen 4a.

[0069] FIG. 8 is a schematic diagram showing an example of a setting screen 4b of the setting screen 4, which shows the result ER1 of evaluating the practical answer PA.

[0070] 8, the setting screen 4b has an area 411, which shows the evaluation items EI and the results for each evaluation item EI from the result ER1 of evaluating the practice answer PA. Specifically, the area 411 has areas 412a to 412e which show the evaluation items EI, respectively, and areas 413a to 413e which show the results for each evaluation item EI. The areas 412a to 412e and the areas 413a to 413e are arranged so that the evaluation items EI and the results corresponding to the evaluation items EI are paired on the left and right.

[0071] Area 413a is located to the right of area 412a, which is labeled "Accuracy." Area 413a shows "D" out of five classes A to E, as a result of using the accuracy classes that are set in stages in advance.

[0072] Area 413b is located to the right of area 412b, which is labeled "Evaluation of Search Results." Area 413b displays "45%" as a percentage evaluation of the validity, comprehensiveness, etc. of the information retrieved from the information group IG.

[0073] Area 413c is located to the right of area 412c, which is labeled "Degree of agreement with model answer (5-point scale)." Area 413c displays "3.3" as the result of a 5-point scale evaluation of the agreement between model answer ED2 and practice answer PA.

[0074] Area 413d is located to the right of area 412d, which is labeled "Degree of agreement with model answer (correct answer rate)." Area 413d displays "66%" as the result of evaluating the agreement between model answer ED2 and practice answer PA in terms of the correct answer rate.

[0075] To the right of area 412e, which is labeled "Reflection of Intention," is area 413e. Area 413e displays a score of "3.3" as a result of a five-point evaluation of the consistency between the intention of the answer in model answer ED2 and the intention of the answer in practice answer PA.

[0076] In this way, the setting screen 4b displays the evaluation result ER1 of the practical answer PA, which is a result of evaluating the practical answer PA from multiple aspects using multiple evaluation items EI. Note that, on the setting screen 4b, the evaluation result ER1 of five types of practical answer PA is displayed in areas 412a-412e and areas 413a-413e, but is not limited to this. The evaluation result ER1 of the practical answer PA may be any result that allows evaluation of the accuracy of the generated practical answer PA. For example, the evaluation result ER1 of at least one type of practical answer PA may be displayed, or the evaluation result ER1 of five or more types of practical answer PA may be displayed.

[0077] FIG. 9 is a schematic diagram showing an example of a setting screen 4c of the setting screen 4, which shows the result ER1 of evaluating the practical answer PA.

[0078] 9, the setting screen 4c has an area 414, which displays information depicting a comparison between the practical answer PA and the model answer ED2 from the evaluation result ER1 of the practical answer PA. Specifically, the area 414 has areas 415a to 415c that display information related to the practical answer PA, and areas 416a to 416c that display information related to the evaluation data ED including the model answer ED2.

[0079] The area 415a and the area 416a show the common question ED1. That is, it shows that the practice answer PA and the model answer ED2 are answers obtained based on the common question ED1.

[0080] In the area 415b, a practice answer PA is shown as an answer obtained based on the question ED1, and in the area 416b, a model answer ED2 is shown as an answer obtained based on the question ED1.

[0081] Area 415c shows information that was used as the basis for generating the practical answer PA, and area 416c shows information that was used as the basis for generating the model answer ED2.

[0082] In this way, on the setting screen 4c, the result ER1 of the evaluation of the practical answer PA can be confirmed by referring to the information that depicts the practical answer PA in comparison with the model answer ED2, the information that was used as the basis when generating the answer, etc.

[0083] FIG. 10 is a schematic diagram showing an example of a setting screen 4d of the setting screen 4, which shows the result ER1 of evaluation of the practical answer PA before and after the optimization process of the setting information IFs is executed.

[0084] As shown in Fig. 10, the setting screen 4d has an area 421, which displays evaluation items EI and the results for each evaluation item EI, among the results ER1 of evaluating the practical answer PA. Furthermore, the area 421 displays the results ER1 of evaluating the practical answer PA before and after the optimization process for the setting information IFs of the information retrieval application IR. Specifically, the area 421 has areas 422a to 422e that display the evaluation items EI, and areas 423a to 423e that display the results before and after the optimization process for each evaluation item EI. The areas 422a to 422e and the areas 423a to 423e are arranged so that the evaluation items EI and the results corresponding to the evaluation items EI are paired on the left and right.

[0085] Area 423a is located to the right of area 422a, which is labeled "Accuracy." Area 423a shows that, as a result of using preset accuracy classes, the accuracy has improved from "D" to "A" out of five classes from A to E.

[0086] Area 423b is located to the right of area 422b, which is labeled "Evaluation of Search Results." Area 423b shows the results of evaluating the validity, comprehensiveness, etc. of the information retrieved from the information group IG as a percentage, indicating that the evaluation has improved from "45%" to "81%."

[0087] Area 423c is located to the right of area 422c, which is labeled "Degree of agreement with model answer (5-point scale)." Area 423c shows the result of evaluating the agreement between model answer ED2 and practice answer PA on a 5-point scale, which indicates that the agreement has improved from "3.3" to "4.2."

[0088] Area 423d is located to the right of area 422d, which is labeled "Degree of agreement with model answer (correct answer rate)." Area 423d shows the evaluation of the agreement between model answer ED2 and practice answer PA in terms of the correct answer rate, which indicates that the agreement has improved from "66%" to "88%."

[0089] Area 423e is located to the right of area 422e, which is labeled "Reflection of Intention." Area 423e shows an improvement from "3.3" to "4.2" as a result of evaluating the consistency between the intention of the answer in model answer ED2 and the intention of the answer in practice answer PA on a 5-point scale.

[0090] In this case, the setting information IFs of the information retrieval application IR is different information before and after the optimization process is performed. Specifically, for example, the setting information IFs includes at least first setting information IFs1, which is the setting information IFs before the optimization process is performed, and second setting information IFs2, which is the setting information IFs after the optimization process is performed. Then, a first practical answer PA1 is generated as the practical answer PA when the second reference information RI2 is set by the first setting information IFs1. Furthermore, a second practical answer PA2 is generated as the practical answer PA when the second reference information RI2 is set by the second setting information IFs2. In other words, the generation unit 234 generates the first practical answer PA1 and the second practical answer PA2 based on the question ED1, the information group IG, and the second reference information RI2. In this case, the first practical answer PA1 is the practical answer PA when the second reference information RI2 is set by the first setting information IFs1. Furthermore, the second practical answer PA2 is the practical answer PA when the second reference information RI2 is set by the second setting information IFs2. The evaluation result ER includes the result of evaluating the first practical answer PA1 and the second practical answer PA2, obtained using the model answer ED2 as a reference. The results of evaluating the first practical answer PA1 and the second practical answer PA2 are displayed in areas 423a to 423e of the setting screen 4d, for example, as the results before and after performing the optimization process. According to this embodiment, it is possible to compare and confirm the evaluation of the first practical answer PA1 and the evaluation of the second practical answer PA2, and it is possible to grasp the impact of changing the setting information IFs.

[0091] FIG. 11 is a schematic diagram showing an example of a setting screen 4e of the setting screen 4, which shows the result ER1 of evaluation of the practical answer PA after the optimization process of the setting information IFs of the information search application IR has been executed.

[0092] 11, the setting screen 4e has an area 424, which displays information depicting a comparison between the second practical answer PA2 and the model answer ED2 from the evaluation result ER1 of the practical answer PA. Specifically, the area 424 has areas 425a to 425c that display information about the second practical answer PA2, and areas 426a to 426c that display information about the evaluation data ED including the model answer ED2.

[0093] The area 425a and the area 426a show the common question ED1. That is, the second practice answer PA2 and the model answer ED2 are shown as answers obtained based on the common question ED1.

[0094] In area 425b, a second practice answer PA2 is shown as an answer obtained based on question ED1, and in area 426b, a model answer ED2 is shown as an answer obtained based on question ED1.

[0095] Area 425c shows information that was used as the basis for generating the second practical answer PA2, and area 426c shows information that was used as the basis for generating the model answer ED2.

[0096] In this way, on the setting screen 4e, the result ER1 of the evaluation of the second practical answer PA2 can be confirmed by referring to the information displayed in comparison between the second practical answer PA2 and the model answer ED2, the information used as the basis for generating the answer, etc. Furthermore, by comparing it with the information displayed in comparison between the first practical answer PA1 and the model answer ED2 on the setting screen 4c, it becomes possible to compare and confirm the evaluation of the first practical answer PA1 and the evaluation of the second practical answer PA2, and the impact of changing the setting information IFs can be grasped.

[0097] [others] The information processing system 1 according to the above embodiment may be configured as follows.

[0098] In the above embodiment, the case where the receiving unit 231 receives the instruction Ins, and then the acquiring unit 232 acquires the specific document Dc1 and the generating unit 234 generates the evaluation data ED has been described as an example, but this is not limiting. For example, the process of the acquiring unit 232 acquiring the specific document Dc1 and the generating unit 234 generating the evaluation data ED may be executed in advance, rather than by receiving the instruction Ins. In other words, the process of the acquiring unit 232 acquiring the specific document Dc1 and the generating unit 234 generating the evaluation data ED, which is a preparatory process for the optimization process, may be executed before the optimization process starts. According to this aspect, the preparatory process for the optimization process can be completed in advance, thereby shortening the time required for the optimization process.

[0099] In the above embodiment, the first reference information RI1, the second reference information RI2, and the third reference information RI3 are used in the processing, but this is not limiting. For example, common reference information RI may be used for all or part of the first reference information RI1, the second reference information RI2, and the third reference information RI3.

[0100] In the above embodiment, the generated evaluation data ED is used without verification. However, this is not limiting. For example, a process for verifying the generated evaluation data ED may be executed. Specifically, for example, the necessity of the specific content Cn1, which is the information on which the model answer ED2 is based, may be verified, or logical inconsistencies may be examined between the question ED1, the model answer ED2, and the specific content Cn1. For example, if the model answer ED2 can be generated based on the question ED1 and the reference information RI without inputting the specific content Cn1, which is the information on which the model answer ED2 is based, then the specific content Cn1 is unnecessary and the data is not suitable as evaluation data ED. Furthermore, it is advisable to verify logical inconsistencies between the question ED1, the model answer ED2, and the specific content Cn1 based on the question ED1, the model answer ED2, the specific content Cn1, and the reference information RI.

[0101] In the above embodiment, the question ED1 is a question about vacations or the like, and the model answer ED2 is generated for the question ED1. However, this is not limiting. For example, the question ED1 may be composed of multiple questions, and a model answer ED2 may be generated for each of the multiple questions. In this manner, having multiple questions ED1 and model answers ED2 allows the practical answer PA to be evaluated from multiple perspectives.

[0102] In the above embodiment, the optimization process is performed until the output evaluation result ER satisfies a predetermined standard, but the present invention is not limited to this. For example, multiple pieces of setting information IFs may be generated in advance using reference information RI such as a mathematical optimization algorithm, and evaluation results ER may be output using the generated multiple pieces of setting information IFs, thereby identifying the best evaluation result ER.

[0103] Furthermore, an information processing method including each step of the information processing system 1 may be implemented. That is, this information processing method includes each step of the information processing system 1. According to this aspect, it is possible to build an information search tool (for example, an information search application IR) that can acquire high-quality and reliable answers through information processing. Furthermore, the above embodiment may be implemented as a distributable program. That is, this program causes at least one computer to execute each step of the information processing system 1. According to this aspect, it is possible to build an information search tool that can acquire high-quality and reliable answers through information processing by the program.

[0104] At least one of the devices included in the information processing system 1 may be installed outside Japan. For example, the information processing device 2 may be installed outside Japan, and a user in Japan may access the information processing device 2 using his or her own terminal 3, or the information processing device 2 may be installed in Japan, and a user outside Japan may access the information processing device 2 using his or her own terminal 3.

[0105] In one embodiment, the reception unit 231, acquisition unit 232, calculation unit 233, generation unit 234, change unit 235, optimization unit 236, output unit 237, and display control unit 238 are described as functional units implemented by the control unit 23 of the information processing device 2. However, at least some of these may be implemented as functional units implemented by the control unit 33 of the terminal 3, a control unit of another external device, or the like. Specifically, for example, the other external device may be a server that provides a service using a generation AI. Furthermore, the various pieces of information described in the above example may be stored in a distributed manner not only in the storage unit 22 of the information processing device 2 but also in other external devices. In such cases, distributed ledger management based on blockchain or the like may be performed.

[0106] It may be provided in the following manner.

[0107] (1) An information processing system comprising at least one processor, the processor being configured to execute the following steps by reading a program: an acquisition step of acquiring a specific document, wherein the specific document is a document selected based on arbitrary conditions from a group of information including a plurality of documents; a first generation step of generating evaluation data based on the specific document and preset first reference information, wherein the evaluation data includes a question regarding specific content contained in the specific document and a model answer to the question obtained based on the specific content; a second generation step of generating a practical answer to the question based on the question, the group of information, and preset second reference information; and an output step of outputting an evaluation result based on the model answer, the practical answer, and preset third reference information, wherein the evaluation result includes a result of evaluating the practical answer obtained based on the model answer.

[0108] According to this aspect, it is possible to improve the accuracy of practical answers using the obtained evaluation results, and as a result, it is possible to construct an information search tool that can obtain high-quality, highly reliable answers.

[0109] (2) In the information processing system described in (1) above, the second reference information includes setting information that can be set arbitrarily, where the setting information has a plurality of parameters used when extracting arbitrary information from the group of information, and includes first setting information and second setting information different from the first setting information; in the second generation step, a first practical answer and a second practical answer are generated based on the question, the group of information, and the second reference information, where the first practical answer is the practical answer when the second reference information is set by the first setting information, and the second practical answer is the practical answer when the second reference information is set by the second setting information; and the evaluation result includes a result of evaluating the first practical answer and the second practical answer obtained based on the model answer.

[0110] According to this aspect, it becomes possible to compare and confirm the evaluation of the first practical answer and the evaluation of the second practical answer, and it is possible to grasp the influence of a change in the setting information.

[0111] (3) In the information processing system described in (1) or (2) above, the results of evaluating the practice answer include a score based on the model answer.

[0112] According to this aspect, the practice answer can be evaluated using a score based on the model answer.

[0113] (4) In the information processing system according to any one of (1) to (3) above, the results of evaluating the practical answers include results of evaluation based on a plurality of evaluation items.

[0114] According to this embodiment, the practical answers can be evaluated from multiple perspectives using multiple evaluation items.

[0115] (5) In the information processing system described in any one of (1) to (4) above, the processor is configured to further execute the following steps: the second reference information includes setting information that can be set arbitrarily, and the setting information has a plurality of parameters used when extracting arbitrary information from the group of information; in the change step, the setting information is changed using a mathematical optimization algorithm so that the evaluation result, indicated by a score based on the model answer, becomes a high score as an objective function; and in the output step, a plurality of evaluation results are output based on a plurality of practical answers obtained by repeating the processing of the change step and the second generation step.

[0116] According to this aspect, it is possible to efficiently optimize the setting information using a mathematical optimization algorithm.

[0117] (6) In the information processing system described in (5) above, the processor is configured to further execute the following steps: in the receiving step, an instruction to perform optimization is received; in the optimization step, an optimization process is executed in response to the reception of the instruction; and the optimization process is a process of executing the processing of the change step, the second generation step, and the output step until the objective function satisfies a predetermined criterion.

[0118] According to this embodiment, optimization can be performed automatically.

[0119] (7) An information processing method, comprising the steps of the information processing system according to any one of (1) to (6) above.

[0120] According to this aspect, it is possible to construct an information search tool that can acquire high-quality and highly reliable answers through information processing.

[0121] (8) A program that causes at least one computer to execute each step of the information processing system according to any one of (1) to (6) above.

[0122] According to this embodiment, it is possible to construct an information search tool that can obtain high-quality and highly reliable answers through information processing using a program. Of course, this is not the case.

[0123] Finally, while various embodiments of the present disclosure have been described, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0124] 1: Information processing system 11: Communication Network 2: Information processing equipment 20: Communication bus 21: Communications Department 22: Storage section 23: Control section 231: Reception 232: Acquisition Department 233: Arithmetic section 234 :Generation part 235: Change section 236: Optimization section 237: Output section 238: Display control unit 3: Terminal 30: Communication bus 31: Communications Department 32: Storage section 33: Control section 34:Display section 35: Input section 4: Settings screen 4a: Setting screen 4b: Settings screen 4c: Settings screen 4D: Setting screen 4e: Settings screen 401 :ボタン 410 :ボタン 411: Domain 412a: Domain 412b: Domain 412c: Domain 412d: Domain 412e: Domain 413a: Domain 413b: Domain 413c: Domain 413d: Domain 413e: Domain 414: Domain 415a: Domain 415b: Domain 415c: Domain 416a: Domain 416b: Domain 416c: Domain 420 :ボタン 421: Domain 422a: Domain 422b: Domain 422c: Domain 422d: Domain 422e: Domain 423a: Domain 423b: Domain 423c: Domain 423d: Domain 423e: Domain 424: Domain 425a: Domain 425b: Domain 425c: Domain 426a: Domain 426b: Domain 426c: Domain Cn1: Specific content Dc:ドキュメント Dc1:Specific ドキュメント ED: Commentary ED1: Question ED2: Model answer EI: Evaluation items ER: Evaluation results ER1:Results IFs: Setting information IFs1: First setting information IFs2: Second setting information IG: Information group IR: Information search app Ins :Instructions P: Parameter PA: Practical Answers PA1: First Practical Answer PA2: Second Practical Answer RI: Reference information RI1: Primary Reference Information RI2: Second Reference Information RI3: Third Reference

Claims

1. An information processing system, at least one processor; The processor is configured to execute the following steps by reading the program: In the acquisition step, a specific document is acquired, wherein the specific document is the document selected based on an arbitrary condition from an information group including a plurality of documents; In a first generation step, evaluation data is generated based on the specific document and predetermined first reference information, and the evaluation data includes a question related to specific content included in the specific document and a model answer to the question obtained based on the specific content; In a second generation step, a practical answer to the question is generated based on the question, the information group, and second reference information set in advance; In the output step, an evaluation result is output based on the model answer, the practical answer, and third reference information set in advance, wherein the evaluation result includes a result of evaluating the practical answer obtained using the model answer as a reference; the second reference information includes setting information that can be arbitrarily set, wherein the setting information has a plurality of parameters used when extracting arbitrary information from the information group, and includes first setting information and second setting information different from the first setting information; In the second generating step, a first practical answer and a second practical answer are generated based on the question, the information group, and the second reference information, wherein: the first practical answer is the practical answer when the second reference information is set by the first setting information, the second practical answer is the practical answer when the second reference information is set by the second setting information, The evaluation results include results of evaluating the first practice answer and the second practice answer, obtained using the model answer as a reference.

2. An information processing system, at least one processor; The processor is configured to execute the following steps by reading the program: In the acquisition step, a specific document is acquired, wherein the specific document is the document selected based on an arbitrary condition from an information group including a plurality of documents; In a first generation step, evaluation data is generated based on the specific document and predetermined first reference information, and the evaluation data includes a question related to specific content included in the specific document and a model answer to the question obtained based on the specific content; In a second generation step, a practical answer to the question is generated based on the question, the information group, and second reference information set in advance; In the output step, an evaluation result is output based on the model answer, the practical answer, and third reference information set in advance, wherein the evaluation result includes a result of evaluating the practical answer obtained using the model answer as a reference; the second reference information includes setting information that can be arbitrarily set, and the setting information has a plurality of parameters used when extracting arbitrary information from the information group; In the modification step, the evaluation result indicated by a score based on the model answer is used as an objective function, and the setting information is modified by a mathematical optimization algorithm so that the score becomes a high score; In the output step, the system outputs a plurality of the evaluation results based on a plurality of the practical answers obtained by repeating the processing of the change step and the second generation step.

3. 2. The information processing system according to claim 1, The system wherein the results of evaluating the practice answer include a score relative to the model answer.

4. 3. The information processing system according to claim 1, The system wherein the results of evaluating the practice answers include results of evaluation based on a plurality of evaluation items.

5. 3. The information processing system according to claim 2, The processor is further configured to perform the following steps: In the reception step, instructions to execute optimization are received, In the optimization step, an optimization process is executed in response to receiving the instruction, and the optimization process is a process of executing the processes of the change step, the second generation step, and the output step until the objective function satisfies a predetermined criterion.

6. An information processing method, comprising: A method comprising the steps of the information processing system according to claim 1 or 2.

7. A program, A program causing at least one computer to execute each step of the information processing system according to claim 1 or 2.

Images