Information processing device, information processing method, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program which make it possible to generate a more appropriate answer (consulting response data) to a question regarding the use of an application program. A question from a user regarding a prescribed application program and an operation log at that time are acquired. From the question and the operation log, the degree of understanding of the user with respect to the prescribed application program is estimated. On the basis of the estimated degree of understanding and the question and the operation log, an answer to the question is estimated using a large-scale language model. The present invention can be applied to consulting services.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable realization of a machine learning model that outputs appropriate answers to user questions.

[0002] Recently, there are various application programs available, but many of them require a certain level of prior knowledge to use.

[0003] However, in reality, users may not be able to fully utilize an application program due to insufficient prerequisite knowledge, and may give up on using the application program and abandon it.

[0004] To prevent users who lack sufficient prerequisite knowledge from abandoning the application program, application program providers offer consulting services related to how to use the application program.

[0005] However, because consulting services impose a heavy burden on application providers, they are generally provided for a fee. As a result, users are reluctant to use these services, and in reality, only a small number of users use them.

[0006] Therefore, a technology has been proposed that creates a learning model that realizes an expert clone by machine learning using learning data consisting of pairs of existing questions related to the use of application programs and consulting response data that are correct answers, and automates the generation of answers to questions (see Patent Literature 1).

[0007] JP 2023-114460 A

[0008] However, even if a question about the use of an application program is the same, the appropriate answer will vary depending on the problem the user is facing and the level of prior knowledge the user possesses. Therefore, the content of the answer needs to be appropriately adjusted depending on the problem and the level of prior knowledge the user possesses.

[0009] However, machine learning alone using learning data consisting of pairs of questions related to the use of existing application programs and consulting response data recording the answers of application providers is not enough to output appropriate answers to questions.

[0010] The present disclosure has been made in light of such circumstances, and particularly aims to enable the generation of appropriate consulting response data in response to questions regarding the use of application programs.

[0011] An information processing device and a program according to one aspect of the present disclosure include an information processing device and a program that include a question acquisition unit that acquires a user's question, an operation log acquisition unit that acquires an operation log of the user, and an answer estimation unit that estimates an answer to the question using a large-scale language model based on the question and the operation log.

[0012] An information processing method according to one aspect of the present disclosure is an information processing method including a question acquisition process for acquiring a user's question, an operation log acquisition process for acquiring an operation log of the user, and an answer estimation process for estimating an answer to the question using a large-scale language model based on the question and the operation log.

[0013] In one aspect of the present disclosure, a user's question is acquired, an operation log of the user is acquired, and an answer to the question is estimated based on the question and the operation log using a large-scale language model.

[0014] 7 is a diagram illustrating a technology for generating an answer from a question. FIG. 7 is a diagram illustrating an overview of the present disclosure for generating an answer from a question, an operation log, and a level of understanding. FIG. 7 is a diagram illustrating an overview of the present disclosure for generating an answer from a question, an operation log, and a level of understanding. FIG. 7 is a block diagram illustrating an example configuration of a consulting system of the present disclosure. FIG. 7 is a diagram illustrating an example configuration of the user terminal of FIG. 4. FIG. 7 is a diagram illustrating an example configuration of the data scientist terminal of FIG. 4. FIG. 7 is a diagram illustrating an example configuration of the consulting device of FIG. 4. FIG. 7 is a functional block diagram illustrating functions implemented by the consulting device of FIG. 7. FIG. 7 is a diagram illustrating an example (part 1) of estimating a level of understanding and an answer based on a question and an operation log. FIG. 7 is a diagram illustrating an example (part 2) of estimating a level of understanding and an answer based on a question and an operation log. FIG. 7 is a diagram illustrating an example (part 3) of estimating a level of understanding and an answer based on a question and an operation log. FIG. 7 is a diagram illustrating an example (part 4) of estimating a level of understanding and an answer based on a question and an operation log. FIG. 7 is a flowchart illustrating data scientist answer processing. FIG. 7 is a flowchart illustrating understanding level creation model learning processing. FIG. 7 is a flowchart illustrating answer creation model learning processing. FIG. 7 is a flowchart illustrating consulting device answer processing. Fig. 8 is a functional block diagram illustrating other functions realized by the consulting device of Fig. 7. Fig. 9 is a flowchart illustrating an understanding level generation model-encompassing answer generation model learning process. Fig. 10 is a diagram illustrating an example of the configuration of a general-purpose computer.

[0015] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0016] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the present disclosure 2. Preferred embodiments 3. Modifications 4. Examples of implementation by software

[0017] <<1. Overview of the Present Disclosure>> The present disclosure is directed to generating appropriate consulting response data in response to questions, particularly, regarding the use of application programs. First, an overview of the present disclosure will be described.

[0018] As mentioned above, various application programs have become available in recent years, but users may not be able to fully utilize the application programs due to insufficient prerequisite knowledge, and may give up on using the application programs and abandon them.

[0019] To prevent users who lack sufficient prerequisite knowledge from abandoning the application program, application program providers offer consulting services related to how to use the application program.

[0020] More specifically, consider the case where an existing application program uses so-called predictive analysis technology, which uses machine learning to predict future results from past performance data, and is, for example, a predictive analysis application program for non-experts that automatically builds an optimal predictive model when a data set is input.

[0021] When considering the use of such predictive analysis application programs, the threshold for using predictive analysis technology in business is lowered, but to effectively utilize it, a certain level of prior knowledge of machine learning is still required.

[0022] In this case, prerequisite knowledge refers to, for example, specific measures that should be taken when performing machine learning to improve the accuracy of a predictive model for business use.

[0023] Human-assisted consulting services provided by data scientists (who answer questions) from existing application program providers can eliminate the bottleneck of non-experts' lack of prior knowledge about machine learning.

[0024] However, because this consulting service places a heavy burden on the application program provider, it is generally provided for a fee, which has led to users being reluctant to use it, and in reality, only a small number of users are using it.

[0025] Therefore, there is a growing need to automate this consulting service using machine learning models.

[0026] To meet this need, as shown in the upper part of Figure 1, a technology has been proposed that extracts, in natural language form, existing questions related to the use of an application program (user question "Q") and consulting response data (data scientist answer "A"), which is the correct answer, from information recorded during a consultation by a data scientist with a user (consultation recording), and then uses machine learning using the training data consisting of the extracted pairs to create a learning model consisting of large language models (LLM) that realizes expert cloning, thereby automating the generation of answers to questions.

[0027] However, as shown in the lower part of Figure 1, in reality, even if a question (user question "Q") related to the use of an application program is the same question, the appropriate answer will vary depending on the problem the user is facing and the level of prior knowledge the user possesses. Therefore, the answer (data scientist's answer "A") to the question (user question "Q") needs to be appropriately adjusted depending on the problem and the level of prior knowledge the user possesses.

[0028] For this reason, it is known that machine learning alone using learning data consisting of pairs of questions related to the use of existing application programs (user questions "Q") and consulting response data recording the answers of application program providers (data scientist answers "A") cannot generate a learning model that outputs an appropriate answer (data scientist answers "A") to a question (user questions "Q").

[0029] More specifically, for example, answers to the question, "I would like to improve the accuracy of the predictive model a little more," could include a variety of answers, such as "You should increase the amount of data," "There are not enough valid columns," "The problem is not set up in a way that can be solved by machine learning in the first place, so you should rewrite the problem," or "How about using a time series predictive model?" In addition, some users may not understand why "You should increase the amount of data."

[0030] Therefore, in this disclosure, as shown in Figure 2, in addition to the existing question (user question "Q") related to the use of the application program and the consulting response data (data scientist's answer "A") which is the correct answer, the operation log "L" of the application program is converted into natural language and extracted based on predetermined rules, and notes left by the data scientist during consulting that express the target user's level of understanding "U" are converted into natural language and extracted.

[0031] The memo expressing this level of understanding "U" is a memo expressing general information that can be used to estimate the user's level of understanding of the application program, including what the data scientist noticed, felt, and actually happened during the interaction when accepting a question from the user and providing consulting services, as well as the user's characteristics. In other words, the level of understanding "U" is a general natural language expression that can be used to estimate not only whether the user has specific knowledge about the application program, but also the level of understanding in terms of the depth of involvement with the application program, such as whether the user is familiar with it or not.

[0032] Then, as shown in the lower part of Figure 3, a learning model consisting of a large-scale language model that estimates the level of understanding "U" from the operation log "L" and the user's question "Q" is learned based on machine learning using learning data that pairs the operation log "L" and the user's question "Q" with the level of understanding "U."

[0033] Furthermore, as shown in the upper part of Figure 3, a learning model consisting of a large-scale language model that estimates consulting response data (data scientist's answer "A") from the operation log "L", the level of understanding "U", and the user's question "Q" is generated based on machine learning using a learning pair of the operation log "L", the level of understanding "U", and the user's question "Q" with the consulting response data that is the correct answer (data scientist's answer "A").

[0034] This allows the consulting response data (data scientist's answer "A") to be estimated using the user's question "Q," as well as the operation log "L" and the level of understanding "U," thereby enabling the estimation of appropriate consulting response data (data scientist's answer "A").

[0035] More specifically, for example, if the existing application program is a predictive analysis application, the operation log can be used to estimate the user's uncertainties, allowing an appropriate answer to be selected depending on the uncertainties.

[0036] Furthermore, by utilizing the user's level of understanding, for example, in response to an answer such as "The number of data should be increased," it is possible to add information that the user can understand, such as "Why it needs to be increased" and "How much should it be increased?", thereby outputting a more appropriate answer.

[0037] <<2. Preferred Embodiment>> <Configuration Example of Consulting System> Next, with reference to FIG. 4, a configuration example of a consulting system according to the present disclosure will be described.

[0038] The consulting system 11 in FIG. 4 is configured to include a user terminal 31, a data scientist terminal 32, and a consulting device 33, which are capable of communicating with each other via a network 34 such as the Internet.

[0039] The user terminal 31 is a terminal device such as a PC or a tablet terminal used by a user who uses a predetermined application program. The user uses the predetermined application program by operating the user terminal 31 on which the predetermined application program is installed.

[0040] Furthermore, by operating the user terminal 31, the user can supply a question about a specific application program to the data scientist terminal 32 or the consulting device 33 via the network 34, and can obtain and present a corresponding answer. When supplying a question to the data scientist terminal 32 or the consulting device 33, the user terminal 31 also supplies an operation log by the user.

[0041] The data scientist terminal 32 is a terminal managed and operated by the provider of a specified application program, and is operated by a data scientist who generates answers to questions from users supplied from the user terminal 31, accepts input of answers to questions, and notifies the user terminal 31.

[0042] At this time, the data scientist operates the data scientist terminal 32 to record information indicating the user's level of understanding of the specified application program based on questions exchanged with the user via the user terminal 31, in a memo-like manner.

[0043] In addition, the data scientist recognizes the user's level of understanding of a specified application program based on the operation log supplied along with the question, and generates an answer to the question according to the recognized level of understanding by operating the data scientist terminal 32.

[0044] When notifying the user terminal 31 of an answer to a question from the user terminal 31, the data scientist terminal 32 also notifies the consulting device 33 via the network 34 of information indicating the question, operation log, level of understanding, and answer.

[0045] Unlike the data scientist terminal 32 operated by the data scientist, the consulting device 33 accepts questions from the user terminal 31 on behalf of the data scientist, generates corresponding answers, and supplies them to the user terminal 31.

[0046] The consulting device 33 generates a learning model consisting of an LLM that generates answers to questions through machine learning based on information indicating questions, operation logs, understanding levels, and answers supplied from the data scientist terminal 32.

[0047] The consulting device 33 then uses a learning model consisting of an LLM generated by machine learning from the questions, operation logs, understanding levels, and answers supplied from the data scientist terminal 32 to generate answers to the questions supplied from the user terminal 31 and supplies them to the user terminal 31.

[0048] More specifically, the consulting device 33 uses a learning model consisting of an LLM generated by machine learning to estimate the level of understanding based on the questions and operation logs, and then estimates an answer based on the estimated level of understanding, the questions, and the operation logs, and supplies it to the user terminal 31.

[0049] In other words, the consulting device 33 needs to be used after a learning model has been completed to some extent through machine learning based on the questions, operation logs, understanding levels, and answers supplied from the data scientist terminal 32.

[0050] However, once the learning model has been completed to a certain extent, the consulting device 33 will be able to improve the accuracy of the answers it generates by repeating machine learning using past questions, operation logs, and understanding levels, as well as correct answers.

[0051] As a result, once the learning model has been completed to a certain extent, the consulting device 33 will be able to generate answers to questions on its own, and at the same time, repeat machine learning of the learning model using past questions, operation logs, levels of understanding, and answers, making it possible to operate the system without performing machine learning based on questions, operation logs, levels of understanding, and answers from the data scientist terminal 32.

[0052] <Example of Hardware Configuration of User Terminal> Next, an example of the hardware configuration of the user terminal 31 will be described with reference to FIG.

[0053] The user terminal 31 is composed of a control unit 51, an input unit 52, an output unit 53, a memory unit 54, a communication unit 55, a drive 56, and a removable storage medium 57, which are connected to each other via a bus 58 and can send and receive data and programs.

[0054] The control unit 51 is composed of a processor and a memory, and controls the overall operation of the user terminal 31. The control unit 51 also includes a question acquisition unit 71, an operation log acquisition unit 72, and an answer presentation unit 73.

[0055] The question acquisition unit 71 accepts and acquires questions input by the user via the input unit 52 regarding the use of a specific installed application program, and controls the communication unit 55 to transmit the questions to the data scientist terminal 32 or the consulting device 33.

[0056] The operation log acquisition unit 72 acquires an operation log generated when a user operates the input unit 52 in relation to the use of a specific installed application program, and when a question is sent, controls the communication unit 55 to send the log to the data scientist terminal 32 or the consulting device 33.

[0057] The answer presentation unit 73 controls the communication unit 55 to acquire answers to questions supplied by the question acquisition unit 71 from the data scientist terminal 32 or the consulting device 33, and presents the answers on a display included in the output unit 53.

[0058] The input unit 52 is composed of input devices such as a keyboard, a mouse, and a touch panel for inputting various information, and supplies the control unit 51 with various signals corresponding to the input information.

[0059] The output unit 53 is controlled by the control unit 51 and includes a display (not shown) and an audio output unit (not shown). The display is controlled by the answer presentation unit 73 and presents answers corresponding to questions that are acquired by the question acquisition unit 71 and transmitted to the data scientist terminal 32 or the consulting device 33.

[0060] The audio output unit is made up of an audio output device such as a speaker, and outputs various types of voice, music, sound effects, and the like as audio.

[0061] The input unit 52 and the output unit 53 function as a user interface 61 that is a touch panel having both functions.

[0062] The storage unit 54 is composed of a hard disk drive (HDD), a solid state drive (SSD), or a semiconductor memory, and is controlled by the control unit 51 to write or read various data and programs. Furthermore, questions input by operating the input unit 52 and answers acquired by the answer presentation unit 73 may be temporarily written or read from the storage unit 54 as needed under the control of the control unit 51.

[0063] The communication unit 55 is controlled by the control unit 51 and realizes communication via wired or wireless means, such as LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs to and from other information processing devices via the network as necessary.

[0064] The drive 56 reads and writes data from and to a removable storage medium 57 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0065] <Example of Hardware Configuration of Data Scientist Terminal> Next, an example of the hardware configuration of the data scientist terminal 32 will be described with reference to FIG. 6 .

[0066] The data scientist terminal 32 is composed of a control unit 81, an input unit 82, an output unit 83, a memory unit 84, a communication unit 85, a drive 86, and a removable storage medium 87, which are connected to each other via a bus 88 and can send and receive data and programs.

[0067] In addition, the control unit 81, input unit 82, output unit 83, memory unit 84, communication unit 85, drive 86, and removable storage medium 87, as well as the mutual bus 88 correspond to the control unit 51, input unit 52, output unit 53, memory unit 54, communication unit 55, drive 56, and removable storage medium 57, as well as to the bus 58, respectively, and therefore explanations thereof will be omitted as appropriate.

[0068] The control unit 81 includes a question acquisition unit 91 , an operation log acquisition unit 92 , an understanding level acquisition unit 93 , and an answer acquisition unit 94 .

[0069] The question acquisition unit 91 controls the communication unit 85 to acquire a question relating to the use of a predetermined application program supplied from the user terminal 31 and presents the question on the display of the output unit 83 .

[0070] The operation log acquisition unit 92 controls the communication unit 85 to acquire an operation log relating to the use of a predetermined application program supplied from the user terminal 31 and presents it on the display of the output unit 83 .

[0071] The understanding level acquisition unit 93 accepts and acquires the user's understanding level (including information such as notes that can be used to estimate the understanding level) recognized through the user's questions and dialogue, which is input by the data scientist operating the input unit 82. The understanding level acquisition unit 93 controls the communication unit 55 to transmit the acquired understanding level information to the consulting device 33.

[0072] The answer acquisition unit 94 accepts operational input of answers to questions based on the level of understanding and operation log, which are input by the data scientist operating the input unit 82, and controls the communication unit 55 to send the answers to the user terminal 31 and the consulting device 33.

[0073] <Example of Hardware Configuration of Consulting Apparatus> Next, an example of the hardware configuration of the consulting apparatus 33 will be described with reference to FIG.

[0074] The consulting device 33 is composed of a control unit 101, an input unit 102, an output unit 103, a memory unit 104, a communication unit 105, a drive 106, and a removable storage medium 107, which are interconnected via a bus 108 and can send and receive data and programs.

[0075] In addition, the control unit 101, input unit 102, output unit 103, memory unit 104, communication unit 105, drive 106, and removable storage medium 107, as well as the mutual bus 108 correspond to the control unit 51, input unit 52, output unit 53, memory unit 54, communication unit 55, drive 56, and removable storage medium 57 of the user terminal 31, as well as to the bus 58, respectively, and therefore explanations will be omitted as appropriate.

[0076] The control unit 101 is composed of a processor and a memory, and controls the overall operation of the consulting device 33. The control unit 101 also includes a question acquisition unit 131, an operation log acquisition unit 132, an understanding level acquisition unit 133, an answer acquisition unit 134, an understanding level creation model learning unit 135, an understanding level creation model estimation unit 136, an answer creation model learning unit 137, and an answer creation model estimation unit 138.

[0077] In both the data scientist answering process in which a data scientist generates an answer to a question and the consulting device answering process in which the consulting device 33 generates an answer to a question, the question acquiring unit 131 controls the communication unit 105 to acquire a question related to the use of a predetermined application program supplied from the user terminal 31. Then, the question acquiring unit 131 registers the acquired question in the database (DB) 181 of the storage unit 104 as part of the consulting data 161.

[0078] In both the data scientist response processing and the consulting device response processing, the operation log acquisition unit 132 controls the communication unit 105 to acquire an operation log related to the use of a predetermined application program supplied from the user terminal 31. Then, the operation log acquisition unit 132 associates the acquired operation log with a question in the consulting data 161 and registers it as operation log data 162 in the database (DB) 181 of the storage unit 104.

[0079] In the data scientist response processing, the understanding level acquisition unit 133 controls the communication unit 105 to acquire information on the understanding level in response to a question about the use of a specific application program, which is supplied from the data scientist terminal 32. Then, the understanding level acquisition unit 133 associates the acquired understanding level information with the question in the consulting data 161 and registers it in the database (DB) 181 of the storage unit 104.

[0080] In the data scientist response processing, the response acquisition unit 134 controls the communication unit 105 to acquire information related to the response to the question about the use of a predetermined application program, which is supplied from the data scientist terminal 32. Then, the response acquisition unit 134 registers the acquired response information in the database (DB) 181 of the storage unit 104 in association with the question in the consulting data 161.

[0081] The understanding creation model learning unit 135 learns an understanding creation model that takes the question and the operation log as input and outputs the corresponding understanding level based on learning data that pairs the question registered as consulting data 161 and the operation log registered as operation log data 162, which are obtained in the data scientist response processing, and outputs the corresponding understanding level, and supplies the learning result to the understanding creation model estimation unit 136.

[0082] In the consulting device response processing, the understanding level creation model estimation unit 136 estimates the understanding level corresponding to the question and operation log supplied from the user terminal 31 based on the understanding level creation model learned by the understanding level creation model learning unit 135.

[0083] The answer creation model learning unit 137 learns an answer creation model that takes the question, the level of understanding, and the operation log as input and outputs an answer based on learning data that pairs the question and level of understanding registered as consulting data 161, and the operation log registered as operation log data 162 and the answer registered as consulting data 161, and supplies the learning results to the answer creation model estimation unit 138.

[0084] In the consulting device answering process, the answer creation model estimating unit 138 estimates an answer corresponding to the question, operation log, and corresponding understanding level supplied from the user terminal 31, based on the answer creation model learned by the answer creation model learning unit 137. Then, the answer creation model estimating unit 138 controls the communication unit 105 to transmit the estimated answer to the user terminal 31.

[0085] <Functions Realized by Consulting Apparatus> Next, functions realized by the consulting apparatus 33 of FIG. 7 will be described with reference to the functional block diagram of FIG.

[0086] In the database (DB) 181, consulting data 161 consisting of questions, answers, and understanding levels, and operation log data 162 consisting of operation logs are stored in the storage unit 104 in association with the questions.

[0087] The understanding level creation model learning unit 135 reads out the questions and understanding levels from the consulting data 161 from the DB 181, converts them into natural language together with the operation logs from the operation log data 162, and generates an understanding level creation model consisting of an LLM that takes the questions and operation logs as input and outputs the understanding levels through machine learning using learning data consisting of pairs of questions, operation logs, and corresponding understanding levels, and supplies the generated understanding level creation model to the understanding level creation model estimation unit 136.

[0088] The understanding creation model estimation unit 136 uses the understanding creation model supplied by the understanding creation model learning unit 135 to convert the questions and operation logs input by operating the user interface 61 into natural language, estimate the understanding level, and supply the estimated understanding level to the answer creation model estimation unit 138.

[0089] The answer creation model learning unit 137 reads out the questions, answers, and understanding levels of the consulting data 161 from the DB 181, converts them into natural language together with the operation logs of the operation log data 162, and generates an answer creation model that takes the questions, operation logs, and understanding levels as inputs and outputs answers through machine learning using learning data consisting of pairs of questions, operation logs, and understanding levels with corresponding answers, and supplies the generated answer creation model to the answer creation model estimation unit 138.

[0090] The answer creation model estimation unit 138 uses the answer creation model supplied by the answer creation model learning unit 137 to convert the questions and operation logs input by operating the user interface 61, as well as the understanding level supplied by the understanding level creation model estimation unit 136, into natural language, estimates an answer to the question, supplies it to the user interface unit 171, and presents it to the user.

[0091] <Example (Part 1) of Estimating Level of Understanding and Answer Based on Question and Operation Log> Next, with reference to Fig. 9, an example (Part 1) of estimating level of understanding and answer based on a question and operation log input via the user interface 61 will be described. Note that the following explanation of examples (Parts 1 to 4) of estimating level of understanding and answer based on a question and operation log will be given assuming that the application program is a predictive analysis application program.

[0092] As shown in the left part of FIG. 9, the question acquisition unit 91 controls the user interface 61 to present a comment C1 such as "Hello! Is there anything I can help you with?"

[0093] In response to this, when the user operates the user interface 61 and inputs a comment C2 such as "I don't understand the cause of the error of insufficient columns," the question acquisition unit 91 acquires a question Q1 consisting of this comment C2 and outputs it to the understanding creation model estimation unit 136 and the answer creation model estimation unit 138.

[0094] At the same time, the operation log acquisition unit 92 acquires an operation log for the application program when the user operates the user interface 61, and further converts this into natural language to acquire an operation log L1 such as, for example, "T is the prediction target for data in X rows and Y columns...prediction started with a model with 99% accuracy...an error of insufficient columns occurred during prediction," and outputs this to the understanding creation model estimation unit 136 and the answer creation model estimation unit 138.

[0095] Based on the operation log L1 and the question Q1 (operation log L1 + question Q1), the understanding creation model estimation unit 136 estimates the understanding U1 expressed in natural language, such as, for example, "I don't know the cause of the error of insufficient columns during prediction. The concept of leaks needs to be explained." and outputs the estimation result to the answer creation model estimation unit 138.

[0096] Based on the operation log L1, the level of understanding U1, and the question Q1 (operation log L1 + level of understanding U1 + question Q1), the answer creation model estimation unit 138 estimates an answer A1 such as, "A model is being created with 'T' as the prediction target, but 'TT' is included in the explanatory variables. These may look like similar columns, but is 'TT' a column that can be prepared at the time of prediction? If not, please recreate the prediction model using training data that excludes this column. Creating a prediction model by adding a column that should not be used like this can result in a phenomenon known as leakage, where the accuracy is unreasonably high." and displays this as a comment C3 on the user interface 61.

[0097] Through the above process, the level of understanding U1, "I don't know the cause of the missing column error" is estimated from the question Q1, "I don't know the cause of the missing column error," and the operation log L1, which is expressed in natural language as "I set T as the prediction target with data from X rows and Y columns...I started the prediction with a model with 99% accuracy...I got a missing column error during prediction." The concept of leak needs to be explained.

[0098] Based on question Q1, "I don't know what caused the missing column error," operation log L1, which is translated into natural language, such as "I set T as the prediction target with data from rows X and columns Y...I started the prediction with a model with 99% accuracy...I got a missing column error during prediction," and comprehension U1, which states, "I don't know what caused the missing column error during prediction. The concept of leak needs to be explained.", answer A1 is inferred and presented: "I'm creating a model with "T" as the prediction target, and "TT" is included in the explanatory variables. These columns appear to be similar, but is "TT" a column that can be prepared during prediction? If not, please re-create the prediction model using training data that excludes this column. Creating a prediction model by adding a column that shouldn't be used can sometimes result in an unreasonably high accuracy, a phenomenon known as leak."

[0099] As a result, the level of understanding U1 is estimated from not only question Q1 but also operation log L1, such as "T was set as the prediction target with data of X rows and Y columns...prediction was started with a model with 99% accuracy...an error occurred due to insufficient columns during prediction." This makes it possible to properly estimate the items in which the user has insufficient understanding, such as the fact that the error occurred during prediction, that an explanatory variable "TT" similar to the prediction target "T" was used, and that the concept of leaks does not appear to be understood.

[0100] As a result, answer A to question Q1 is generated based on the level of understanding U1 so as to mention items that are not fully understood, making it possible to make answer A1 appropriate for question Q1.

[0101] <Example (part 2) of estimating the level of understanding and answers based on questions and operation logs> Next, referring to FIG. 10 , an example (part 2) of estimating the level of understanding and answers based on questions input via the user interface 61 and operation logs will be described.

[0102] As shown in the left part of FIG. 10, the question acquisition unit 91 controls the user interface 61 to present a comment C11 such as "Hello! Is there anything I can help you with?"

[0103] In response to this, when the user operates the user interface 61 and inputs a comment C12 such as "The prediction accuracy is not sufficient at all, so I would like to improve it," the question acquisition unit 91 acquires the question Q11 consisting of this comment C12 and outputs it to the understanding degree creation model estimation unit 136 and the answer creation model estimation unit 138.

[0104] At the same time, the operation log acquisition unit 92 acquires an operation log for the application program when the user operates the user interface 61, and further converts this into natural language to acquire an operation log L11 such as, for example, "A numerical prediction model was learned that predicts the 'number of visitors' from the 'date' and 'store'...a model with an accuracy of 40% was created...", and outputs this to the understanding creation model estimation unit 136 and the answer creation model estimation unit 138.

[0105] The understanding level creation model estimation unit 136 estimates the understanding level U11 expressed in natural language, for example, "I don't know that there are types of numerical prediction models and time-series prediction models. You should explain whether there is a trend," based on the operation log L11 and the question Q11 (operation log L11 + question Q11), and outputs the estimation result to the answer creation model estimation unit 138.

[0106] Based on the operation log L11, comprehension level U11, and question Q11 (operation log L11 + comprehension level U11 + question Q11), the answer creation model estimation unit 138 estimates an answer A11 such as, "We are creating a numerical prediction model that predicts the number of store visits based on date and store information. This is an algorithm that predicts the number of store visits based only on information such as "February 21, 2024" and "Tokyo store." If there is a trend in the number of store visits (e.g., an increasing trend year by year), it may be better to use a time series prediction model instead of a numerical prediction model. The time series prediction model makes predictions using not only date and store information but also past store visit counts (e.g., the number of store visits the day before, the number of store visits seven days ago, the number of store visits 28 days ago, the number of store visits one year ago, etc.). This may allow us to capture trends in store visits and enable more accurate predictions." The answer generation model estimates the answer A11 and presents it as a comment C13 on the user interface 61.

[0107] Through the above process, the natural language understanding U11 of "I don't know that there are types of numerical prediction models and time series prediction models. I need an explanation about whether there is a trend" is estimated from the question Q1, "The prediction accuracy is not sufficient at all, so I would like to improve it," and the operation log L11, which is expressed in natural language, such as "I learned a numerical prediction model that predicts the number of store visits from the date and store...I have created a model with an accuracy of 40%..."

[0108] Based on the question Q11, "The prediction accuracy is not sufficient, so I would like to improve it," the operation log L11, which is translated into natural language, such as "I trained a numerical prediction model to predict 'number of visits' from 'date' and 'store'... I created a model with 40% accuracy...," and the natural language understanding U11, which states, "I don't know that there are different types of numerical prediction models and time series prediction models. You should explain whether there is a trend.", the answer A11 is estimated and presented as follows: "I am creating a numerical prediction model that predicts number of visits based on date and store information. This is an algorithm that predicts number of visits based only on information such as '2 / 21 / 2024' and 'Tokyo store.' If there is a trend in number of visits (e.g., an annual increase), it may be better to use a time series prediction model instead of a numerical prediction model. A time series prediction model inputs not only date and store information but also past number of visits (e.g., number of visits the day before, number of visits seven days ago, number of visits 28 days ago, number of visits one year ago, etc.) to make predictions. This may allow me to capture trends in number of visits and enable more accurate predictions."

[0109] As a result, the level of understanding U11 is estimated from the question Q11 as well as the operation log L11, which states, "We learned a numerical prediction model that predicts the number of store visits from the date and store...a model with an accuracy of 40% was created..." Therefore, it is appropriately estimated that the user has insufficient understanding of items such as not knowing that there are types of numerical prediction models and time-series prediction models, and whether or not there are trends.

[0110] As a result, answer A11 to question Q11 is generated based on the level of understanding U11 so as to mention items that are not fully understood, making it possible to make answer A11 appropriate for question Q11.

[0111] <Example (part 3) of estimating the level of understanding and answers based on questions and operation logs> Next, referring to FIG. 11 , an example (part 3) of estimating the level of understanding and answers based on questions input via the user interface 61 and operation logs will be described.

[0112] As shown in the left part of FIG. 11, the question acquisition unit 91 controls the user interface 61 to present a comment C21 such as "Hello! Is there anything I can help you with?"

[0113] In response to this, when the user operates the user interface 61 and inputs a comment C22 such as "Why is file input required during prediction?", the question acquisition unit 91 acquires a question Q21 consisting of this comment C22 and outputs it to the understanding degree creation model estimation unit 136 and the answer creation model estimation unit 138.

[0114] At the same time, the operation log acquisition unit 92 acquires an operation log for the application program when the user operates the user interface 61, and further converts this into natural language to acquire an operation log L21 such as, for example, "Learning a binary classification model that predicts 'continue / cancel membership' from 'customer rank,' 'past purchase amount,' and 'place of residence,'...", and outputs this to the understanding creation model estimation unit 136 and the answer creation model estimation unit 138.

[0115] The understanding level creation model estimation unit 136 estimates the understanding level U21 expressed in natural language, for example, "I don't understand that predictive analysis has learning and prediction steps," based on the operation log L21 and the question Q21 (operation log L21 + question Q21), and outputs the estimation result to the answer creation model estimation unit 138.

[0116] Based on the operation log L21, comprehension level U21, and question Q21 (operation log L21 + comprehension level U21 + question Q21), the answer creation model estimation unit 138 calculates the following three steps for predictive analysis: "Learning," "Evaluation," and "Prediction." First, the input data is divided into training data and evaluation data. The predictive model learns patterns from this training data to predict "Continuance / Cancellation" based on "Customer Rank," "Past Purchase Amount," and "Place of Residence." This is "Learning." To verify the quality of the resulting predictive model, evaluation data that the predictive model has not yet seen is used. "Continuance / Cancellation" is predicted based on the "Customer Rank," "Past Purchase Amount," and "Place of Residence" in the evaluation data, and the actual "Continuance / Cancellation" is confirmed to be consistent. This is "Evaluation." Once a good predictive model is created through evaluation, customers whose "Continuance / Cancellation" status is not yet known are classified based on their "Customer Rank," "Past Purchase Amount," and "Place of Residence." This is "Prediction." For this reason, at the time of prediction, it is necessary to input a file containing information on "customer rank," "past purchase amount," and "place of residence" of customers whose cancellation is not yet known." The system infers an answer A21 such as this and presents it as a comment C23 on the user interface 61.

[0117] Through the above process, a natural language understanding U21, "Does not understand that predictive analysis involves learning and prediction steps," is estimated from the question Q21, "Why is file input required during prediction?" and the operation log L21, which is natural language such as "Learned a binary classification model that predicts 'continue / cancel' from 'customer rank,' 'past purchase amount,' and 'place of residence.'"

[0118] Based on the question Q21, "Why am I asked to input a file when making a prediction?", the natural language operation log L11, which states, "We trained a binary classification model to predict 'retention / cancellation' based on 'customer rank,' 'past purchase amount,' and 'place of residence,'...", and the natural language understanding U21, which states, "I don't understand that predictive analysis has two steps: learning and prediction," we conclude that predictive analysis has three main steps: 'learning,' 'evaluation,' and 'prediction.' First, the input data is divided into training data and evaluation data. From this training data, the predictive model learns patterns to predict 'retention / cancellation' based on 'customer rank,' 'past purchase amount,' and 'place of residence.' This is the 'training' step. To verify the quality of the resulting predictive model, evaluation data that the predictive model has not yet seen is used. The evaluation data's 'retention / cancellation' is predicted based on 'customer rank,' 'past purchase amount,' and 'place of residence,' and the actual 'retention / cancellation' is confirmed to be consistent. This is the 'evaluation' step. Once a good prediction model has been created through evaluation, customers whose "continuation / cancellation" status is not yet known will be classified based on their "customer rank," "past purchase amount," and "place of residence." This is "prediction." For this reason, when making predictions, a file containing information on "customer rank," "past purchase amount," and "place of residence" for customers whose cancellation status is not yet known must be input. The answer A21 is estimated and presented.

[0119] As a result, the level of understanding U21 is estimated not only from question Q21 but also from operation log L21, such as "Learning a binary classification model that predicts 'continuation / cancellation' from 'customer rank,' 'past purchase amount,' and 'place of residence,'...", so that items such as not understanding that predictive analysis involves learning and prediction steps are appropriately estimated as items where the user's understanding is insufficient.

[0120] As a result, the answer A21 to the question Q21 is generated based on the level of understanding U21 so as to mention the items that are not fully understood, thereby making it possible to make the answer A21 appropriate for the question Q21.

[0121] <Example (part 4) of estimating the level of understanding and answers based on questions and operation logs> Next, referring to Figure 12, we will explain an example (part 4) of estimating the level of understanding and answers based on questions entered via the user interface 61 and operation logs.

[0122] As shown in the left part of FIG. 12, the question acquisition unit 91 controls the user interface 61 to present a comment C31 such as "Hello! Is there anything I can help you with?"

[0123] In response to this, when the user operates the user interface 61 and inputs a comment C32 such as "Normality is predicted well, but failure prediction is poor," the question acquisition unit 91 acquires the question Q31 consisting of this comment C32 and outputs it to the understanding degree creation model estimation unit 136 and the answer creation model estimation unit 138.

[0124] At the same time, the operation log acquisition unit 92 acquires an operation log for the application program when the user operates the user interface 61, and further converts this into natural language to acquire an operation log L31 such as, for example, "Learned a binary classification model that predicts 'normal / failed' from 'model number,' 'months in operation,' 'last inspection,' and 'installation location,'" and outputs this to the understanding creation model estimation unit 136 and the answer creation model estimation unit 138.

[0125] The understanding level creation model estimation unit 136 estimates the understanding level U31 expressed in natural language, for example, "I do not know that the threshold can be adjusted, and I do not know the concepts of Precision and Recall," based on the operation log L31 and question Q31 (operation log L31 + question Q31), and outputs the estimation result to the answer creation model estimation unit 138.

[0126] Based on the operation log L31, comprehension level U31, and question Q31 (operation log L31 + comprehension level U31 + question Q31), the answer creation model estimation unit 138 calculates the following evaluation indices for binary classification: Precision and Recall. Precision is the percentage of data predicted as "failure" that actually turned out to be "failure." Therefore, a high Precision can be said to have fewer false positives. On the other hand, Recall is the percentage of data that actually turned out to be "failure" that was predicted to be "failure." Therefore, a high Recall can be said to have fewer overlooked failures. The output of a binary classification prediction model is a number between 0 and 1, which can be interpreted as a failure probability. Here, the boundary value that determines whether a value above a certain level should be classified as "failure" or below a certain level should be classified as "normal" is called the "threshold." Adjusting this threshold changes the Precision and Recall values, but there is a trade-off between Precision and Recall; increasing one will worsen the other. In your case, consider which is more important, Precision or Recall (if fewer false positives are important, a higher Precision is better, and if fewer oversights are important, a higher Recall is better), and adjust the threshold accordingly, and you may be able to achieve a level of accuracy that is practical." The system then estimates an answer A31 such as this, and presents it as a comment C33 on the user interface 61.

[0127] Through the above process, the level of understanding U31 expressed in natural language, such as "Does not know that the threshold can be adjusted, and does not know the concepts of precision and recall," is estimated from the question Q31, which states, "Normality is predicted well, but failure prediction is poor," and the operation log L31, which is expressed in natural language, such as "Learn a binary classification model that predicts 'normal / failure' from 'model number,' 'months in operation,' 'last inspection,' and 'installation location.'"

[0128] Based on question Q31, which states, "It predicts normal conditions well, but it predicts failures poorly," and the natural-language operation log L11, which states, "We trained a binary classification model to predict 'continuation / cancellation' based on 'customer rank,' 'past purchase amount,' and 'place of residence,'..." and the natural-language comprehension U21, which states, "They do not understand that predictive analysis involves learning and prediction steps," we conclude, "Binary classification evaluation metrics include precision and recall. Precision is the percentage of data predicted as "failure" that actually turned out to be "failure." Therefore, high precision can be said to have fewer false positives. On the other hand, recall is the percentage of data predicted as "failure" out of actual "failure." Therefore, high recall can be said to have fewer missed failures. The output of a binary classification predictive model is a number between 0 and 1, which can be interpreted as a failure probability. Here, the boundary value above which this value should be classified as "failure" and below which value should it be classified as "normal" is called the "threshold." Adjusting this threshold will change the Precision and Recall values, but there is a trade-off between Precision and Recall; increasing one will worsen the other. Adjusting the threshold based on which is more important in your case, Precision or Recall (if fewer false positives are important, a higher Precision is better, and if fewer oversights are important, a higher Recall is better) may result in a practical level of accuracy." The estimated answer, A31, is presented.

[0129] As a result, the level of understanding U31 is estimated from the operation log L31, such as "Learning a binary classification model that predicts 'normal / failure' from 'model number,' 'months in operation,' 'last inspection,' and 'installation location,'" in addition to question Q31, so that the ability to adjust the threshold and ignorance of the concepts of precision and recall are appropriately estimated as items in which the user has insufficient understanding.

[0130] As a result, answer A31 to question Q31 is generated based on the level of understanding U31 to refer to items that are not fully understood, making it possible to make answer A31 appropriate for question Q31.

[0131] <Data Scientist Response Processing> Next, the data scientist response processing will be described with reference to the flowchart of FIG.

[0132] In step S31, the question acquisition unit 71 of the user terminal 31 determines whether or not the user has operated the user interface 61 to input a question.

[0133] If it is determined in step S31 that a question has been input, the process proceeds to step S32.

[0134] In step S32, the question acquisition unit 71 of the user terminal 31 accepts a question input by the user operating the user interface 61.

[0135] In step S33, the operation log acquisition unit 72 acquires an operation log in which the user operates the user interface 61 and serves as an operation input to the application program.

[0136] In step S34 , the question acquisition unit 71 and the operation log acquisition unit 72 each control the communication unit 55 to transmit the acquired question and operation log to the data scientist terminal 32 and the consulting device 33 .

[0137] In step S51, the question acquisition unit 91 and the operation log acquisition unit 92 of the data scientist terminal 32 each control the communication unit 85 to determine whether a question and an operation log have been transmitted from the user terminal 31.

[0138] If it is determined in step S51 that a question and an operation log have been transmitted, the process proceeds to step S52.

[0139] In step S52, the question acquisition unit 91 and the operation log acquisition unit 92 each control the communication unit 85 to acquire the question and operation log sent from the user terminal 31 and present them by displaying them on the display of the output unit 83.

[0140] In step S53, the understanding level acquisition unit 93 receives an input of the understanding level of the user of the user terminal 31 that sent the question, which is input by the data scientist operating the input unit 82.

[0141] In step S54, the answer acquisition unit 94 accepts an answer to the question input by the data scientist operating the input unit 82.

[0142] In step S55, the answer acquisition unit 94 controls the communication unit 85 to transmit to the user terminal 31 the answer to the question input by the data scientist operating the input unit 82.

[0143] In step S35, the answer presentation unit 73 of the user terminal 31 controls the communication unit 55 to obtain the answer to the question sent by the data scientist terminal 32 and present it on the display of the output unit 53.

[0144] If it is determined in step S31 that a question has not been input, the processes of steps S32 to S35 are skipped.

[0145] Meanwhile, in step S56, the understanding level acquisition unit 93 and the answer acquisition unit 94 each control the communication unit 85 to associate the understanding level and the answer to the question entered by the data scientist by operating the input unit 82 with the question and transmit them to the consulting device 33.

[0146] If it is determined in step S51 that the question and operation log have not been transmitted, steps S52 to S56 are skipped.

[0147] In step S71, the question acquisition unit 131 and the operation log acquisition unit 132 of the consulting device 33 each control the communication unit 105 to determine whether a question and an operation log have been sent from the user terminal 31 in association with the question.

[0148] If it is determined in step S71 that the question and the operation log have been transmitted in association with the question, the process proceeds to step S72.

[0149] In step S72 , the question acquisition unit 131 and the operation log acquisition unit 132 each control the communication unit 105 to acquire the question and the operation log transmitted from the user terminal 31 .

[0150] In step S73, the question acquiring unit 131 and the operation log acquiring unit 132 register the acquired question and operation log in the DB 181 as consulting data 161 and operation log data 162, respectively.

[0151] In step S74, the understanding level obtaining unit 133 of the consulting device 33 controls the communication unit 105 to determine whether or not an answer and an understanding level associated with the question have been transmitted from the user terminal 31.

[0152] If it is determined in step S74 that an answer and a level of understanding have been transmitted in association with the question, the process proceeds to step S75.

[0153] In step S75, the understanding level acquisition unit 133 controls the communication unit 105 to acquire the answer and the understanding level in association with the question.

[0154] In step S76, the understanding level acquiring unit 133 associates the acquired answer and understanding level with the question and registers them as consulting data 161 in the DB 181.

[0155] Furthermore, if it is determined in step S71 that the question and operation log have not been transmitted, the processing of steps S72 to S73 is skipped, and if it is determined in step S74 that the answer and understanding level have not been transmitted, the processing of steps S75 to S76 is skipped.

[0156] If the end of the process is not instructed in steps S36, S57, and S77, the process returns to steps S31, S51, and S71, respectively, and the subsequent processes are repeated. If the end of the process is instructed in steps S36, S57, and S77, the process ends.

[0157] Through the above process, when a user inputs a question about the use of an application program into the user terminal 31 , the question is sent together with the immediately preceding operation log to the data scientist terminal 32 and the consulting device 33 .

[0158] As a result, a question and an operation log are presented on the data scientist terminal 32, and when the data scientist inputs the user's level of understanding on the user terminal 31 based on the presented question and operation log, it is accepted in association with the question.

[0159] Furthermore, when the data scientist inputs an answer to the question, the answer is sent to the user terminal 31 and presented.

[0160] This allows users to obtain answers to their questions from data scientists, and resolve any doubts they may have about using application programs.

[0161] Meanwhile, in the consulting device 33, the question and operation log from the user terminal 31 are associated with the question and registered in the DB 181 as consulting data 161 and operation log data 162, respectively. Furthermore, in the consulting device 33, the answer and understanding level are associated with the question from the data scientist terminal 32 and registered in the DB 181 as consulting data 161, associated with the question.

[0162] Through this process, the question, the level of understanding, and the answer are registered as consulting data 161, and the operation log is registered as operation log data 162 in DB 181. This makes it possible to use the information in DB 181 to realize the understanding level creation model learning process and the answer creation model learning process, which will be described later.

[0163] <Understanding Level Creation Model Learning Process> Next, the understanding level creation model learning process performed by the consulting device 33 will be described with reference to the flowchart of FIG.

[0164] In step S101, the understanding level creation model learning unit 135 acquires understanding level information corresponding to the question from the consulting data 161 in the DB 181 stored in the memory unit 104, and also acquires operation log information corresponding to the question from the operation log data 162.

[0165] In step S102, the understanding level creation model learning unit 135 learns an understanding level creation model consisting of an LLM that receives an operation log and a question as input and outputs a corresponding understanding level.

[0166] Through the above processing, the questions, operation logs, and levels of understanding from the information in DB181 accumulated by the data scientist response processing are used to train an understanding creation model consisting of an LLM that takes operation logs and questions as input and outputs the corresponding levels of understanding based on learning data consisting of pairs of questions, operation logs, and levels of understanding, and can be applied to the understanding creation model estimation unit 136.

[0167] <Answer Generation Model Learning Process> Next, the answer generation model learning process performed by the consulting device 33 will be described with reference to the flowchart of FIG.

[0168] In step S121, the answer creation model learning unit 137 acquires information on the answers and understanding levels corresponding to the questions expressed in natural language from the consulting data 161 in the DB 181 stored in the memory unit 104, and also acquires information on the operation logs expressed in natural language corresponding to the questions from the operation log data 162.

[0169] In step S122, the understanding level generation model learning unit 135 learns an answer generation model consisting of an LLM that receives the operation log, the question, and the understanding level as input and outputs the corresponding answer.

[0170] Through the above processing, by utilizing the natural language questions, operation logs, levels of understanding, and answers, which are information stored in DB181 by the data scientist response processing, an answer creation model consisting of an LLM that takes operation logs, questions, and levels of understanding as input and outputs corresponding answers is learned based on learning data consisting of pairs of questions, operation logs, levels of understanding, and corresponding answers, and can be applied to the answer creation model estimation unit 138.

[0171] <Consulting Device Response Processing> Next, the consulting device response processing by the consulting device 33 will be described with reference to the flowchart in Fig. 16. Note that the processing of steps S131 to S136 in the flowchart in Fig. 16 is the same as the data scientist response processing described with reference to the flowchart in Fig. 13, and therefore description thereof will be omitted.

[0172] In step S 151 , the question acquisition unit 131 and the operation log acquisition unit 132 of the consulting device 33 each control the communication unit 105 to determine whether a question and an operation log have been transmitted from the user terminal 31 .

[0173] In step S151, if it is determined that a question has been input by the user at the user terminal 31 and an operation log has been sent along with the question through the processing of steps S131 to S134, for example, then the processing proceeds to step S152.

[0174] In step S152 , the question acquisition unit 131 and the operation log acquisition unit 132 each control the communication unit 105 to acquire the question and the operation log transmitted from the user terminal 31 .

[0175] In step S153, the understanding creation model estimation unit 136 uses the understanding creation model learned by the understanding creation model learning process to estimate the understanding level of the user of the user terminal 31 based on the question and operation log sent from the user terminal 31.

[0176] In step S154, the answer creation model estimation unit 138 uses the answer creation model learned in the answer creation model learning process to estimate an answer based on the question, operation log, and level of understanding sent from the user terminal 31.

[0177] In step S155, the answer creation model estimation unit 138 controls the communication unit 105 to send the estimated answer to the user terminal 31. In response to this, in step S135, the answer presentation unit 73 of the user terminal 31 controls the communication unit 55 to obtain the answer to the question sent by the data scientist terminal 32 and presents it on the display of the output unit 53.

[0178] Furthermore, in step S156, the understanding level acquisition unit 133 and the answer acquisition unit 134 register the understanding level estimated by the understanding level creation model estimation unit 136 and the answer estimated by the answer creation model estimation unit 138 in the DB 181, respectively.

[0179] If it is determined in step S151 that the question and operation log have not been transmitted, steps S152 to S156 are skipped.

[0180] Then, in steps S136 and S157, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to steps S131 and S151, and the subsequent processes are repeated.

[0181] Then, in steps S136 and S157, if an instruction to end the process is given, the process ends.

[0182] Through the above processing, the consulting device 33 acquires the question and the corresponding operation log supplied from the user terminal 31, and estimates the user's level of understanding of the application program from the question and the operation log. Then, the answer to the question is estimated from the question, the operation log, and the level of understanding.

[0183] This makes it possible for the consulting device 33 to estimate and generate an answer based on the question, operation log, and level of understanding, and to present the user with an appropriate answer based on the user's level of understanding. In other words, an answer that refers to information that the user does not fully understand is generated based on the user's level of understanding, making it possible to generate and present an appropriate answer that takes into account the background behind the user's question.

[0184] In addition, in the consulting device response processing, operation logs, levels of understanding, and answers are stored in DB181 in association with questions, and the understanding creation model and answer creation model are re-learned by further executing the understanding creation model learning processing and answer creation model learning processing from the questions, operation logs, levels of understanding, and answers stored in DB181.

[0185] This makes it possible to use the understanding creation model and answer creation model that have been repeatedly retrained, thereby improving the accuracy of the understanding creation model estimation unit 136 and the answer creation model estimation unit 138 in estimating the understanding and answers, and enabling more appropriate answers to questions to be generated.

[0186] As a result, it becomes possible to generate more appropriate answers (consulting response data) to questions about the use of application programs.

[0187] <<3. Variations>> In the above, we have explained an example in which the level of understanding is estimated from a question and an operation log, and the answer to the question is estimated from the question, the operation log, and the level of understanding. However, since the level of understanding is estimated from the question and the operation log, and the answer is essentially estimated from the question and the operation log, the answer may also be estimated using only an answer creation model that includes an understanding creation model.

[0188] FIG. 17 is a functional block diagram illustrating functions realized by a consulting device 33 that estimates an answer using only an answer generation model that includes an understanding level generation model.

[0189] In Figure 17, the understanding creation model learning unit 135 and the understanding creation model estimation unit 136 are omitted, and instead of the answer creation model learning unit 137 and the answer creation model estimation unit 138, an understanding creation model-containing answer creation model learning unit 137' and an understanding creation model-containing answer creation model estimation unit 138' are provided.

[0190] The understanding level creating model-inclusive answer creating model estimating section 138 ′ has a configuration in which the functions of the understanding level creating model estimating section 136 are included in the functions of the answer creating model estimating section 138 .

[0191] Therefore, the understanding-level creation model-encompassing answer creation model estimation unit 138' estimates the level of understanding from the question and the operation log, and then estimates the answer by adding the question and the operation log to the estimated level of understanding. In other words, it can be said that the understanding-level creation model-encompassing answer creation model estimation unit 138' estimates the level of understanding and the answer from the question and the operation log.

[0192] The understanding level creation model-containing answer creation model learning unit 137' learns a model equivalent to an understanding level creation model that estimates understanding level from the questions and operation logs contained in the consulting data 161 based on the consulting data 161 and operation log data 162 registered in DB 181.

[0193] Then, the understanding creation model-containing answer creation model learning unit 137' uses a model corresponding to the learned understanding creation model to estimate the level of understanding from the question and the operation log, and learns an understanding creation model-containing answer creation model that estimates the answer from the estimated level of understanding plus the question and the operation log, and supplies this to the understanding creation model-containing answer creation model estimation unit 138'.

[0194] As a result, the understanding level generation model-inclusive answer generation model learning unit 137' appears to be able to estimate the understanding level and the answer from the question and the operation log.

[0195] However, the understanding-level creation model-inclusive answer creation model learning unit 137′ estimates the level of understanding in the process of estimating an answer from the question and the operation log, and then estimates the answer from the estimated level of understanding, the question, and the operation log. Therefore, the understanding-level creation model-inclusive answer creation model learning unit 137′ essentially has a function that combines the function of the understanding-level creation model estimation unit 136 and the function of the answer creation model estimation unit 138.

[0196] The understanding-level creation model-encompassing answer creation model learning unit 137' may learn an understanding-level creation model-encompassing answer creation model that only estimates the understanding level and the answer directly from the question and the operation log. Furthermore, the understanding-level creation model-encompassing answer creation model estimation unit 138' may estimate only the answer from the question and the operation log, without estimating the understanding level.

[0197] However, if the understanding level creation model-containing answer creation model estimation unit 138' only estimates the answer from the question and operation log, but does not estimate the understanding level, the understanding level estimation results will not be stored in DB 181 when the consulting device answer process is repeated.

[0198] On the other hand, in the data scientist answer processing, a question and an operation log are input, and each time a corresponding answer is generated by the data scientist, information on the degree of understanding with a high degree of accuracy is input by the data scientist and registered in DB181.

[0199] By using the highly accurate understanding levels input by data scientists to retrain the understanding level creation model-embedded answer creation model, it is expected that the accuracy of the answers estimated for questions will be improved.

[0200] Therefore, in the consulting device response processing, a question and an operation log are input, and each time a corresponding response is estimated, the level of understanding is estimated, and the information on the level of understanding is registered in DB 181 and used for re-learning, thereby making it possible to estimate a more appropriate response to a question.

[0201] Furthermore, when relearning, it is possible to use the level of understanding obtained by the data scientist response processing and the level of understanding obtained by the consulting device response processing without distinguishing between them, thereby enabling more appropriate answers to questions to be estimated.

[0202] As a result, in the consulting device answer processing, the level of understanding is accumulated in DB181 each time an answer is estimated, and by re-learning the answer creation model that contains the level of understanding, it becomes possible to estimate a more appropriate answer to a question.

[0203] In addition, in both the data scientist answer processing and the consulting device answer processing, questions and operation logs are supplied, and each time an answer is estimated, the level of understanding is also estimated and registered in DB181.By using the levels of understanding accumulated in both, the understanding creation model-embedded answer creation model can be re-learned, making it possible to estimate more appropriate answers to questions.

[0204] <Understanding Level Creating Model-Inclusive Answer Creating Model Learning Process> Next, the understanding level creating model-inclusive answer creating model learning process will be described with reference to the flowchart of FIG.

[0205] In step S171, the understanding level creation model-containing answer creation model learning unit 137′ acquires answer and understanding level information corresponding to the question from the consulting data 161 in DB 181 stored in the memory unit 104, and also acquires operation log information corresponding to the question from operation log data 162.

[0206] In step S172, the understanding-level creation model-enhanced answer creation model learning unit 137' receives the operation log and the question, outputs the corresponding understanding level, and then learns an understanding-level creation model-enhanced answer creation model that outputs an answer to the question from the operation log, the question, and the understanding level. Alternatively, the understanding-level creation model-enhanced answer creation model learning unit 137' receives the operation log and the question, and learns an understanding-level creation model-enhanced answer creation model that outputs the corresponding understanding level and answer.

[0207] Through the above processing, the information in DB181 accumulated by the data scientist response processing, namely, questions, operation logs, levels of understanding, and answers, is used to estimate the level of understanding from the questions and operation logs based on learning data consisting of pairs of questions, operation logs, levels of understanding, and corresponding answers.Then, an understanding creation model-inclusive answer creation model that estimates the level of understanding as an estimated result and the answer from the question and operation log is learned, and can be applied to the understanding creation model-inclusive answer creation model estimation unit 138'.

[0208] The consulting device response process by the consulting device 33 in FIG. 17 is the same as the process explained with reference to the flowchart in FIG. 16, and therefore will not be explained further.

[0209] However, if the understanding level creation model-containing answer creation model estimation unit 138′ in the consulting device 33 of FIG. 17 estimates only the answer from the question and the operation log, the processing of steps S153 and S154 will be processing of directly estimating the answer from the question and the operation log.

[0210] <<4. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program that constitutes the software is installed from a recording medium into a computer that is built into dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0211] 19 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0212] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0213] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0214] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0215] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0216] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0217] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0218] 19. The CPU 1001 in FIG. 19 realizes the functions of the control unit 101 in FIG.

[0219] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0220] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0221] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0222] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0223] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0224] The present disclosure may also be configured as follows: <1> An information processing device comprising: a question acquisition unit that acquires a question from a user; an operation log acquisition unit that acquires an operation log of the user; and an answer estimation unit that estimates an answer to the question based on the question and the operation log using a large-scale language model. <2> The information processing device described in <1>, wherein the question is a question about a predetermined application program, and the operation log is based on an operation when using the predetermined application program, and further includes an understanding level estimation unit that estimates a level of understanding of the predetermined application program related to the question based on the question and the operation log, and the answer estimation unit estimates an answer to the question based on the question, the operation log, and the level of understanding using the large-scale language model. <3> The information processing device according to <2>, further including an understanding level creation model learning unit that learns an understanding level creation model that takes the question and the operation log as input and outputs the understanding level, based on the past question, the past operation log, and the past level of understanding, wherein the understanding level estimation unit estimates the level of understanding based on the question and the operation log using the understanding level creation model. <4> The information processing device according to <3>, further including an answer creation model learning unit that learns an answer creation model that takes the question, the operation log, and the level of understanding as input, based on the past question, the past operation log, the past level of understanding, and the past answer, and that outputs the answer, wherein the answer estimation unit estimates the answer using the answer creation model, based on the question and the operation log and the level of understanding estimated by the understanding level estimation unit from the question and the operation log. <5> The information processing device described in <3>, wherein the past level of understanding is natural language information that expresses the level of understanding of the question, recognized through interaction with the user, and is input by a provider of the specified application program that generates an answer to the question based on the past question and the past operation log.<6> The information processing device according to <4>, wherein the answer to the question estimated by the answer estimation unit is associated with the question and registered in a database together with the operation log and the level of understanding, and the understanding level creation model learning unit re-learns the understanding level creation model based on the question, the operation log, and the level of understanding registered in the database. <7> The information processing device according to <6>, wherein the answer to the question estimated by the answer estimation unit is associated with the question and registered in the database together with the operation log and the level of understanding, and the understanding level creation model learning unit re-learns the answer creation model based on the question, the operation log, the level of understanding, and the answer registered in the database. <8> The information processing device according to <2>, further including an answer creation model training unit that outputs the level of understanding based on an input consisting of the question and the operation log based on the past question, the past operation log, the past level of understanding, and the past answer, and then trains an answer creation model that outputs the answer based on an input that adds the output level of understanding to the question and the operation log, wherein the answer estimation unit estimates the level of understanding from the question and the operation log using the answer creation model, and then estimates the answer based on the estimated level of understanding in addition to the question and the operation log. <9> The information processing device according to <2>, wherein the predetermined application program includes a predictive analysis application program. <10> An information processing method including: a question acquisition process that acquires a user's question; an operation log acquisition process that acquires the user's operation log; and an answer estimation process that estimates an answer to the question using a large-scale language model based on the question and the operation log. <11> A program that causes a computer to function as: a question acquisition unit that acquires a user's question; an operation log acquisition unit that acquires an operation log of the user; and an answer estimation unit that estimates an answer to the question by using a large-scale language model based on the question and the operation log.

[0225] REFERENCE SIGNS LIST 11 Consulting system, 31 User terminal, 32 Data scientist terminal, 33 Consulting device, 61 User interface, 71 Question acquisition unit, 72 Operation log acquisition unit, 73 Answer presentation unit, 91 Question acquisition unit, 92 Operation log acquisition unit, 93 Understanding level acquisition unit, 94 Answer acquisition unit, 131 Question acquisition unit, 132 Operation log acquisition unit, 133 Understanding level acquisition unit, 134 Answer acquisition unit, 135 Understanding level creation model learning unit, 136 Understanding level creation model estimation unit, 137 Answer creation model learning unit, 137' Understanding level creation model-inclusive answer creation model learning unit, 138 Answer creation model estimation unit, 138' Understanding level creation model-inclusive answer creation model estimation unit

Claims

1. An information processing device comprising: a question acquisition unit that acquires a user's question; an operation log acquisition unit that acquires the user's operation log; and an answer estimation unit that estimates an answer to the question using a large-scale language model based on the question and the operation log.

2. The information processing device of claim 1, wherein the question is a question about a specified application program, the operation log is based on operations performed when using the specified application program, and the information processing device further includes an understanding level estimation unit that estimates a level of understanding of the specified application program related to the question based on the question and the operation log, and the answer estimation unit estimates an answer to the question using the large-scale language model based on the question, the operation log, and the level of understanding.

3. An information processing device as described in claim 2, further comprising an understanding creation model learning unit that learns an understanding creation model that takes the question and the operation log as input and outputs the understanding level based on the past question, the past operation log, and the past understanding level, and the understanding level estimation unit uses the understanding creation model to estimate the understanding level based on the question and the operation log.

4. The information processing device according to claim 3, further comprising an answer creation model learning unit that learns an answer creation model that takes the question, the operation log, and the level of understanding as input and outputs the answer based on the past question, the past operation log, the past level of understanding, and the past answer, and the answer estimation unit uses the answer creation model to estimate the answer based on the question and the operation log and the level of understanding estimated by the understanding estimation unit from the question and the operation log.

5. The information processing device described in claim 3, wherein the past level of understanding is natural language information that expresses the level of understanding of the question recognized through interaction with the user, and is input by the provider of the specified application program that generates an answer to the question based on the past question and the past operation log.

6. The information processing device described in claim 4, wherein the answer to the question estimated by the answer estimation unit is associated with the question and registered in a database together with the operation log and the level of understanding, and the understanding level creation model learning unit re-learns the understanding level creation model based on the question, the operation log, and the level of understanding registered in the database.

7. The information processing device according to claim 6, wherein the answer creation model learning unit re-learns the answer creation model based on the questions, the operation logs, the understanding levels, and the answers registered in the database.

8. The information processing device according to claim 2, further comprising an answer creation model learning unit that outputs the degree of understanding based on an input consisting of the question and the operation log, based on the past question, the past operation log, the past degree of understanding, and the past answer, and then learns an answer creation model that outputs the answer based on an input consisting of the question and the operation log plus the output degree of understanding, wherein the answer estimation unit uses the answer creation model to estimate the degree of understanding from the question and the operation log, and then estimates the answer based on the estimated degree of understanding in addition to the question and the operation log.

9. The information processing device according to claim 2, wherein the predetermined application program includes a predictive analysis application program.

10. An information processing method including: a question acquisition process for acquiring a user's question; an operation log acquisition process for acquiring an operation log of the user; and an answer estimation process for estimating an answer to the question using a large-scale language model based on the question and the operation log.

11. A program that causes a computer to function as a question acquisition unit that acquires a user's question, an operation log acquisition unit that acquires the user's operation log, and an answer estimation unit that estimates an answer to the question using a large-scale language model based on the question and the operation log.

Images