Information processing device, information processing method, and program

Abstract

An information processing device according to one aspect of the present disclosure comprises a generation unit that generates display information for displaying, on a display device, one or more analysis results in an output format output by a language model by sequentially inputting, into the language model, one or more prescribed analysis instructions with respect to an inference result of a machine learning model and an output format of one or more analysis results for each of the one or more analysis instructions.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] In recent years, various events have been predicted using machine learning models, and the prediction results have been utilized. However, because the prediction process using machine learning models is a black box, a method called SHAP (SHapley Additive exPlanations) (Non-Patent Document 1) has been used to explain the prediction results.

[0003] Scott Lundberg, Su-In Lee, A Unified Approach to Interpreting Model Predictions, arXiv:1705.07874 [cs.AI].

[0004] However, if you want to analyze the basis (reason) behind the prediction results of a machine learning model and countermeasures for those prediction results, advanced knowledge of data analysis is required, and it takes a lot of time and effort.

[0005] The present disclosure has been made in consideration of the above points, and aims to provide a technology that can analyze the prediction results of a machine learning model.

[0006] An information processing device according to one aspect of the present disclosure has a generation unit that generates display information for displaying, on a display device, one or more analysis results in the output formats output from the language model by sequentially inputting one or more predetermined analysis instructions for the inference results of a machine learning model and one or more output formats of the analysis results for each of the one or more analysis instructions into the language model.

[0007] A technology is provided that can analyze the prediction results of machine learning models.

[0008] 1 is a diagram illustrating an example of the overall configuration of a prediction result analysis system according to an embodiment of the present invention; FIG. 2 is a diagram illustrating an example of the functional configuration of a prediction result analysis system according to an embodiment of the present invention; FIG. 3 is a diagram illustrating an example of first analysis target data; FIG. 4 is a diagram illustrating an example of second analysis target data; FIG. 5 is a sequence diagram illustrating an example of an operation of a prediction result analysis system according to an embodiment of the present invention; FIG. 6 is a diagram illustrating an example of prior knowledge; FIG. 7 is a diagram illustrating an example of an analysis target selection screen; FIG. 8 is a diagram illustrating an example of first instruction content; FIG. 9 is a diagram illustrating an example of second instruction content; FIG. 10 is a diagram illustrating an example of third instruction content; FIG. 11 is a diagram illustrating an example of language model output; FIG. 12 is a diagram illustrating an example of an analysis result screen.

[0009] An embodiment of the present invention will be described in detail below with reference to the drawings. In the following embodiment, a prediction result analysis system 1 will be described that, under a situation where a prediction result of a machine learning model that predicts a solution to a certain target problem (task), a confidence level of the prediction result, and a SHAP value for the prediction result are obtained, performs various analyses of the prediction result using a language model such as a large language model (LLM).

[0010] Hereinafter, a machine learning model that predicts a solution to a certain target problem (task) will be referred to as a "prediction model." A prediction model is represented by a function that takes one or more explanatory variables as input and outputs a predicted value of a dependent variable and its confidence level as a prediction result. The SHAP value refers to the degree of contribution of each explanatory variable to the prediction result of the prediction model (i.e., the predicted value of the dependent variable output by the prediction model). The explanatory variables are also sometimes called "features," etc.

[0011] While the problem (task) targeted in the following embodiments is not limited to a specific problem, as an example, it is assumed that the problem is to predict whether or not a customer (member) of a company that operates a retail business of a certain product will cancel their membership. Furthermore, the analysis of the prediction result involves analyzing the reasons and causes for the prediction result indicating whether or not a customer will cancel their membership, as well as countermeasures for the prediction result.

[0012] However, the above problem is merely an example, and is not limited to predicting whether a customer (member) of a company operating a retail business will cancel their membership, and the following embodiments can be applied to any problem. For example, it is possible to predict purchasing behavior such as whether a customer will purchase a certain product on an e-commerce site and whether they will simultaneously purchase other products, or to predict whether an employee will resign from a company.

[0013] Furthermore, the analysis of the above prediction results is just one example, and is not limited to analyzing the reasons, causes, and countermeasures for the prediction results indicating whether or not a customer (member) will withdraw from membership; the following embodiments can be applied to any analysis.

[0014] <Example of Overall Configuration of Prediction Result Analysis System 1> An example of the overall configuration of the prediction result analysis system 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the overall configuration of the prediction result analysis system 1 according to this embodiment.

[0015] 1, a prediction result analysis system 1 according to this embodiment includes one or more user terminals 10 and a prediction result analysis device 20. Each user terminal 10 and the prediction result analysis device 20 are communicatively connected via a communication network 30 including, for example, the Internet.

[0016] The user terminal 10 is a terminal of any type used by a user who wishes to obtain the results of a predetermined analysis of the prediction results of a prediction model (such as an analysis of the reasons and causes for the prediction results, and countermeasures for the prediction results). The user terminal 10 can be realized, for example, by a PC (personal computer), a smartphone, a tablet terminal, a wearable device, or the like.

[0017] The prediction result analysis device 20 is an information processing device that uses a language model such as a large-scale language model to perform a predetermined analysis of the prediction results of a prediction model (analysis of the reasons and causes for obtaining the prediction results, countermeasures for the prediction results, etc.) The prediction result analysis device 20 can be realized by, for example, a general-purpose server, a PC, etc.

[0018] The overall configuration example of the prediction result analysis system 1 shown in Figure 1 is an example and is not limited to this. For example, the prediction result analysis device 20 may be realized by a system consisting of multiple devices including a general-purpose server, etc. Also, for example, a part of the prediction result analysis device 20 may be realized by a cloud server, etc. Also, for example, the user terminal 10 and the prediction result analysis device 20 may be configured integrally.

[0019] <Example of Functional Configuration of Prediction Result Analysis System 1> An example of the functional configuration of the prediction result analysis system 1 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating an example of the functional configuration of the prediction result analysis system 1 according to this embodiment.

[0020] 2, the user terminal 10 according to this embodiment has a UI control unit 101. The UI control unit 101 is realized, for example, by a process in which one or more programs installed in the user terminal 10 are executed by a processor such as a CPU (Central Processing Unit). In particular, the UI control unit 101 may be realized, for example, by a process in which a web browser, a dedicated application program, or the like is executed by a processor such as a CPU.

[0021] The UI control unit 101 accepts various operations by the user and displays various screens to the user. In addition, in response to the acceptance of various operations by the user, the UI control unit 101 transmits various requests corresponding to the operations (e.g., an analysis start request or an analysis request, which will be described later) to the prediction result analysis device 20, and receives responses to the requests from the prediction result analysis device 20.

[0022] The user can perform various operations using various input devices such as a keyboard, a mouse, a touch panel, etc. Furthermore, various screens for the user are displayed on various display devices such as a display.

[0023] <Prediction Result Analysis Device 20> As shown in FIG. 2 , the prediction result analysis device 20 according to this embodiment includes a request receiving unit 201, an analysis unit 202, a UI generation unit 203, and a response unit 204. These units are implemented, for example, by a processor, such as a CPU, executing one or more programs installed in the prediction result analysis device 20. The prediction result analysis device 20 according to this embodiment also includes an analysis target data storage unit 205. The analysis target data storage unit 205 is implemented, for example, by a storage area of ​​an auxiliary storage device, such as a hard disk drive (HDD), a solid state drive (SSD), or flash memory. However, the analysis target data storage unit 205 may also be implemented, for example, by a storage area of ​​a storage device connected to the prediction result analysis device 20 via a communication network 30.

[0024] The request receiving unit 201 receives various requests (for example, an analysis start request, an analysis request, etc.) sent from the user terminal 10 .

[0025] In response to the request received by the request receiving unit 201, the analysis unit 202 uses the language model 1000 to analyze the analysis target data (first analysis target data and second analysis target data described below) stored in the analysis target data storage unit 205. At this time, the analysis unit 202 inputs the first analysis target data and the second analysis target data, their descriptions, and the statistics of each explanatory variable as prior knowledge to the language model 1000, and then sequentially inputs instruction contents to the language model 1000, including analysis instructions for a predetermined analysis and a format for outputting analysis results corresponding to the analysis instructions. That is, of the prior knowledge, analysis instructions, and formats that are components of a prompt, which is an instruction given to the language model 1000, the analysis unit 202 first inputs the prior knowledge to the language model 1000, and then sequentially inputs one or more analysis instructions and formats to the language model 1000. This makes it possible to expect that highly accurate analysis results for the first analysis target data and the second analysis target data can be obtained as the output of the language model 1000.

[0026] Here, as the language model 1000, for example, a machine learning model called a large-scale language model (LLM) or the like is used. In the language model 1000, when a command called, for example, a prompt or the like is input, an output according to the command is generated. Such a language model 1000 or a service applying the same is also called generative AI (artificial intelligence) or generative AI. Note that the language model 1000 may be stored in a storage area such as an auxiliary storage device included in the prediction result analysis device 20, or may be stored in a storage area such as an auxiliary storage device included in a device different from the prediction result analysis device 20. Furthermore, the language model 1000 may be available as, for example, a cloud service or the like.

[0027] The UI generation unit 203 generates display information (e.g., display information such as an analysis target selection screen and an analysis result screen, which will be described later) for displaying the analysis results by the analysis unit 202 and other information on the user terminal 10. If the UI control unit 101 of the user terminal 10 is implemented by a web browser or the like, the UI generation unit 203 may generate screen definition information, such as HTML (HyperText Markup Language) or CSS (Cascading Style Sheets), as display information. On the other hand, if the UI control unit 101 of the user terminal 10 is implemented by a dedicated application program or the like and a screen is generated by that application program, the UI generation unit 203 may generate information necessary for generating the screen (e.g., information set in variable parts such as lists, pull-down menus, and text display fields) as display information.

[0028] The response unit 204 transmits a response including the display information generated by the UI generation unit 203 to the user terminal 10 .

[0029] The analysis target data storage unit 205 stores first analysis target data including labels such as member IDs, objective variables, explanatory variables, and prediction results of a prediction model, and second analysis target data including labels such as member IDs and the contribution of the explanatory variables to the prediction results (SHAP values). The first analysis target data and second analysis target data are obtained in advance and stored in the analysis target data storage unit 205. Examples of the first analysis target data and second analysis target data will be described later.

[0030] <<Example of First Data to be Analyzed>> An example of the first data to be analyzed will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the first data to be analyzed. Hereinafter, the first data to be analyzed 2100 shown in Fig. 3 will also be referred to as "data1".

[0031] As shown in Figure 3, the first analysis target data 2100 is composed of one or more records, and each record includes a label that identifies the record, the objective variable that is the target of prediction by the prediction model, the explanatory variables (features) used for prediction by the prediction model, and the prediction result of the prediction model.

[0032] Each record constituting the first analysis target data 2100 shown in FIG. 3 includes "member_id" as a label. It also includes "member_sts" as a target variable. It also includes "membership_period," "period_from_last_login," "inspmem_access_count," "service_access_count," and "servicelog_access_count" as explanatory variables (features). It also includes "predicted_member_sts" and "score" as prediction results of the prediction model.

[0033] Here, the label "member_id" represents the member ID. The objective variable "member_sts" represents the actual member status (i.e., whether or not the member is a member), with a value of 0 representing continued membership and a value of 1 representing withdrawal. The explanatory variable "membership_period" represents the number of days of membership continuation, "period_from_last_login" represents the number of days elapsed since the last login, "inspmem_access_count" represents the number of annual accesses to the member information change page, "service_access_count" represents the number of annual accesses to the purchase amount statement page, and "servicelog_access_count" represents the number of annual accesses to the purchase product details page. Furthermore, the prediction result of the prediction model, "predicted_member_sts", represents the member status predicted by the prediction model (0 means continued membership, 1 means withdrawal), and "score" represents the confidence level of the member status predicted by the prediction model.

[0034] The confidence level is a value between 0 and 1, and in the prediction model, just like a binary classification problem, it is predicted in a two-dimensional vector format, for example, (confidence level that member status is 0, confidence level that member status is 1) (where confidence level that member status is 0 + confidence level that member status is 1 = 1). In this case, for example, if "confidence level that member status is 0" > "confidence level that member status is 1 = 1", then "predicted_member_sts" is set to 0, and if not, "predicted_member_sts" is set to 1.

[0035] <<Example of Second Data to be Analyzed>> An example of the second data to be analyzed will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the second data to be analyzed. Hereinafter, the second data to be analyzed 2200 shown in Fig. 4 will also be referred to as "data2".

[0036] As shown in FIG. 4, the second data to be analyzed 2200 is made up of one or more records, and each record includes a label that identifies the record and the contribution (SHAP value) of the explanatory variable to the prediction result.

[0037] Each record constituting the second analysis target data 2200 shown in Fig. 4 includes a label "member_id." Also, "membership_period," "period_from_last_login," "inspmem_access_count," "service_access_count," and "servicelog_access_count" are included as the contribution of the explanatory variables to the prediction result.

[0038] Here, the label "member_id" represents the member ID. Furthermore, the contributions "membership_period", "period_from_last_login", "inspmem_access_count", "service_access_count", and "servicelog_access_count" represent the contributions (SHAP values) of the explanatory variables with the same names.

[0039] <Example of Operation of Prediction Result Analysis System 1> An example of operation of the prediction result analysis system 1 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a sequence diagram showing an example of operation of the prediction result analysis system 1 according to this embodiment.

[0040] First, the UI control unit 101 of the user terminal 10 accepts an analysis start operation by the user (step S101). Here, the analysis start operation refers to an operation for starting the analysis of the first analysis target data and the second analysis target data. Examples of the analysis start operation include an operation for transitioning to an analysis target selection screen, which will be described later.

[0041] Next, upon receiving the analysis start operation, the UI control unit 101 of the user terminal 10 transmits an analysis start request to the prediction result analysis device 20 (step S102).

[0042] The request receiving unit 201 of the prediction result analysis device 20 receives the analysis start request transmitted from the user terminal 10 (step S103).

[0043] Next, the analysis unit 202 of the prediction result analysis device 20 inputs the first analysis target data and the second analysis target data stored in the analysis target data storage unit 205 as prior knowledge to the language model 1000 (step S104). Here, the analysis unit 202 inputs, for example, prior knowledge 3000 shown in FIG. 6 to the language model 1000. The prior knowledge 3000 shown in FIG. 6 is information representing the first analysis target data 2100 and the second analysis target data 2200 and their explanations, and is one of the components of the prompt provided to the language model 1000. The prior knowledge 3000 shown in FIG. 6 includes an analysis target data explanation field 3100, a first analysis target data setting field 3200, and a second analysis target data setting field 3300.

[0044] The analysis target data description field 3100 lists the label, the objective variable, the explanatory variables, a description of the prediction result, and the like. The analysis target data description field 3100 also lists the median of each explanatory variable. However, the median is only an example of a statistic, and other statistic values ​​such as the mean, maximum value, minimum value, and mode may also be used. The statistic values ​​of each explanatory variable may be calculated in advance, or may be calculated by the analysis unit 202 when inputting prior knowledge to the language model 1000 in step S104 above.

[0045] The first analysis target data 2100 is set and expanded in the first analysis target data setting field 3200. Similarly, the second analysis target data 2200 is set and expanded in the second analysis target data setting field 3300.

[0046] By inputting the prior knowledge 3000 shown in FIG. 6 into the language model 1000, the language model 1000 can acquire the labels, the objective variables, the explanatory variables, the prediction results of the prediction model, the contribution of each explanatory variable to the prediction results, the statistics of each explanatory variable, etc. as prior knowledge for analysis.

[0047] Next, the UI generation unit 203 of the prediction result analysis device 20 generates display information for an analysis target selection screen (described later) (step S105). The analysis target selection screen is a screen for selecting members to be analyzed.

[0048] Next, the response unit 204 of the prediction result analysis device 20 transmits a response including display information of the analysis target selection screen to the user terminal 10 (step S106).

[0049] When the UI control unit 101 of the user terminal 10 receives the display information for the analysis target selection screen, it displays the analysis target selection screen based on this display information (step S107). Here, the UI control unit 101 displays, for example, the analysis target selection screen 4000 shown in FIG. 7. The analysis target selection screen 4000 shown in FIG. 7 is a screen for selecting members to be analyzed. The analysis target selection screen 4000 shown in FIG. 7 includes an analysis target selection field 4100. The analysis target selection field 4100 is a display field in which members to be analyzed can be selected, and the user can select a member corresponding to each row in the analysis target selection field 4100. Hereinafter, the operation of selecting members to be analyzed will be referred to as the "analysis target selection operation."

[0050] Each row in the analysis target selection column 4100 represents a member, and displays the member ID, membership duration, number of days since last login, number of annual accesses to the member information change page, number of annual accesses to the purchase amount statement page, number of annual accesses to the purchased product details page, and cancellation risk of the member corresponding to that row. Here, the member ID of each row is the value of the label "member_id" of the member corresponding to that row. Furthermore, the membership duration is the value of the member's explanatory variable "membership_period," the number of days since the last login is the value of the member's explanatory variable "period_from_last_login," the number of annual accesses to the member information change page is the value of the member's explanatory variable "inspmem_access_count," the number of annual accesses to the purchase amount statement page is the value of the member's explanatory variable "service_access_count," and the number of annual accesses to the purchased product details page is the value of the member's explanatory variable "servicelog_access_count." Furthermore, the cancellation risk is the value of "score" when the value of the member's predicted result "predicted_member_sts" is 1, and is the value obtained by subtracting the value of "score" from 1 otherwise.

[0051] Hereinafter, it is assumed that the user has performed an analysis target selection operation on the analysis target selection screen 4000 shown in FIG.

[0052] The UI control unit 101 of the user terminal 10 accepts an analysis target selection operation by the user (step S108).

[0053] Next, when the UI control unit 101 of the user terminal 10 receives the analysis target selection operation, it sends an analysis request including the member ID of the member selected by this analysis target selection operation to the prediction result analysis device 20 (step S109).

[0054] The request receiving unit 201 of the prediction result analysis device 20 receives an analysis request sent from the user terminal 10 (step S110).

[0055] Next, the analysis unit 202 of the prediction result analysis device 20 sequentially inputs one or more instruction contents to the language model 1000 (step S111). Below, as an example, a case will be described in which three instruction contents (a first instruction content, a second instruction content, and a third instruction content) are sequentially input to the language model 1000. In this case, the analysis unit 202 sequentially inputs each instruction content to the language model 1000 by executing the following steps 1 to 3. Note that each instruction content includes an analysis instruction for a predetermined analysis and a format (form) for outputting the analysis result for that analysis instruction.

[0056] Step 1: First, the analysis unit 202 inputs a first instruction content to the language model 1000. Here, the analysis unit 202 inputs, for example, the first instruction content 5000 shown in FIG. 8 to the language model 1000. The first instruction content 5000 shown in FIG. 8 is information representing an analysis instruction for the reason why a prediction result was obtained by the prediction model and a format for outputting the analysis result in response to the analysis instruction, and is one of the components of a prompt to be provided to the language model 1000. The first instruction content 5000 shown in FIG. 8 includes an analysis instruction column 5100 and a format column 5200.

[0057] An analysis instruction for the reason why the prediction result of the prediction model was obtained is entered in the analysis instruction field 5100. Note that the {target} in the analysis instruction field 5100 is set to the member ID included in the analysis request.

[0058] The format column 5200 describes the format used when outputting the analysis results in response to an instruction to analyze the reasons why the prediction results of the prediction model were obtained.

[0059] 8 is input to the language model 1000, the first analysis target data 2100 and the second analysis target data 2200 are analyzed by the language model 1000 in accordance with the analysis instructions written in the analysis instruction field 5100, and the analysis results are output in accordance with the format written in the format field 5200. In particular, in the example shown in Fig. 8, the analysis instruction field 5100 contains an instruction to have the language model 1000 infer elements enclosed in [ ] in the format, and the positions in the format written in the format field 5200 where elements are set by the language model 1000 are clearly indicated by the symbols [ ]. This achieves both uniformity and flexibility in the format.

[0060] Step 2: Next, the analysis unit 202 inputs a second instruction content to the language model 1000. Here, the analysis unit 202 inputs, for example, the second instruction content 6000 shown in FIG. 9 to the language model 1000. The second instruction content 6000 shown in FIG. 9 is information representing an analysis instruction to formulate a hypothesis and analyze the cause of the prediction result of the prediction model and a format for outputting the analysis result in response to the analysis instruction, and is one of the components of a prompt provided to the language model 1000. The second instruction content 6000 shown in FIG. 9 includes an analysis instruction field 6100 and a format field 6200. The cause refers to a reason that negatively influenced the prediction result among the reasons for the prediction result. The negative influence refers to an influence on the predicted value when predicting the value of the objective variable, which works in a direction other than the desired predicted value (e.g., an influence that works in a direction toward a predicted value of 1 for the membership status).

[0061] The analysis instruction field 6100 contains an analysis instruction for formulating hypotheses regarding the causes of the prediction results of the prediction model and performing an analysis. Note that {hyp} in the analysis instruction field 6100 is set to a predetermined number of hypotheses (or a number of hypotheses specified by the user).

[0062] The format column 6200 describes the format used when outputting the analysis results in response to an analysis instruction that causes a hypothesis to be formulated and analyzed regarding the cause of the prediction result obtained from the prediction model.

[0063] When the second instruction content 6000 shown in Figure 9 is input into the language model 1000, the first analysis target data 2100 and the second analysis target data 2200 are analyzed by the language model 1000 in accordance with the analysis instructions written in the analysis instruction column 6100, and the analysis results are output in accordance with the format written in the format column 6200.

[0064] Step 3: Then, the analysis unit 202 inputs a third instruction content to the language model 1000. Here, the analysis unit 202 inputs, for example, a third instruction content 7000 shown in FIG. 10 to the language model 1000. The third instruction content 7000 shown in FIG. 10 is information representing an analysis instruction for a countermeasure against the prediction result of the prediction model and a format for outputting the analysis result in response to the analysis instruction, and is one of the components of a prompt provided to the language model 1000. The third instruction content 7000 shown in FIG. 10 includes an analysis instruction column 7100 and a format column 7200.

[0065] The analysis instruction field 7100 contains an instruction to analyze measures to prevent withdrawal as a countermeasure to the prediction results of the prediction model. The format field 7200 contains a format for outputting the analysis results in response to the instruction to analyze measures to prevent withdrawal.

[0066] When the third instruction content 7000 shown in Figure 10 is input into the language model 1000, the first analysis target data 2100 and the second analysis target data 2200 are analyzed by the language model 1000 in accordance with the analysis instructions written in the analysis instruction column 7100, and the analysis results are output in accordance with the format written in the format column 7200.

[0067] As described above, instruction contents including analysis instructions and formats relating to a predetermined analysis are sequentially input to the language model 1000. As a result, analysis results corresponding to the analysis instructions are sequentially acquired from the language model 1000.

[0068] 11 shows an example of output when the first instruction content 5000 to the third instruction content 7000 are sequentially input to the language model 1000. The output example 1000 shown in FIG. 11 includes an analysis result 8100 for the first instruction content 5000, an analysis result 8200 for the second instruction content 6000, and an analysis result 8300 for the third instruction content 7000. Note that the analysis results 8100, 8200, and 8300 are sequentially output from the language model 1000 in response to the first instruction content 5000 to the third instruction content 7000 being sequentially input to the language model 1000.

[0069] Next, the UI generation unit 203 of the prediction result analysis device 20 generates display information for an analysis result screen (described later) (step S112). The analysis result screen is a screen on which the analysis results output from the language model 1000 in step S111 are displayed.

[0070] Next, the response unit 204 of the prediction result analysis device 20 transmits a response including display information of the analysis result screen to the user terminal 10 (step S113).

[0071] Upon receiving the display information for the analysis result screen, the UI control unit 101 of the user terminal 10 displays the analysis result screen based on this display information (step S114). Here, for example, if an analysis target selection operation is performed in step S107 above, selecting member ID "0023456" as the analysis target, and analysis results 8100 to 8300 shown in FIG. 11 are obtained, the UI control unit 101 displays the analysis result screen 9000 shown in FIG. 12. The analysis result screen 9000 shown in FIG. 12 is a screen on which the analysis results output from the language model 1000 in step S111 above are displayed. The analysis result screen 9000 shown in FIG. 12 includes a churn risk display field 9100, a SHAP value display field 9200, and analysis result display fields 9300 to 9500.

[0072] The churn risk display field 9100 displays a churn risk of "90%" for the member with member ID "0023456." The SHAP value display field 9200 displays the SHAP values ​​of each explanatory variable (membership duration, number of days since last login, number of annual accesses to the member information change page, number of annual accesses to the purchase amount statement page, and number of annual accesses to the purchased product details page) for the predicted results for the member with member ID "0023456."

[0073] The analysis result display fields 9300 to 9500 display the analysis results output from the language model 1000 in step S111 above. Specifically, the analysis result display field 9300 displays the analysis result 8100 shown in FIG. 11 , the analysis result display field 9400 displays the analysis result 8200 shown in FIG. 11 , and the analysis result display field 9500 displays the analysis result 8300 shown in FIG. 11 . In this way, each analysis result display field included in the analysis result screen displays the analysis result for each instruction in the format included in the instruction. This allows the user to easily check the analysis results, such as the reasons and causes of the prediction results, countermeasures, etc.

[0074] <Modifications> Modifications of this embodiment will now be described.

[0075] Variation 1 In the above embodiment, prior knowledge is input to the language model 1000 in step S103 of FIG. 5 , and then the member ID to be analyzed is selected in step S108. However, this is not limited to this. For example, prior knowledge may be input to the language model 1000 after the member ID to be analyzed is selected on the analysis target selection screen. Furthermore, in this case, when each instruction content is sequentially input to the language model 1000 in step S111 of FIG. 5 , the prior knowledge may be input to the language model 1000 together with the first instruction content. Alternatively, the prior knowledge may be input to the language model 1000 at any time before the first instruction content is input to the language model 1000.

[0076] Variation 2 In the above embodiment, prompts such as prior knowledge and instruction content are set in advance. However, for example, all or part of the analysis instructions and format included in the prior knowledge and instruction content may be specified by the user.

[0077] Variation 3: In the above embodiment, the records of each member ID are analyzed for each member ID. However, instead of analyzing each member ID, for example, members may be classified into multiple clusters based on member attributes, and then analyzed for each cluster. Specifically, the above embodiment may be applied by classifying members into multiple clusters based on their attributes (e.g., age, membership history, occupation, gender, family structure, residential area, etc.) using statistical processing or machine learning models, and treating each cluster as a virtual member (hereinafter referred to as a "virtual member"). The explanatory variable values ​​for each virtual member may be representative values ​​(e.g., mean, median, mode, maximum, minimum, etc.) of the explanatory variables corresponding to the members belonging to the cluster represented by that virtual member. Similarly, the prediction results of the prediction model for each virtual member and the contribution of the explanatory variables to the prediction results may be representative values ​​of the prediction results of the prediction model for the members belonging to the cluster represented by that virtual member and the contribution of the explanatory variables to the prediction results, respectively.

[0078] At this time, in addition to the above-mentioned instruction to analyze the virtual member, the language model 1000 may be instructed to, for example, provide an overview of the virtual member (i.e., an overview of the cluster represented by the virtual member), estimate the attributes of the virtual member, generate an image such as an avatar representing the virtual member, etc. Furthermore, with the aim of increasing the virtual member's motivation to continue membership and purchase, etc., the language model 1000 may be instructed to generate methods of in-store or online interaction with the virtual member, response scripts, email text, etc. In addition, when instructing the generation of methods of instructing the virtual member or response scripts, the language model 1000 may be instructed to generate one or more responses expected from the virtual member and branched approaches and response scripts corresponding to each response. This allows the output of the language model 1000 in response to these instructions to be displayed on the analysis result screen.

[0079] <Summary> As described above, the prediction result analysis system 1 according to this embodiment analyzes the inference results of a machine learning model using a language model 1000 such as a large-scale language model, so that even a user who does not have specialized knowledge of data analysis can easily obtain the analysis results of the inference results.

[0080] Furthermore, the prediction result analysis system 1 according to this embodiment separates prompts input to the language model 1000 into prior knowledge and instruction content, inputs the prior knowledge to the language model 1000, and then sequentially inputs each instruction content to the language model 1000. In other words, the prior knowledge, analysis instruction, and format, which are components of the prompt, are structured as two parts: "prior knowledge" and "analysis instruction and format" (instruction content), and each instruction content is sequentially input to the language model 1000. This allows the language model 1000 to consider the context of each instruction content, which is expected to result in more accurate analysis results. Furthermore, since the prompt is structured into "prior knowledge" and one or more "instruction content consisting of analysis instruction and format," the maintainability of the prompt is also improved. Furthermore, since the analysis results for each instruction content are sequentially output from the language model 1000, it is easier to manage the analysis results, which are the answers from the language model 1000, and it is possible to easily configure the display information on the analysis result screen.

[0081] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0082] REFERENCE SIGNS LIST 1 Prediction result analysis system 10 User terminal 20 Prediction result analysis device 30 Communication network 101 UI control unit 201 Request reception unit 202 Analysis unit 203 UI generation unit 204 Response unit 205 Analysis target data storage unit 1000 Language model

Claims

1. An information processing device having a generation unit that generates display information for displaying, on a display device, one or more analysis results in the output format output from a language model by sequentially inputting one or more predetermined analysis instructions for the inference results of a machine learning model and one or more output formats of the analysis results for each of the one or more analysis instructions into the language model.

2. An information processing device having an analysis unit that inputs prior knowledge including the inference results of a machine learning model into a language model, and then sequentially inputs one or more predetermined analysis instructions for the inference results and one or more output formats of the analysis results for each of the one or more analysis instructions into the language model, thereby obtaining one or more analysis results of the output formats output from the language model.

3. The information processing device according to claim 2, wherein the output format includes predetermined symbols, and the analysis instructions include instructions for setting information contained in the prior knowledge in the language model at the location of the symbols.

4. An information processing method executed by a computer, comprising: a generation procedure for sequentially inputting one or more predetermined analysis instructions for the inference results of a machine learning model and one or more output formats of the analysis results for each of the one or more analysis instructions into a language model, thereby generating display information for displaying one or more analysis results in the output formats output from the language model on a display device.

5. A program that causes a computer to execute a generation procedure for generating display information for displaying, on a display device, one or more analysis results in the output format output from a language model by sequentially inputting one or more predetermined analysis instructions for the inference results of a machine learning model and one or more output formats of the analysis results for each of the one or more analysis instructions into the language model.

Images