Specific program, specific method, and information processing device

Abstract

To build an AI model that incorporates the domain knowledge of stakeholders and can make decisions similar to those of a human. [Solution] The information processing device 100 acquires the attributes of the training data. The information processing device 100 acquires a document relating to the decision-making conditions for a specific attribute of the user. The information processing device 100 outputs the probability of occurrence of the tokens relating to the attributes by inputting a prompt containing the attributes of the training data and tokens from the document relating to the decision-making conditions into a large-scale language model. Based on the probability of occurrence of the output tokens, the information processing device 100 identifies the labels to be learned by the machine learning model from the training data.

Description

Technical Field

[0001] The present invention relates to a specific program and the like.

Background Art

[0002] AI (Artificial Intelligence) systems for supporting decision-making such as loan reviews and personnel recruitment are being utilized, and the development of AI models that make judgments based on the domain knowledge of stakeholders such as loan reviewers and recruitment staff is required.

[0003] Here, stakeholders refer to people involved in the AI system, such as customers without expertise in machine learning, experts engaged in actual operations, and auditors of audit organizations. Also, domain knowledge refers to the knowledge that stakeholders possess regarding the conditions of performance metrics and fairness metrics, as well as the knowledge that serves as the basis for judgments on individual cases.

[0004] Performance metrics and fairness metrics are indicators for measuring the prediction performance of AI. In the case of an AI for loan review, the correspondence between the true value and the predicted value of the AI model regarding the approval of a loan forms a confusion matrix as shown in FIG. 16.

[0005] FIG. 16 is a diagram for explaining performance metrics and fairness metrics. When the predicted value of the AI (AI model) is "approved" and the true value is also "approved", it is a true positive. A true positive indicates a case where an applicant who should have a loan approved is approved. A true positive is denoted as TP (True Positive).

[0006] When the predicted value of the AI is "approved" and the true value is "not approved", it is a false positive. A false positive indicates a case where an applicant who should not have a loan approved is approved. A false positive is denoted as FP (False Positive).

[0007] A false negative occurs when the AI's prediction is "rejected" and the true value is "accepted." A false negative indicates a case where an applicant who should have been approved for the loan was rejected. False negatives are indicated by FN (False Negative).

[0008] If the AI's prediction is "rejected" and the true value is also "rejected," it is considered a true negative. A true negative indicates a case where an applicant who should not have received the loan was rejected. A true negative is indicated by TN (True Negative).

[0009] Performance metrics include indicators such as Precision, Recall, and Accuracy. For example, Accuracy is defined as shown in equation (1).

[0010] Accuracy=(TP+TN) / (TP+FN+FP+TN)...(1)

[0011] Fairness metrics, such as Parity, are indicators that evaluate whether the AI ​​is not biased towards specific attributes (e.g., gender). For example, Accuracy Parity is expressed as the odds of accuracy for men and women and is defined as shown in equation (2).

[0012] Accuracy Parity=Accuracy Parity(Female) / Accuracy Parity(Male)...(2)

[0013] The above explains the performance metrics and fairness metrics for domain knowledge.

[0014] AI model tuning techniques can mimic domain-based decision-making. For example, AI model tuning techniques include hyperparameter tuning, learning with performance metrics and fairness metrics as objective functions, and learning using weighting of classes and attributes.

[0015] In addition, there are conventional techniques for discovering performance matrices desired by stakeholders in binary classification problems. These conventional techniques present stakeholders with two different confusion matrices multiple times and identify performance metrics based on their real-time responses. [Prior art documents] [Patent Documents]

[0016] [Patent Document 1] Special Publication No. 2024-508502 [Patent Document 2] Japanese Patent Publication No. 2023-162816 [Patent Document 2] U.S. Patent Application Publication No. 2019 / 0220705 Specification [Overview of the project] [Problems that the invention aims to solve]

[0017] However, the conventional technologies described above have a problem in that they cannot incorporate the domain knowledge of stakeholders and build AI models that closely resemble human judgment.

[0018] In conventional technologies, when trying to discover performance metrics focused on specific attributes based on stakeholder opinions, it is impossible to identify performance metrics if the attributes included in those opinions are unclear. For example, if a stakeholder in a loan application AI has the opinion that "not only the applicant's age but also their living situation should be considered," and there is no attribute that matches "living situation," then it is impossible to identify performance metrics.

[0019] In one aspect, the present invention aims to provide a specific program, a specific method, and an information processing device that can incorporate the domain knowledge of stakeholders and construct an AI model capable of making human-like judgments. [Means for solving the problem]

[0020] In the first proposal, the computer executes the following processes. The computer acquires the attributes of the training data. The computer acquires a document regarding the decision-making conditions for specific attributes of the user. The computer inputs a prompt including the attributes of the training data and the tokens of the document regarding the decision-making conditions into the large language model, and outputs the appearance probability of the tokens regarding the attributes. The computer specifies the label of the learning target of the machine learning model from the training data based on the appearance probability of the output tokens.

Advantages of the Invention

[0021] Incorporate the domain knowledge of stakeholders and build an AI model that can make human-like judgments.

Brief Description of the Drawings

[0022] [Figure 1] Figure 1 is a diagram showing the results by a simple solution. [Figure 2] Figure 2 is a diagram showing an example of the data structure of the training dataset and the attribute list. [Figure 3] Figure 3 is a diagram showing an example of the user's opinion. [Figure 4] Figure 4 is a flowchart showing the processing procedure of the mapping process. [Figure 5] Figure 5 is a diagram supplementarily explaining step S10. [Figure 6] Figure 6 is a diagram showing an example of the data structure of the probability table. [Figure 7] Figure 7 is a diagram supplementarily explaining steps S13 to S15. [Figure 8] Figure 8 is a flowchart showing the processing procedure of the ambiguity score calculation process. [Figure 9] Figure 9 is a flowchart showing the processing procedure of the additional question process. [Figure 10] Figure 10 is a diagram supplementarily explaining steps S32 to S34. [Figure 11] Figure 11 is a flowchart showing the processing procedure of the label specification process. [Figure 12] Figure 12 is a diagram that provides supplementary explanation for steps S41 and S42. [Figure 13] Figure 13 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 14] Figure 14 is a flowchart showing the processing procedure of the information processing device according to this embodiment. [Figure 15] Figure 15 shows an example of a computer hardware configuration that achieves similar functions to the information processing device described in the embodiment. [Figure 16] Figure 16 is a diagram illustrating performance metrics and fairness metrics. [Modes for carrying out the invention]

[0023] The following describes in detail, with reference to the drawings, embodiments of the specific program, specific method, and information processing device disclosed in this application. However, this invention is not limited by these embodiments. [Examples]

[0024] As described above, with conventional technology, when trying to discover performance metrics focused on specific attributes based on stakeholder opinions, it is impossible to identify performance metrics if the attributes included in those opinions are unclear. Without the ability to identify performance metrics, it becomes impossible to build an AI model that incorporates stakeholders' domain knowledge and can perform human-like judgments.

[0025] One simple solution for identifying attributes from stakeholder opinions is to use a large-scale language model (LLM). Hereafter, we will refer to large-scale language models as LLMs. For example, in this simple solution, an instruction such as "Please select attributes related to your opinion" is output to the LLM, and through dialogue between the stakeholders and the LLM, the attributes that match the stakeholders' opinions are revealed.

[0026] The dialogues between stakeholders and the LLM are defined as the following dialogues 1-1, 1-2, 1-3, and 1-4. Dialogue 1-1 (Stakeholder): I would like the loan application review AI to take into account not only the applicant's age but also their living situation. Dialogue 1-2 (LLM): Does "living situation" refer to "billing amount from two months ago", "billing amount from one month ago", "expenses from two months ago", "expenses from one month ago", "income", "employment status", and "housing status"? Dialogue 1-3 (Stakeholder): We would like to know the general spending patterns, personal information, and family situation. Dialogue 1-4 (LLM): So, does "living situation" refer to "billing amount from two months ago", "billing amount from one month ago", "expenses from two months ago", "expenses from one month ago", "income", "type of housing", and "family composition"?

[0027] Figure 1 shows the results of a simple solution. Graph G1 in Figure 1 is a two-dimensional graph showing the vector space when attribute names are vectorized within the LLM. Axis (1) of graph G1 corresponds to one of the two dimensions, and axis (2) corresponds to the other dimension.

[0028] In graph G1, the names of each attribute are positioned according to the values ​​of the vectors. The names of the attributes are "Billing amount 2 months ago", "Billing amount 1 month ago", "Expenses 2 months ago", "Expenses 1 month ago", "Income", "Employment type", "Housing type", "Family composition", and "Age".

[0029] Area A1 is the range of attributes estimated by the LLM through dialogue 1-2. Area A1 includes “Invoice amount two months ago”, “Invoice amount one month ago”, “Expenses two months ago”, “Expenses one month ago”, “Income”, “Employment type”, and “Housing type”.

[0030] Region A2 is the range of attributes estimated by LLM through dialogue 1-4. Region A2 contains This includes "billing amount from two months ago," "billing amount from one month ago," "expenses from two months ago," "expenses from one month ago," "income," "type of housing," and "family composition."

[0031] Area A3 represents the range of attributes considered by stakeholders. Area A3 includes "spending amount two months ago," "spending amount one month ago," "income," "employment status," "housing status," "age," and "family structure."

[0032] As shown in Figure 1, in a simple solution, the range of attributes in area A2 and area A3 are different, and even with repeated dialogue between stakeholders and the LLM, it is not possible to identify attributes that match the stakeholders' opinions. The state in which attributes that match the stakeholders' opinions can be identified is when the range of attributes in area A2 and area A3 are identical.

[0033] Next, the information processing device according to this embodiment will be described. The information processing device according to this embodiment will be referred to as "information processing device 100". Also, stakeholders will be referred to as "users". The information processing device 100 analyzes the user's ambiguous opinion using LLM and identifies attributes that match the opinion. The information processing device 100 ultimately generates training data with labels indicating acceptance or rejection that reflect the user's intentions.

[0034] For example, the information processing device 100 combines existing attributes to create new attributes in order to concretize ambiguous opinions. Furthermore, the information processing device 100 quantifies the ambiguity of an opinion using an ambiguity score and identifies specific attributes through additional questions.

[0035] In this embodiment, the attributes included in the training data are used as an attribute list. Multiple training data sets are referred to as "training datasets." Figure 2 shows an example of the data structure of a training dataset and an attribute list.

[0036] The training dataset 141 includes an ID to identify the training data and several attributes. In the example shown in Figure 2, the attributes include "Loan approval / rejection," "Investment," "Savings," "Income," "Employment status," "Occupation," "Education level," "Family structure," "Housing type," "Age," "Gender," "Billing amount one month ago," "Billing amount two months ago," "Expenses one month ago," and "Expenses two months ago." At this stage, the training dataset 141 does not have labels assigned to it, which will be assigned at the end of the processing described later.

[0037] The information processing device 100 retrieves the attributes of the training dataset 141 and generates an attribute list 10. The attribute list 10 includes the following attributes: "Loan approval / rejection", "Investment", "Savings", "Income", "Employment status", "Occupation", "Education level", "Family structure", "Housing type", "Age", "Gender", "Billing amount one month ago", "Billing amount two months ago", "Expenses one month ago", and "Expenses two months ago". The combinations of attributes set in the attribute list 10 become the combinations of existing attributes.

[0038] The information processing device 100 receives input from the user, and for ambiguous attributes included in the opinion that are not included in the attribute list, it associates such ambiguous attributes with combinations of attributes included in the attribute list.

[0039] For example, users enter their opinions into LLM in a free-text format. Figure 3 shows an example of a user's opinion. For example, opinion 2a entered by user 1a is, "Even if a woman is single and unemployed, she should be eligible for a loan if she has substantial savings." The words “single,” “unemployed,” “female,” and “savings” in opinion 2a are attributes included in attribute list 10 and can be expressed using existing attributes.

[0040] User 1b's opinion 2b states, "Even if the employment status is non-regular, if there is a stable income, the application should be approved." The term "non-regular" in opinion 2b is an attribute included in attribute list 10 and can be expressed using existing attributes. On the other hand, the term "stable income" in opinion 2b is not included in attribute list 10 and cannot be expressed using existing attributes.

[0041] User 1c's input, Opinion 2c, is "I would like you to consider not only the applicant's age but also their living situation." The "age" in Opinion 2c is an attribute included in Attribute List 10 and can be expressed using existing attributes. On the other hand, the "living situation" in Opinion 2c is not included in Attribute List 10 and cannot be expressed using existing attributes.

[0042] User 1d's opinion 2d is "Even if someone has defaulted on loan payments in the past, improvement should be taken into account in the evaluation." The terms "default" and "improvement" included in opinion 2d are not included in attribute list 10 and cannot be expressed using existing attributes.

[0043] Of the opinions 2a to 2d shown in Figure 3, opinion 2a is the least ambiguous, followed by opinion 2b, opinion 2c, and opinion 2d in increasing order of ambiguity. In this embodiment, the processing of the information processing device 100 will be explained using opinion 2c. Opinions 2a to 2d correspond to "documents concerning the conditions for decision-making regarding specific user attributes."

[0044] For example, the information processing device 100 performs mapping processing, ambiguity score calculation processing, additional question processing, and label identification processing for opinion 2c. Each process will be explained in order below.

[0045] First, let's explain the mapping process. Figure 4 is a flowchart showing the processing steps of the mapping process. As shown in Figure 4, the information processing device 100 performs tokenization (step S10). The information processing device 100 generates a prompt indicating the relationship between tokens and attributes (step S11). The information processing device 100 uses LLM to calculate the probability that a token corresponds to an attribute (step S12).

[0046] The information processing device 100 selects one unselected token (step S13). The information processing device 100 obtains the attribute with the highest probability among the probabilities of attributes corresponding to the selected token (step S14). If the highest probability is not equal to or greater than a threshold (step S15, No), the information processing device 100 proceeds to step S17.

[0047] On the other hand, if the maximum probability is greater than or equal to a threshold (step S15, Yes), the information processing device 100 adds the selected attribute to "the attribute whose corresponding probability is greater than or equal to a threshold" (step S16).

[0048] If the information processing device 100 has not selected all tokens (step S17, No), it proceeds to step S13. On the other hand, if the information processing device 100 has not selected all tokens (step S17, Yes), it terminates the mapping process.

[0049] Next, I will provide a supplementary explanation of the mapping process shown in Figure 4.

[0050] Step S10 is explained in more detail. Figure 5 is a diagram illustrating step S10. The information processing device 100 performs morphological analysis on the text "I would like you to consider not only the applicant's age but also their living situation." included in opinion 2c. The information processing device 100 then divides the text into units (tokens) such as words and phrases, and removes punctuation marks and particles to generate token information 3.

[0051] For example, token information 3 includes multiple tokens such as “applicant,” “age,” “living situation,” and “considerations.”

[0052] Next, step S11 will be explained in more detail. For each token contained in the token information 3, the information processing device 100 generates a prompt that combines all the attributes of the attribute list 10. As a result, the information processing device 100 generates a prompt equal to "number of tokens" × "number of attributes".

[0053] The information processing device 100 generates a prompt using "prompt=f"{tokens} means {attributes}". For example, a prompt based on the token "Applicant" and the attribute "Housing Type" would be "prompt=f"{Applicant} means {Housing Type}".

[0054] Next, step S12 will be explained in more detail. For example, the information processing device 100 inputs "prompt=f"{Applicant} means {Type of Residence}" into the LLM, and obtains from the LLM the probability that the token "Applicant" corresponds to the attribute "Type of Residence". The information processing device 100 also inputs prompts corresponding to the relationship between other tokens and attributes into the LLM, and obtains the probability that a token corresponds to an attribute.

[0055] Here, the information processing device 100 utilizes the characteristic that the LLM can output the probability of token occurrence (log probability). The information processing device 100 inputs a prompt to the LLM and obtains the occurrence probability output from the LLM as the probability of the attribute for the token.

[0056] The information processing device 100 generates a probability table by performing the above processing. Figure 6 shows an example of the data structure of the probability table. As shown in Figure 6, the probability table 15 holds the probability that a token corresponds to an attribute. For example, it indicates that the probability that the token "Applicant" corresponds to the attribute "Housing Type" is "0.10".

[0057] Next, we will provide a supplementary explanation of steps S13 to S15. Figure 7 is a diagram that provides a supplementary explanation of steps S13 to S15. For example, if the information processing device 100 selects the token "Applicant," the attribute with the highest probability among all the attributes will be "Income," which has a probability of "0.20." The information processing device 100 associates the token "Applicant," the attribute "Income," and the probability "0.20" and registers them in Table 15-1.

[0058] If the information processing device 100 selects the token "age," the attribute with the highest probability among all attributes will be "age," which has a probability of "0.98." The information processing device 100 associates the token "age," the attribute "age," and the probability "0.98" and registers them in table 15-1.

[0059] When the information processing device 100 selects the token "Living Situation," the attribute with the highest probability among all attributes is "Housing Type," which has a probability of "0.82." The information processing device 100 associates the token "Living Situation," the attribute "Housing Type," and the probability "0.82" and registers them in Table 15-1.

[0060] If the information processing device 100 selects the token "Consider," the attribute with the highest probability among all attributes will be "Employment Type," which has a probability of "0.01." The information processing device 100 associates the token "Consider," the attribute "Employment Type," and the probability "0.01" and registers them in Table 15-1.

[0061] The information processing device 100 selects tokens from table 15-1 whose probability is greater than or equal to a threshold, and from probability table 15, selects all attributes corresponding to the selected tokens that are greater than or equal to the threshold, and registers them in table 15-2. For example, let's set the threshold to 0.5.

[0062] For example, the information processing device 100 selects tokens "age" and "living situation" from the tokens registered in table 15-1 whose probability is equal to or greater than a threshold (0.5).

[0063] The information processing device 100 selects the attribute "age" from the attributes corresponding to the token "age" in probability table 15 that is equal to or greater than the threshold (0.5). The information processing device 100 associates the token "age", the attribute "age", and the probability "0.98" and registers them in table 15-2.

[0064] The information processing device 100 selects the following attributes from the probability table 15 that correspond to the token "living situation" and are equal to or greater than the threshold (0.5): "housing type", "family structure", "employment type", "income", "age", "expenses one month ago", "expenses two months ago", "invoice one month ago", and "invoice two months ago". As shown in Figure 7, the information processing device 100 associates the token "living situation", each selected attribute, and the probability of each attribute and registers them in table 15-2. For example, the probability of the attribute "housing type" is "0.82". The probability of the attribute "family structure" is "0.79". The probability of the attribute "employment type" is "0.63". The probability of the attribute "income" is "0.62". The probability of the attribute "age" is "0.58". The probability of the attribute "expenses one month ago" is "0.58". The probability of the attribute "Amount spent 2 months ago" is 0.57. The probability of the attribute "Amount billed 1 month ago" is 0.53. The probability of the attribute "Amount billed 2 months ago" is 0.52.

[0065] The above explains the mapping process.

[0066] Next, the ambiguity score calculation process will be explained. Figure 8 is a flowchart showing the processing steps for the ambiguity score calculation process. As shown in Figure 8, the information processing device 100 counts the number of attributes associated with each token based on Table 15-2 (step S20). The information processing device 100 identifies the token with the largest number of associated attributes (step S21).

[0067] The information processing device 100 selects one unselected token from the tokens in table 15-2 (step S22). The information processing device 100 calculates the ambiguity weight of the token (step S23). If the information processing device 100 has not selected all tokens in table 15-2 (step S24, No), it proceeds to step S22.

[0068] On the other hand, if the information processing device 100 selects all tokens in table 15-2 (step S24, Yes), it calculates the opinion ambiguity score (step S25).

[0069] Next, we will provide a supplementary explanation of the ambiguity score calculation process shown in Figure 8.

[0070] Let me provide some additional information about step S20. For example, the information processing device 100 identifies the number of attributes associated with the token "age" as "1" based on the table 15-2 shown in Figure 7. The information processing device 100 identifies the number of attributes associated with the token "living situation" as "9".

[0071] Let me provide some additional information about step S21. For example, since the number of associated attributes for the token "age" is "1" and the number of associated attributes for the token "living situation" is "9", the information processing device 100 identifies the token "living situation" as the token with the largest number of associated attributes.

[0072] Steps S22 to S24 are explained below. The information processing device 100 calculates the ambiguity weights of the tokens "age" and "living situation" registered in table 15-2 based on equation (3). The maximum number of attributes is the number of attributes of the token identified in step S21.

[0073] Token ambiguity weight = Number of attributes the token is associated with / Maximum number of attributes ... (3)

[0074] For example, the information processing device 100 calculates the ambiguity weight of the token "age" as "1 / 9 = 0.11". The information processing device 100 also calculates the ambiguity weight of the token "living situation" as "9 / 9 = 1.00".

[0075] Let me provide some additional explanation regarding step S25. The information processing device 100 calculates the opinion ambiguity score based on equation (4). The total number of opinion tokens in equation (4) is the number of tokens set in table 15-2, and in the example shown in Figure 7, the total number of tokens is "2".

[0076] Opinion ambiguity score = Σ(ambiguity weight of each token) / total number of opinion tokens ... (4)

[0077] Based on equation (4), the information processing device 100 calculates the ambiguity score of opinion 2c, and the ambiguity score is "(0.11 + 1.00) / 2 = 0.55".

[0078] The above explains the process for calculating the ambiguity score.

[0079] Next, the additional question processing will be explained. Figure 9 is a flowchart of the processing procedure for additional questions. As shown in Figure 9, the information processing device 100 terminates the additional question processing if the opinion ambiguity score is not equal to or greater than the threshold (0.5) (step S30, No). On the other hand, if the opinion ambiguity score is equal to or greater than the threshold (0.5) (step S30, Yes), the information processing device 100 proceeds to step S31.

[0080] The information processing device 100 asks additional questions to identify attributes related to the opinion and attributes unrelated to the opinion (step S31). The information processing device 100 assigns the value "0.00" to the "attributes related to the opinion" (step S32). The information processing device 100 excludes the "attributes unrelated to the opinion" (step S33).

[0081] The information processing device 100 stores the attributes related to the opinion (step S34). The information processing device 100 updates the ambiguity weight using the attributes related to the opinion (step S35). The information processing device 100 calculates the ambiguity score of the opinion (step S36) and proceeds to step S30.

[0082] Next, we will provide further explanation regarding the processing of additional questions shown in Figure 9.

[0083] Let me provide some additional information about step S31. The information processing device 100 instructs the LLM to confirm with the user which attributes are related and which are not, using the attributes set in table 15-2 in Figure 7: “Housing type”, “Family structure”, “Employment type”, “Income”, “Age”, “Expenses from one month ago”, “Expenses from two months ago”, “Invoice amount from one month ago”, and “Invoice amount from two months ago”. Then the LLM generates the following dialogue 2-1.

[0084] Dialogue 2-1 (LLM): Please clarify the attributes related to your opinion. From the list below, please identify the attributes that are related to your opinion and those that are not. List: “Housing type”, “Family composition”, “Employment type”, “Income”, “Age”, “Expenses one month ago”, “Expenses two months ago”, “Invoice one month ago”, “Invoice two months ago”

[0085] Here, a user who referred to dialogue 2-1 is assumed to have entered the following dialogue 2-2.

[0086] Dialogue 2-2 (User): The relevant attributes are "Housing type", "Family structure", "Employment type", "Income", "Age", "Expenses one month ago", and "Expenses two months ago". The irrelevant attributes are "Bill amount one month ago" and "Bill amount two months ago".

[0087] Based on the content of dialogue 2-2, the information processing device 100 identifies that the relevant attributes are "type of residence," "family structure," "employment status," "income," "age," "expenses one month ago," and "expenses two months ago," while the irrelevant attributes are "invoice amount one month ago" and "invoice amount two months ago."

[0088] Steps S32 to S34 are explained in detail. Figure 10 is a diagram illustrating the steps S32 to S34. The information processing device 100 sets the probability of related attributes in table 15-2 to "0.00". The information processing device 100 also deletes unrelated attributes in table 15-2. As a result, the information processing device 100 generates and saves table 15-3.

[0089] Let me provide some additional information about step S35. Based on table 15-3, the information processing device 100 updates the ambiguity weight of each token by the average of the probabilities of the associated attributes. For example, the ambiguity weight of the token "age" is "0.00 / 1=0". The ambiguity weight of the token "living situation" is "0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+ / =0".

[0090] Let me provide some additional explanation regarding step S36. The information processing device 100 calculates the ambiguity score of the opinion using the ambiguity weights of each token obtained in step S35. Based on equation (4), the information processing device 100 calculates the ambiguity score of opinion 2c, which is "(0+0) / 2=0".

[0091] This concludes the explanation regarding the handling of additional questions.

[0092] Next, the label identification process will be explained. Figure 11 is a flowchart showing the processing steps for the label identification process. As shown in Figure 11, the information processing device 100 asks for the value of the attribute that matches the opinion (step S40).

[0093] The information processing device 100 obtains the answers to the questions (step S41). Based on the answers to the questions, the information processing device 100 sets labels for the training dataset 141 (step S42).

[0094] Next, we will provide a supplementary explanation regarding the label identification process shown in Figure 11.

[0095] Let me provide some additional information about step 40. The information processing device 100 instructs the LLM to obtain specific values ​​for the attributes set in table 15-3 in Figure 10 that are related to the opinion. For example, the attributes related to the opinion are "type of housing", "family structure", "employment type", "income", "age", "expenditure one month ago", and "expenditure two months ago". Then the LLM generates dialogue 3-1 as shown below.

[0096] Dialogue 3-1 (LLM): Select the values ​​for each attribute if you determine that the loan is "approved" under the following conditions. "Family structure": Single, Married with no children, Married with children "Housing type": Owned home, rented, living with parents "Income": High, Medium, Low "Employment status": Full-time employee, contract employee, part-time employee, self-employed, unemployed "Age": 18-25, 26-35, 36-45, 46-55, 56-66, 67 and over “Expenses from one month ago”, “Expenses from two months ago”: 10,000-100,000 100,001-200,000 200,001-300,000 300,001-500,000 500,001-100,000 100,001 or more

[0097] Here, a user who has referred to dialogue 3-1 is assumed to have entered the following dialogue 3-1.

[0098] Dialogue 3-2 (User): If your “expenditure one month ago” and “expenditure two months ago” are between 200,001 and 300,000 yen or less, your “family structure” is either married with no children or married with children, your “housing type” is homeowner or renter, your “income” is high, your “employment type” is full-time employee or self-employed, and your “age” is between 36-45, 46-55, or 56-65, then you are “able to repay a loan.”

[0099] Steps S41 and S42 will be explained in more detail. Figure 12 is a diagram that provides supplementary explanations for steps S41 and S42. The information processing device 100 sets labels on the training dataset 141 based on the attribute and its value corresponding to "Loan repayment possible" shown in dialogue 3-2.

[0100] The information processing device 100 sets the label of training data that satisfies the attribute and value corresponding to "loan repayment possible" to "1". A label "1" indicates that the loan is approved. On the other hand, the information processing device 100 sets the label of training data that does not satisfy the attribute and value corresponding to "loan repayment possible" to "0". A label "0" indicates that the loan is not approved.

[0101] For example, among the training data included in the training dataset 141, the training data with IDs "1" and "4" satisfy the attribute and value corresponding to "loan repayment possible". Therefore, the information processing device 100 sets the label of the training data with IDs "1" and "4" to "1".

[0102] Among the training data included in the training dataset 141, the training data with IDs "2", "3", and "5" do not satisfy the attribute and value corresponding to "loan repayment possible". Therefore, the information processing device 100 sets the label of the training data with IDs "2", "3", and "5" to "0".

[0103] The label identification process has now been explained.

[0104] Next, an example configuration of the information processing device 100 that performs the mapping process, ambiguity score calculation process, additional question processing, and label identification process described above will be explained. Figure 13 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Figure 13, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0105] The communication unit 110 performs data communication with the user terminal used by the user via the network. The communication unit 110 may also connect to an external device and receive training datasets 141, etc., from the external device.

[0106] The input unit 120 inputs various information to the control unit 150. The user may interact with the LLM151 via the network, or by operating the input unit 120.

[0107] The display unit 130 displays the information output from the control unit 150.

[0108] The memory unit 140 contains a training dataset 141 and a machine learning model 142. The memory unit 140 is a memory, etc.

[0109] Training dataset 141 contains multiple training data points. Each training data point has multiple attributes and corresponding values. The initial label values ​​for each training data point are not set. The description of training dataset 141 is the same as that described in Figures 2 and 12.

[0110] Machine learning models 142 include Neural Networks (NNs), etc.

[0111] The control unit 150 includes an LLM 151, a specific unit 152, and a training unit 153. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0112] LLM151 has the functionality of a large-scale language model and interacts with the user. When LLM151 receives instructions from the specific unit 152, it interacts with the user in accordance with those instructions. LLM151 may implement the functionality of a large-scale language model using an LLM server connected via a network.

[0113] An LLM can output the probability of token occurrence because it learns the statistical patterns of language through pre-training. For example, a large amount of text data is prepared as training data, and each piece of text data is divided into tokens. As an objective function, a goal is set to predict the probability of the next token based on a given context (the preceding and succeeding tokens). For example, an architecture called a Transformer is used to train the LLM based on backpropagation so that the LLM's prediction results are as accurate as possible to the goal. Once such training is complete, the LLM can calculate the probability of occurrence of each token and construct sentences.

[0114] The identification unit 152 monitors the interaction between the user and the LLM 151, identifies the labels of the training dataset 141 based on the user's feedback, and updates the training dataset 141.

[0115] For example, when the identification unit 152 obtains a user's opinion from a user terminal or input unit 120, it performs mapping processing, ambiguity score calculation processing, additional question processing, and label identification processing on the opinion.

[0116] The mapping process performed by the identification unit 152 corresponds to the process described in Figure 4, etc. The ambiguity score calculation process performed by the identification unit 152 corresponds to the process described in Figure 8, etc. The additional questioning process performed by the identification unit 152 corresponds to the process described in Figure 9, etc. The label identification process performed by the identification unit 152 corresponds to the process described in Figure 11.

[0117] The training unit 153 trains the machine learning model 142 using the training dataset 141. For example, the training unit 153 trains the machine learning model 142 based on backpropagation, using the attribute values ​​of the training data as explanatory variables and the labels as the target variable.

[0118] Next, an example of the processing procedure of the information processing device 100 according to this embodiment will be described. Figure 14 is a flowchart of the processing procedure of the information processing device according to this embodiment. As shown in Figure 14, interaction is initiated between the user and the LLM151 of the information processing device 100 (step S101).

[0119] The identification unit 152 of the information processing device 100 acquires the user's opinion (step S102). The identification unit 152 performs mapping processing (step S103). The identification unit 152 performs ambiguity score calculation processing (step S104).

[0120] The identification unit 152 performs additional question processing (step S105). The identification unit 152 performs label identification processing (step S106). The training unit 153 of the information processing device 100 trains the machine learning model 142 based on the training dataset 141 (step S107).

[0121] Here, the processing steps for the mapping process shown in step S103 of Figure 14 correspond to the processing steps described in Figure 4. The processing steps for the ambiguity score calculation process shown in step S104 correspond to the processing steps described in Figure 8. The processing steps for the additional questioning process shown in step S105 correspond to the processing steps described in Figure 9. The processing steps for the label identification process shown in step S106 correspond to the processing steps described in Figure 11.

[0122] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 obtains attributes from training data, obtains opinions from users, and inputs prompts for tokens and attributes included in the opinions into the LLM 151 to obtain the probability of occurrence of tokens related to attributes. Based on the probability of occurrence of tokens related to attributes, the information processing device 100 identifies labels corresponding to the training data.

[0123] By using training data with identified labels, it is possible to incorporate the domain knowledge of stakeholders and construct a machine learning model 142 capable of making decisions similar to those of humans. Furthermore, in areas such as loan application review, the opinions of non-experts in customer and audit organizations can be treated the same as those of domain experts, allowing the opinions of diverse stakeholders to be reflected in the machine learning model 142.

[0124] The information processing device 100 calculates a score regarding the ambiguity of an opinion based on the probability of occurrence of tokens related to attributes, and accepts the selection of attributes related to the opinion and attributes not related to the opinion from among multiple attributes related to tokens in opinions whose score is above a threshold. The information processing device 100 also sets the probability of occurrence of attributes related to the opinion to zero from among multiple attributes related to the token, and calculates a score regarding the ambiguity of an opinion based on the probability of occurrence of each attribute of the token obtained by excluding attributes not related to the opinion from the multiple attributes related to the token. The information processing device 100 identifies a label based on the attributes from among multiple attributes related to tokens in opinions whose score is below a threshold, excluding attributes not related to the opinion. In this way, by converting ambiguous opinions into combinations of existing attributes, implicit information and nuances contained in opinions can be transformed into a form that is easy for the model to learn. In addition, the basis for the attribute combinations presented by LLM151 can be made clear.

[0125] Next, an example of a computer hardware configuration that achieves the same functions as the information processing device 100 described above will be explained. Figure 15 is a diagram showing an example of a computer hardware configuration that achieves the same functions as the information processing device according to the embodiment.

[0126] As shown in the figure, the computer 200 includes a CPU 201 that performs various calculations, an input device 202 that receives data input from the user, and a display 203. The computer 200 also includes a communication device 204 and an interface device 205 that exchange data with user terminals, external devices, etc., via a wired or wireless network. The computer 200 also includes a RAM 206 for temporarily storing various information and a hard disk drive 207. Each of the devices 201 to 207 is connected to a bus 208.

[0127] The hard disk drive 207 contains an LLM program 207a, a specific program 207b, and a training program 207c. The CPU 201 reads each of the programs 207a to 207c and loads them into the RAM 206.

[0128] The LLM program 207a functions as the LLM process 206a. The specific program 207b functions as the specific process 206b. The training program 207c functions as the training process 206c.

[0129] The processing of LLM process 206a corresponds to the processing of LLM 151. The processing of specific process 206b corresponds to the processing of specific unit 152. The processing of training process 206c corresponds to the processing of training unit 153.

[0130] Furthermore, programs 207a to 207c do not necessarily have to be stored on the hard disk drive 207 from the beginning. For example, each program could be stored on a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 200. Then, the computer 200 could read and execute each program 207a to 207c.

[0131] With regard to embodiments including each of the above examples, the following additional information is disclosed.

[0132] (Note 1) Obtain the attributes of the training data, Obtain documents regarding the criteria for decision-making based on specific user attributes. By inputting a prompt into a large-scale language model that includes the attributes of the training data and tokens from a document relating to the decision-making conditions, the probability of occurrence of the tokens relating to the attributes is output. Based on the probability of the outputted tokens appearing, the labels to be learned by the machine learning model are identified from the training data. A specific program characterized by causing a computer to perform a process.

[0133] (Note 2) The identification program according to Note 1, characterized in that it outputs the probability of occurrence of the token and the attribute for each attribute, identifies the token for which any of the probability of occurrence is above a threshold, and causes the computer to further perform a process of calculating a score regarding the ambiguity of the document based on the application probability for each attribute of the identified token.

[0134] (Note 3) The specific program described in Note 2, characterized in that it causes a computer to further perform a process of accepting the selection of attributes related to the document and attributes not related to the document from among a plurality of attributes relating to the token contained in the document whose score is equal to or greater than a threshold.

[0135] (Note 4) The specific program described in Note 3, characterized in that the computer further performs a process of calculating a score regarding the ambiguity of the document based on the probability of occurrence of each attribute of the token, which is obtained by setting the probability of occurrence of attributes related to the document among the multiple attributes of the token to zero, and excluding attributes unrelated to the document from the multiple attributes of the token.

[0136] (Note 5) The identification program described in Note 4, characterized in that the process of identifying the label is to identify the label based on the attributes of a plurality of attributes relating to tokens contained in a document whose score is less than a threshold, excluding the attributes that are not related to the document.

[0137] (Note 6) The specific program described in Note 1, characterized in that it causes a computer to further perform a process of training the machine learning model based on the training data.

[0138] (Note 7) Obtain the attributes of the training data, Obtain documents regarding the criteria for decision-making based on specific user attributes. By inputting a prompt into a large-scale language model that includes the attributes of the training data and tokens from a document relating to the decision-making conditions, the probability of occurrence of the tokens relating to the attributes is output. Based on the probability of the outputted tokens appearing, the labels to be learned by the machine learning model are identified from the training data. A method for identifying a process, characterized in that the process is performed by a computer.

[0139] (Appendix 8) The identification method according to Appendix 7, characterized in that the computer further performs a process of outputting the probability of occurrence of the token and the attribute for each attribute, identifying the token for which any of the probability of occurrence is above a threshold, and calculating a score regarding the ambiguity of the document based on the application probability for each attribute of the identified token.

[0140] (Note 9) The identification method according to Note 8, characterized in that the computer further performs a process of accepting the selection of attributes related to the document and attributes not related to the document from among a plurality of attributes relating to the token contained in the document whose score is equal to or greater than a threshold.

[0141] (Note 10) The identification method according to Note 9, characterized in that the computer further performs a process of calculating a score regarding the ambiguity of the document based on the probability of occurrence of each attribute of the token, which is obtained by setting the probability of occurrence of attributes related to the document to zero among the multiple attributes of the token, and excluding attributes that are not related to the document from the multiple attributes of the token.

[0142] (Note 11) The method for identifying the label according to Note 10, characterized in that the process for identifying the label is to identify the label based on the attributes obtained by excluding the attributes that are not related to the document from among a plurality of attributes relating to the token contained in the document whose score is less than the threshold.

[0143] (Note 12) The identification method according to Note 7, characterized in that a computer further performs a process of training the machine learning model based on the training data.

[0144] (Note 13) Obtain the attributes of the training data, Obtain documents regarding the criteria for decision-making based on specific user attributes. By inputting a prompt into a large-scale language model that includes the attributes of the training data and tokens of a document relating to the decision-making conditions, the probability of occurrence of the tokens relating to the attributes is output. Based on the probability of the outputted tokens appearing, the labels to be learned by the machine learning model are identified from the training data. An information processing device having a control unit that performs processing.

[0145] (Note 14) The information processing apparatus according to Note 13, characterized in that the control unit outputs the probability of occurrence of the token and the attribute for each attribute, identifies the token for which any of the probability of occurrence is greater than or equal to a threshold, and further performs a process to calculate a score regarding the ambiguity of the document based on the application probability for each attribute of the identified token.

[0146] (Note 15) The information processing apparatus according to Note 14, characterized in that the control unit further performs a process of accepting the selection of attributes related to the document and attributes not related to the document from among a plurality of attributes relating to the token contained in the document whose score is equal to or greater than a threshold.

[0147] (Note 16) The information processing apparatus according to Note 15, wherein the control unit further performs a process of calculating a score regarding the ambiguity of the document based on the probability of occurrence of each attribute of the token obtained as a result of excluding attributes unrelated to the document from the multiple attributes of the token, setting the probability of occurrence of attributes related to the document to zero, and excluding attributes unrelated to the document from the multiple attributes of the token.

[0148] (Note 17) The information processing apparatus according to Note 16, wherein the process for identifying the label is to identify the label based on the attributes of a plurality of attributes relating to tokens contained in a document whose score is less than a threshold, excluding the attributes that are not related to the document.

[0149] (Note 18) The information processing apparatus according to Note 13, characterized in that the control unit causes the computer to further execute a process to train the machine learning model based on the training data. [Explanation of Symbols]

[0150] 100 Information Processing Devices 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 training datasets 142 Machine Learning Models 150 Control Unit 151 LLM 152 Specific part 153 Training Department

Claims

1. Obtain the attributes of the training data, Obtain documents regarding the criteria for decision-making based on specific user attributes. By inputting prompts containing the attributes of the training data and tokens from a document relating to the decision-making conditions into a large-scale language model, the probability of occurrence of the tokens relating to the attributes is output. Based on the probability of the outputted tokens appearing, the labels to be learned by the machine learning model are identified from the training data. A specific program characterized by causing a computer to perform a process.

2. The identification program according to claim 1, characterized in that it outputs the probability of occurrence of the token and the attribute for each attribute, identifies the token whose probability of occurrence is above a threshold, and causes the computer to further perform a process of calculating a score regarding the ambiguity of the document based on the application probability for each attribute of the identified token.

3. The specific program according to claim 2, further characterized in that it causes a computer to perform a process of accepting the selection of attributes related to the document and attributes not related to the document from among a plurality of attributes relating to the token contained in the document whose score is equal to or greater than a threshold.

4. The specific program according to claim 3, further characterized in that the computer is instructed to perform a process of calculating a score regarding the ambiguity of the document based on the probability of occurrence of each attribute of the token, obtained by setting the probability of occurrence of attributes related to the document to zero among a plurality of attributes relating to the token, and excluding attributes unrelated to the document from the plurality of attributes relating to the token.

5. The identification program according to claim 4, wherein the process of identifying the label is to identify the label based on the attributes of a plurality of attributes relating to tokens contained in a document whose score is less than a threshold, excluding the attributes that are not related to the document.

6. The specific program according to claim 1, characterized in that it causes a computer to further perform a process of training the machine learning model based on the aforementioned training data.

7. Obtain the attributes of the training data, Obtain documents regarding the criteria for decision-making based on specific user attributes. By inputting a prompt into a large-scale language model that includes the attributes of the training data and tokens from a document relating to the decision-making conditions, the probability of occurrence of the tokens relating to the attributes is output. Based on the probability of the outputted tokens appearing, the labels to be learned by the machine learning model are identified from the training data. A method for identifying a process, characterized in that the process is performed by a computer.

8. Obtain the attributes of the training data, Obtain documents regarding the criteria for decision-making based on specific user attributes. By inputting a prompt into a large-scale language model that includes the attributes of the training data and tokens from a document relating to the decision-making conditions, the probability of occurrence of the tokens relating to the attributes is output. Based on the probability of the outputted tokens appearing, the labels to be learned by the machine learning model are identified from the training data. An information processing device having a control unit that performs processing.

Images