Training and use of bayesian networks and language models for agribusiness delinquency risk assessment

Abstract

In some embodiments, a computer-implemented method includes reading a multiple dependencies. Each dependency can be between a pair of features of multiple features, the multiple features including a target feature. The method can comprise reading training data correlating each pair. The training data can comprise loan application data for multiple loans. The method can include determining, from the training data, a conditional probability for each pair of features. Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature. The method can include constructing a directed graph comprising multiple nodes and multiple edges. Each node can correspond to one of the features. Nodes corresponding to a dependency can be connected by an edge. The method can further include assigning the conditional probability to each edge corresponding to that pair.

Description

TRAINING AND USE OF BAYESIAN NETWORKS AND LANGUAGE MODELS FORAGRIBUSINESS DELINQUENCY RISK ASSESSMENTCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application. No. 63 / 529,201 filed July 27, 2023, and U.S. Provisional Application. No. 63 / 529,205 filed on July 27, 2023, the teachings of which are hereby incorporated by reference in their entirety.BACKGROUND

[0002] Agriculture is critical to food security, and thus, global stability and propriety. Seventy percent of the world's food production is financed by private sources, such as commercial banks, development banks, investment funds, and industry suppliers. The consequences of an incorrect investments are not only problematic for the lender, but also the global food supply, and especially vulnerable populations, which are more sensitive to fluctuations in commodities prices.BRIEF SUMMARY

[0003] In some embodiments, a computer-implemented method includes reading a plurality of dependencies. Each dependency can be between a pair of features of a plurality of features, the plurality of features including a target feature. The method can comprise reading training data correlating each pair of features. The training data can comprise loan application data for a plurality of loans. The method can include determining, from the training data, a conditional probability for each pair of features. Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature. The method can include constructing a directed graph.The directed graph can comprise a plurality of nodes and a plurality of edges. Each of the plurality of nodes can correspond to one of the plurality of features. Nodes corresponding to a dependency of a pair of features can be connected by one of the plurality of edges. The method can further include assigning, for each pair of features, the conditional probability to the edge corresponding to that pair.

[0004] In some embodiments, the loan application data comprises one or more of the plurality of features and the target feature, the target feature being an outcome of each loan. In some embodiments, the loan application data comprises agricultural loan application data.

[0005] In some embodiments, the directed graph is a Bayesian network and the Bayesian network is configured to output a probability of the target variable. The probability can represent a credit risk assessment based on financial data of an entity.

[0006] In some embodiments, the plurality of dependencies includes (a) the target variable depending on a first variable of the plurality of variables, and (b) the first variable depending on a second variable of the plurality of variables.

[0007] In some embodiments, the plurality of variables comprises score features, the score features being derived from another computer-implemented model.

[0008] In some embodiments, the score features include one or more of an agronomic score, a market score, a financial score, and a behavior score.

[0009] In some embodiments, the plurality of variables comprise data based features, the data based features loaded from a database.

[0010] In some embodiments, the data based features include a ratio of short-term debt to total planted area (RSTDTPA), a ratio of long-term debt to the farm area (RLTDFA), a land lease cost, a credit history, and a main crop.

[0011] In some embodiments, the method further comprises converting the training data into discrete values.

[0012] In some embodiments, the method further comprises calculating a measure of concept drift for one or more of the plurality of features. The method can further comprise normalizing the features based on the measure of concept drift.

[0013] In some embodiments, the directed graph is a Bayesian network.

[0014] In some embodiments, a computer-implemented method includes loading a directed graph. The directed graph can comprise a plurality of nodes and a plurality of edges. Each of the plurality of nodes can correspond to one of a plurality of features. The nodes can correspond to a dependency of a pair of features are connected by one of the plurality of edges. Each edge can represent a conditional probability for each pair of features. Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature. The method can further comprise reading a plurality of financial features of an entity. The method can further comprise applying the plurality of financial features to corresponding nodes of the directed graph, the directed graph thereby generating a target feature. The method can further output, based on the target feature, a recommendation to provide or deny lending to the entity.

[0015] In some embodiments, the method can further comprise reading a loan application and extracting the plurality of financial features from the loan application.

[0016] In some embodiments, the loan application can comprise an agricultural loan application.

[0017] In some embodiments, the generated target feature provides a credit risk assessment based on the financial features of the entity.

[0018] In some embodiments, plurality of dependencies includes (a) the target feature depending on a first feature of the plurality of features, and (b) the first feature depending on a second feature of the plurality of features.

[0019] In some embodiments, the plurality of features comprises score features, the score features being derived from another computer-implemented model.

[0020] In some embodiments, the score features include one or more of an agronomic score, a market score, a financial score, and a behavior score.

[0021] In some embodiments, the plurality of features comprises data-based features, the data- based features loaded from a database.

[0022] In some embodiments, the data based features include a ratio of short-term debt to total planted area (RSTDTPA), a ratio of long-term debt to the farm area (RLTDFA), a land lease cost, a credit history, and a main crop.

[0023] In some embodiments, the method further comprises calculating a measure of concept drift for one or more of the plurality of financial features and normalizing the financial features based on the measure of concept drift.

[0024] In some embodiments, the directed graph is a Bayesian network.

[0025] In some embodiments, a computer-implemented method can comprise reading, by a language model, a text representation of a directed graph. The text representation can comprise a representation of plurality of dependencies. Each dependency can be between a pair of features of a plurality of features and a representation of a conditional probability for each pair of features. The plurality of features can include a target feature. Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature. The method cancomprise reading, at the language model, loan application data of an entity. The method can comprise applying, by the language model, the loan application data to the directed graph. The method can comprise generating, by the language model, a risk score on the loan application data of the entity.

[0026] In some embodiments, loan application data can include one or more of the entity’s financial status and previous credit history.

[0027] In some embodiments, the method can include generating, by the language model, a description of a set of features that influenced generation of the risk score.

[0028] In some embodiments, the method can include reading, at the language model, a prompt to provide a description of the set of features that influenced generation of the risk score.

[0029] In some embodiments, the directed graph is a Bayesian network.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0030] Fig. 1 is a network diagram illustrating a Bayesian network for domain specific loan credit risk assessment according to embodiments of the present disclosure.

[0031] Fig. 2 is a block diagram illustrating a model factory generating a directed graph according to embodiments of the present disclosure.

[0032] Fig. 3 is a diagram illustrating the use of a directed graph according to embodiments of the present disclosure.

[0033] Fig. 4 is a diagram illustrating an example embodiment of providing an engineered prompt to a language model according to embodiments of the present disclosure.

[0034] Fig. 5 is a diagram illustrating a prompt generation sever according to embodiments of the present disclosure.

[0035] Fig. 6 is a flowchart illustrating a method of training a model according to embodiments of the present disclosure.

[0036] Fig. 7 is a flowchart illustrating a method of using a model according to embodiments of the present disclosure.

[0037] Fig. 8 is a flowchart illustrating an example embodiment of a method of using a language model to generate a risk score according to embodiments of the present disclosure.

[0038] Fig. 9 is a diagram illustrating a computing node according to embodiments of the present disclosure.DETAILED DESCRIPTIONBayesian Networks Improve Out-of-Distribution Calibration for Agribusiness Delinquency Risk Assessment Tasks

[0039] Given the importance of agriculture, private sources that finance agricultural activity need to assess the risk of their investments in that field. Automated credit risk assessment can have an important role in financial institutions providing lending to agricultural businesses. However, credit risk assessment in the agricultural domain has unique challenges because of the impacts of weather, pest outbreaks, commodities market dynamics, and other volatile forces that drive risk. Training a model to account for these factors can require large data sets, or data that is challenging to obtain. In some previous implementations, credit risk assessment models are trained using data from single-institutions that often focus on dedicated geographical regions, or singular commodities. Therefore, agricultural credit risk models can exhibit poor out-of-domain performance.

[0040] In some embodiments of the present disclosure, a Bayesian network model for loan delinquency classification is employed. The Bayesian network can be trained on historical loandata from lenders (e.g., a dataset describing approximately 100,000 historical loans, sourced from nine large agricultural lenders). In some embodiments, this Bayesian network model exhibited improved calibration (e.g., a relative improvement in Brier score) in out-of-domain performance tests when compared to three state-of-the-art credit risk scoring approaches, including logistic regression (72 ± 23% improvement), XGBoost (72 ± 23% improvement), and an Artificial Neural Networks (6 ± 14% improvement). In some embodiments, the Bayesian network provides better modeling of agricultural credit risk by combining data assets with expert domain knowledge. In addition, the approach used herein can generalize to credit risk assessment tasks, even outside of the agriculture lending, where small sample sizes is of concern.

[0041] In alternative methods, risk is assessed with human analysts that leverage a combination of human experience in the field and financial data (e.g., from credit bureaus) to assess the credit worthiness of farmers and agricultural businesses seeking loans. However, despite the expertise and experience of human credit analysts, this alternative methods are prone to errors for two reasons. First, there are no global standards to perform agricultural credit risk assessment. Second, existing analytic tools to support the analysts (e.g., consumer credit risk scores) are not appropriate because they were not purpose-built for a particular domain, such as agriculture. A consumer credit risk score does not necessarily translate to a comprehensive assessment of risk in a specific domain (e.g., agriculture) because many of the factors that impact risk for that domain are completely beyond the control of the farmer (e.g., weather, global conflicts, etc.).

[0042] In some embodiments, the machine learning model of the present disclosure meets a need for more efficient and robust risk assessment in the specific domains such as agriculture. While these models can benefit from collecting data specific to the domain, such as agriculture, it can be challenging to collect and process such data at scale. For example, agriculture, by its nature,is a sector influenced by many uncontrollable and unpredictable factors, such as weather patterns, pest outbreaks, plant disease, macroeconomic events, geopolitical conflicts, and commodity price volatility. These factors can introduce high variance and irregularities into the data that traditional models struggle to capture effectively. Additionally, variance is also introduced with temporal and spatial differences in agricultural practices across regions. For example, the temporal aspect of agricultural data also introduces seasonality effects that are difficult to account for. As another example, the industry is impacted by intricate policy changes and market dynamics that can vary by location or jurisdiction. Another source of variance is infrequent and inconsistent data collection methods. Other sources of variance can exist in the data as well.

[0043] As a result of the above and other factors, existing data can have a high degree of data sparsity and lack of homogeneity. As a whole, the above and other complexities result in data sets that are small and incomplete. Therefore, machine learning models trained using that data fail to generalize effectively. Therefore, the industry continues to rely on humans alone to perform risk assessment. Therefore, there is a need for automated systems and methods that can use a combination of limited data and expert knowledge to provide robust assessments of risk in situations beyond their original scope of learning.

[0044] Automated credit risk assessment using machine learning can be a promising approach to resolving unique credit risk complexities within the agricultural ecosystem. There are currently no existing studies have examined the out-of-distribution generalization capabilities of proposed models for estimating credit risk. Therefore, a model that provides out-of-distribution generalization capabilities would be an advantage over alternative systems and methods.

[0045] In alternative methods, frequentist machine learning models, despite their extensive use in credit risk assessment, struggle with accounting for uncertainty and generalizing to new data. In contrast, the Bayesian networks employed by embodiments of the present disclosure can encapsulate inherent uncertainty in the data through the use of prior distributions. Bayesian networks can quantify the uncertainty or confidence around point predictions. This attribute becomes particularly advantageous when dealing with data not represented in the training set. Furthermore, when represented graphically on a display, a Bayesian networks can allow a user to understand and validate the model’s learning process. The capabilities of Bayesian networks can potentially enhance the model’s robustness to out-of-distribution generalization, when the model development process is thoughtfully designed. In some embodiments, the present model provides a contribution to the field by demonstrating that Bayesian networks achieve enhanced out-of-distribution performance on credit risk assessment in agricultural finance. For example, in a Bayesian model, when data is lacking for one or more features, it can be accounted for by quantifying the uncertainty of that feature due to the lack of data.

[0046] Bayesian networks can be employed effectively for out-of-distribution generalization. Bayesian networks, by facilitating causal representation learning, are at the forefront of methods that effectively counter spurious correlations by accounting for latent variables. Additionally, Bayesian networks can address data shift, given their structural capacity to integrate shift axes as features. The impact of these shifts on the target variable can be reflected in the prior distribution, which provides model robustness against underlying distribution changes. These capabilities position Bayesian networks as a model that excels at targeting out-of-distribution generalization. In some embodiments, the present Bayesian network builds on this theoretical foundation and offers empirical substantiation for the efficacy of this approach.

[0047] In the quest for robust credit risk models, the emphasis on calibration is paramount, particularly where delinquency definitions differ, such as in Agricultural Finance. Indeed, in Al's application to finance, well-calibrated classification models are arguably of greater relevance to credit risk assessment than those of excellent classification performance alone. Calibration is important for risk assessment because it measures the reliability of the model's predictions; hence well-calibrated models contribute to efficient portfolio management and improved financial performance. In some embodiments, addressing out-of-distribution generalization, as measured by calibration, provides robust and reliable credit risk models.

[0048] In some embodiments, a Bayesian network can overcome challenges in credit risk assessment in agricultural finance, especially in out-of-distribution scenarios. Notably, Bayesian networks can incorporate prior knowledge, which is valuable when data is limited or of low quality, which is the case for domains such as the agricultural credit risk domain. Bayesian networks can also serve against overfitting because of their capacity to assimilate domain expertise contributing significantly to model resilience when tackling out-of-distribution inferences.

[0049] In some embodiments, the Bayesian network’s superior out-of-distribution performance is demonstrated by calibration, which is a relevant metric in this context. In some embodiments, the Bayesian network can predict agricultural loan delinquency based on information such as the sector’s characteristics as derived from small datasets, low-quality data, and the necessity to incorporate domain knowledge. In some embodiments, the model encapsulates the complexities of agricultural credit risk knowledge effectively and provides calibrated estimates for robust out- of-distribution predictions.

[0050] In some embodiments, a dataset is developed that comprises 97,235 agricultural loans granted to 31,900 Brazilian farmers. The dataset is sourced from nine of the largest financial and supply chain institutions in Brazil. The nine institutions are anonymized and shown by capital letters from A to I in this study. The dataset described above is exemplary and can have other amounts of loan data, other amounts of recipients, and other amounts of financial institutions. Due to differing credit policies, the characteristics of the data was significantly different across the nine institutions. The loans were typically issued at the onset of the farming season and repayment was expected to be fully executed by the end of the crop season, upon harvest and sales. In some embodiments, a target variable of the model using this dataset is a dichotomous variable indicating if the loan was unsuccessfully repaid within 90 days of the loan’s due date (e.g., a delinquency event was coded as a hit, or a binary ‘1’). Table 1, below, shows example characteristics of each institution, including a number of loans issued, a count of unique borrowers, a time range of the data, and respective delinquency rates. Notably, the number of loans sources across the institutions varied significantly, ranging from 1,019 loans (see row I, Table 1) to a maximum of 31,095 loans (see row A, Table 1). In addition, delinquency rates varied among these institutions from 1.05% (Institution A) to 22.27% (Institution F). This variability highlights the disparate credit policies across these institutions emphasizing the need for a modeling framework that can effectively generalize out-of-distribution.Table 1

[0051] Each datapoint (e.g., loan) in can be characterized with multiple features (e.g., nine features). In some embodiments, when nine features are used, four of the features are the output of separate models (e.g., scores ranging from 0 to 1000, or having normalized values) designed to characterize key contributors to risk, and five of the features are descriptive of the farmer and their farm. Exemplary features of each category are described below.

[0052] Score features (e.g., from other models)

[0053] Agronomic Score: In some embodiments, an agronomic Score is the output of a model (e.g, a machine learning model) that uses historical agronomic and weather patterns to predict the expected yield per hectare for a given farmer, at a given location, in a given season. A higher yield results in a higher agronomic score. This feature can be useful to the model because of the correlation between crop yield and the farmer’s income, which then becomes the source of the loan repayment.

[0054] Market Score: In some embodiments, market score is the output of model (e.g., a machine learning model) that uses historical commodities sales price, the crop portfolio, product! on / logistics costs, and expected yield to predict the farmer’s operational profits. A higher farmer income results in a higher market score. This feature can be useful because a higher operational margin allows for more cash available to facilitate loan repayment.

[0055] Financial Score: In some embodiments, a financial score is the ratio of: the farmer’s outstanding indebtedness to his / her expected profit. A higher ratio can result in a higher financial score. The financial score can reflect the proceeds available for the repayment of the loan. When this feature is closer to zero, it means that a more significant part of the farmer's profit is used to pay current debts, and less profit remains to repay the requested loan on the due date.

[0056] Behavior Score: In some embodiments, a behavior score is a consumer credit risk score sourced from a credit bureau that is fine-tuned for use in agricultural finance. A higher consumer credit score results in a higher behavior score. Although most of the farming activity signals and a farmer’s financial life are not captured by credit bureau data, they have valuable information about farmer’s consumer behavior, such as paying bills. A good credit bureau score shows that the farmer has good behavior regarding paying bills, even if not related to his / her business. Delay or delinquency on the payment of consumer loans and bills can be an early indication of financial stress or irresponsibility. Such a financial stress may impact the farmer’s business.

[0057] Farm and Farmer Characteristics

[0058] Ratio of Short-Term Debt to Total Planted Area (RSTDTPA): In some embodiments, RSTDTPA represents the ratio of the farmer’s outstanding short-term debt to their total planted area. Short-term debt can be defined as all debts that the debtor is obliged to pay in the next twelve months. The loans in the dataset can have a duration of crop season. Therefore, shortterm debt includes all debts that the farmer should pay before the due date of the loan.

[0059] Ratio of Long-Term Debt to the Farm Area (RLTDFA): In some embodiments, RLTDFA represents the ratio of the farmer’s long-term debt over their total farm area. Long-term debt can be debts that the debtor is obliged to pay in a period longer than one crop season. Long-termdebts can be used for investments in the farm, such as machinery renewal, installing irrigation systems, etc.

[0060] Land Lease Costs: In some embodiments, land lease costs represent the cumulative costs paid by the farmer in the crop season to rent the land from a third party. If the farmer owned their land, this value would be 0.

[0061] Credit History: In some embodiments, credit history represents the credit history of the farmer with the institution from which they are requesting the loan. The feature can be in one of three categories reflecting if the farmer's last loan was (1) successfully paid, (2) delinquent, or (3) if the farmer was a new client in the institution's portfolio.

[0062] Main Crop: In some embodiments, main crop represents the main crop grown by the farmer. Main crop can have eight categories: soybean, summer corn, winter corn, wheat, rice, Arabic coffee, robusta coffee, and sugarcane. The farmer may plant various crops during the loan’s life cycle, but the crop identified as the main crop is the one with the most significant planting area.

[0063] Table 2, below, presents quantifications of data for each feature for a first quartile (QI), second quartile (Q2), and third quartile (Q3) (e.g., financial quarters), and a percentage of data that were missing for each continuous feature, in an example data set. As can be seen, Table 2 illustrates that many of the features are missing for a significant fraction of the loans (e.g., the behavior score is missing for 95.9% of loans). The missing data statistics are consistent with expectations when dealing with agricultural loans sourced from multiple institutions. For a given loan, missing values can be addressed by assigning the median value for any missing features.

[0064] Regarding the categorical variables (e.g., credit history and main crop) of this example data set, there are no missing values. In the example dataset, the credit history feature has 73,621samples paid their last loan successfully, 1,954 samples delinquent on the last loan, and 21,660 samples of new farmers in the intuition's portfolio. In the example dataset, the main crop feature has 66.3% of the samples being soybean, 21.3% being sugarcane, and the remaining were summer corn, Arabic coffee, rice, wheat, robusta coffee, and winter corn in decreasing order of sample size.Table 2Institution QI Q2 Q3 Missing(%)Agronomic score 366 500 625 62.2Market score 275 377 550 62.2Financial score 650 850 950 97.4Behavior score 806 831 852 95.9RSTDTPA 0 17 235 71.3RLTDFA 0 0 157 41.1Land Lease Costs 0 0 50900 81.2

[0065] Assessment of Concept Drift. In some embodiments, data for concept drift (e.g., a change in the distribution of the data) can be assessed across four variables or axes: institutions, crop, state, and year. Concept drift can be assessed using the Jensen-Shannon Divergence (JSD) test. To assess conceptual drift, a tensor of variables can be defined as A = {Institution, State, Crop, Year}, and, for a given feature the following set of JSDs can be defined:

[0066] where fat is the vector of feature values f for all samples belong to axis a(, and a denotes the number of unique values in the axis a. Then, the 90thpercentile of <7 / is computed by: soff) = percentile^, 90).

[0067] Subsequently, the total number of JSD values above the goth percentile for each axis are determined. This count can be calculated by the following relationship:

[0068] where 11 stands for the indicator function, and Jf,ais the subset of <7 / in which their axis is a. Example values for all features and axes are reported in Table 3, below. As can be seen by the example values in Table 3, most data drift happened along the institution axis. In other words, the greatest shifts in the data distribution for this example occurred when the institution was changed.

[0069] Table 3 presents example values of an analysis of concept drift across various features using Jensen-Shannon divergence. The table’s example values quantify the change of distribution when characteristics of the samples vary along the four axes: institution, crop, state, and year. Each cell indicates the number of IDS values above the 90th percentile corresponding to each axis-feature pair. Most data drift occurs in the constitution axis, suggesting significant shifts in data distribution of samples across institutions.Table 3AxisInstitution Crop State YearAgronomic score 8 2 5 0Market score 6 4 4 1Financial score 11 2 0 0Behavior score 3 10 2 0RSTDTPA 6 0 9 0RLTDFA 4 2 7 1Land Lease Costs 0 5 6 0Main crop 7 0 5 0Credit history 14 0 0 0Sum 59 25 38 2

[0070] Because most data drift is observed when the institution is changed among the samples, the label shift is also analyzed inside each institution. To this end, the Mann- Whitney U test isemployed. For all samples inside each institution, the samples are divided into two subsets of successful and unsuccessful loan payments. Then, the Mann-Whitney U test is computed for all continuous features in these two subsets, resulting in p-values. Examples of such p-values are illustrated by Table 4, below. Table 4 illustrates that, for all institutions in this example dataset, there are evident label shifts when analyzing RSTDTPA and financial score features. When an entry is denoted by NA in this table, it means that for the corresponding feature and institution, all samples in the payment and delinquency subsets were identical, and the Mann- Whitney U test cannot be computed. This was true for institutions D, G, H, and I, where the land lease cost was zero.Table 4

[0071] In some examples, the Mann- Whitney U test can be only calculable for numerical variables. For non-numerical variables such as label shift in categorical variables, the x2 test is employed. Table 5, below, illustrates results of the x2 test using an example dataset. The example dataset of Table 5 illustrates that the label shift is noticeable for all institutions when evaluating credit history and main crop features.Table 5

[0072] In some embodiments, a method is employed to assess delinquency risk in agricultural loans. As described above, data in the agricultural finance sector can suffer from several quality issues: (1) sample sizes can be relatively small (see Table 1), many features can be missing values (Table 2) and the data distributions between different institutions can change significantly (Table 3) To account for these data quality issues, human credit analysts can rely heavily on "experts" to provide apriori beliefs about the importance of the features. Therefore, Bayesian networks can be an excellent fit for modeling risk in this domain because of the limitations of the data, and the existence of strong priors from experts position.

[0073] In some embodiments, a Bayesian network is a probabilistic graph model that can illustrates a set of variables (e.g., nodes) and their conditional relationship and dependencies through a directed acyclic graph (DAG). Each node in the DAG can represent a feature of the data. An edge can be a connection between two nodes. An edge can define probabilistic relationship(s) between the two nodes (e.g., a probabilistic relationship linking the corresponding features). A relationship between the two nodes can be a dependency. Dependencies between features can have different strengths. Dependencies are measured by conditional probability distributions. Bayesian networks are capable of integrating observed data with prior knowledge to improve predictions under uncertainty. Bayesian networks bring forth an explicit and natural perception of the dependencies among features. Therefore, they permit comprehensible interpretation and causal reasoning.

[0074] Fig. 1 is a network diagram 100 illustrating a Bayesian network 130 for domain specific loan credit risk assessment according to embodiments of the present disclosure. The Bayesian Network 130 outputs credit performance 120 as its model target. The Bayesian network 130 can utilize expert knowledge in the agricultural credit risk domain to construct the structure of the network (e.g., which nodes to use and how to connect the nodes). In other words, expert knowledge can define the nodes (e.g., features, variables) of the network and their relationships based on a deep understanding of the actual conditions influencing credit delinquency in agribusiness. Therefore, the proposed network structure is a close representation of the real- world intricacies of credit delinquency scenarios in agricultural loans.

[0075] In some embodiments, the Bayesian network 130 begins its assessment of a loan application based on a main crop (e.g., crop type) 102. The crop type 102 can impact the agronomic score 112, (e.g., expected crop yield production), market score 114 (e.g., expected operational profit), financial score 116 (e.g., the ratio between the debts and profit), and credit performance 120. Thus, edges between the main crop node 102 and agronomic score 112, market score 114, financial score 116, and credit performance 120 define respective probabilistic relationships.

[0076] In some embodiments, an edge connects the agronomic score node 112 to the market score node 114 because higher expected crop yield production implies higher expected operational profit and lower expected crop yield production implies lower expected operational profit. In some embodiments, an edge connects the agronomic score 112 to the credit performance node 120.

[0077] The market score 114 is connected to the credit performance node 120 by an edge because the farmer’s operational profit amount is directly related to the loan repayment capacity.Additionally, the market score node 114 is connected to the financial score node 116 because the financial score is the ratio between the farmer's debt and profit.

[0078] The RLTDFA node 104 is linked to the RSTDTPA node 106 and credit performance node 120. The edge between the RLTDFA node 104 and the RSTDTPA node 106 can represent installments of the long-term debts that the debtor is obligated to pay on a short-term basis (e.g., within the following twelve months of the loan request date). Those payments can be considered short-term debts. However, the RLTDFA node 104 does not need to be directly connected to the financial score because it impacts this score indirectly via the RSTDTPA node 106.

[0079] In some embodiments, the credit history node 110 is connected to three nodes. The credit history node 110 can be connected to the RSTDTPA 106 because if the farmer did not pay the last loan, they are carrying the debts of the previous season to the current season, which should be paid in the following twelve months. Also, the credit history node 110 can be connected to the behavior score 118 because this score is a measure of the farmer's behavior on consumer loans and daily bill payments. For example, when the previous season loan has not been paid successfully, there is a probability that some consumer loans or bills have defaulted because they were less critical from the farmer's perspective and not related to his / her business. In addition, the credit history node 110 node is directly connected to the credit performance 120 because the last loan performance is an indicator of the subsequent loan output, as observed in the label shift analysis in Table 5, above.

[0080] In some embodiments, an edge between the RSTDTPA node 106 and the financial score node 116 represents that the short-term debt is used in the financial score computation. In some embodiments, the RSTDTPA node 106 is connected to the credit performance 120. Asillustrated by Table 4, RSTDPA has a different distribution between successfully paid and delinquency samples in all institutions.

[0081] In some embodiments, the land lease costs node 108 is linked to the financial score node 116 because the land lease costs are similar to the short-term debt. That is, the land lease should be paid during the season, at least partially, and therefore is used in the financial score computation.

[0082] In some embodiments, the financial score node 116 is connected to the behavior score node 118 and credit performance nodes 120. The financial score can be a measure of the farmer's indebtedness and can describes the farmer's capacity for successful payments. Therefore, besides the credit performance, the financial score impacts the credit bureau data and, consequently, the behavior score.

[0083] In some embodiments, the behavior score node 118 is linked to the credit performance node 120 because negative records in the consumer loan and bill payments are indicators of financial stress, which can impact the farmer's business.

[0084] In some embodiments, the present method employs a discretization approach. For data to be analyzed using the Bayesian Network described above, all continuous features were first discretized. The bins have thresholds defined such that, when a feature was shifted from one category to another, a significant change in the downstream risk attributable to that feature can be changed according to the probabilistic relationships defined in the edges. The prior conditional probability distribution of the credit performance node is defined based on training data of respective features. The priors can manage the heterogeneity of data drawn from population-wide information and assumptions about agricultural credit risk. These priors can enable incorporating additional knowledge into the proposed model, filling in the gapsintroduced by heterogeneous data. This integration was intended to improve the model’s robustness and overall performance in the out-of-distribution generalization.

[0085] In some embodiments, three baseline models can be employed: extreme Gradient Boosting (XGBoost), L2 penalized Logistic Regression, and Artificial Neural Networks. However, it can be recognized that other models can be used in addition to or in replacement of any of these models. The selected models can represent an example modeling framework. All model hyperparameters can be selected using Bayesian optimization.

[0086] Fig. 2 is a block diagram 200 illustrating a model factory 206 generating a directed graph 208 according to embodiments of the present disclosure. A user provides a plurality of conditional dependencies 202 of respective features. It can be recognized that other amounts of features and other configurations of dependencies can be employed.

[0087] In Fig. 2, these features can be represented as A, B, C, and T, where T is the target feature or output feature. For each two connected feature, an arrow represents a conditional probability of a first feature given a second feature (e.g., P(FI|F2), where Fi is a first feature and F2 is a second feature). The arrow can represents the first feature at the arrowhead end and the second feature at the originating end of the line.

[0088] In some embodiments, training data 204 includes data relating features to each other.The training data 204 can be gathered from existing data, such as existing loan applications and their outcomes. The training data 204 can include model scores or characteristic data. In some embodiments, the training data 204 can be cleaned or pre-processed. For example, any missing features in the training data 204 can be replaced by average values (e.g., median, mode, mean) or by another metric. While Fig. 2 illustrates one correlation between each pair of dependent features, the training data 204 can include larger data sets of many examples of each pair.T1

[0089] A model factory 206 calculates conditional probabilities of each dependency 202 based on the training data. Then, the model factory 206 generates a directed graph 208 having the dependencies of 202 and the conditional probabilities (e.g., probability distributions) assigned to each edge of the graph (e.g., Ei, E2, E3, E4, and Es).

[0090] Fig. 3 is a diagram 300 illustrating the use of a directed graph 308 according to embodiments of the present disclosure. In some embodiments, a loan application 302 is provided. The loan application 302 includes values of one or more features in common with the directed graph 308. The one or more features of the loan application 302 are provided to a financial institution 306 (e.g., a computer, server, or cloud computing service of the financial institution). The financial institution 306 also receives the directed graph 308. The financial institution 306 applies one or more features of the loan application 302 to the directed graph 308. The directed graph 308 can therefore output a credit analysis 304 including a grant or denial of a loan.

[0091] In some embodiments, the Bayesian Network can be configured to explore different dependencies and model complexities. For example, the Bayesian network can be configured through Node Addition / Removal, Edge modification, and the use of hierarchical / dynamic models. Nodes can be added or removed based on the availability of data and relevance to the credit performance task. For example, when additional data on weather patterns becomes available, a node representing that factor can be incorporated.

[0092] The structure of the edges can also be altered to represent different hypothesized relationships. For example, edges can be added to capture indirect effects or removed to test or verify a simpler models. More complex hierarchical Bayesian networks can model multi-level dependencies, such as those between different geographic regions or crop types. TemporalBayesian networks can be implemented to account for time-varying factors, capturing the evolution of risk over multiple crop seasons.Enhancing Credit Risk Reports Generation with Language Models: An Integration of Bayesian Networks and Labeled Guide Prompting

[0093] Fig. 4 is a diagram 400 illustrating an example embodiment of providing an engineered prompt 402 to a language model 404 according to embodiments of the present disclosure. As described in relation to Fig. 2, a model factory 206 generates a directed graph 208 based on dependencies 202 and training data 204. In relation to Fig. 4, the directed graph 208 is converted to an engineered prompt 402. In the example of Fig. 4, an engineered prompt 402 can describe the structure of the directed graph as with the following: “The Bayesian network structure is as follows: A is connected to B, D, and T (the output), B is connected to T (the output), D is connected to B.”

[0094] A language model 404, such as a large language model, receives the engineered prompt 402 and a loan application 410 having one or more features in common with the directed graph 208. The language model 404 then applies the features of the loan application 410 to the directed graph 208 and generates a credit risk assessment report 406.

[0095] It can be appreciated that the engineered prompt 402 can be generated by the dependencies 202 (not shown). In such a case, the language model 404 can separately receive the conditional probabilities of each edge of the graph in order to process the input data (e.g., the loan application) and generate the credit risk assessment report 406.

[0096] In some embodiments, language models (e.g., large language models, GPT-4, GPT-4.0 etc. can be applied to generate a credit risk report that traditionally is produced by human analysts. In some embodiments, engineered prompts are provided to the language model havingcredit risk report specifications. In some embodiments, model(s) including Bayesian networks are used to support the language model’s reasoning tasks. For example, the Bayesian network is provided to the language model as part of the prompt. Such a model (e.g., a graphical model) guides the language model to explain the impact of each variable on the target, align with the network structure, produce plausible scenarios, and promote abductive reasoning.

[0097] In some embodiments, a prompt engineering technique is employed, sometimes referred to herein as “Labeled Guide Prompting.” Labeled Guide Prompting ensures that the language model correctly addresses the specific aspects of the task.

[0098] The study's findings, indicating a statistically significant preference for GPT-4 generated reports, confirm the LLM's ability to generate credit risk reports. In addition, our results show a statistically significant improvement in the insightfulness of the reports generated by GPT-4 post-prompt engineering, as measured by well-defined metrics. Consequently, we present a robust and user-centric approach for automating credit risk reports using LLMs, hold promising implications for the future integration of language models in credit risk assessment.

[0099] Credit risk analysis is a complex process that involves a wide range of cognitive abilities, including contextual understanding, logical reasoning, the application of domain- specific knowledge, implicit reasoning, and causal reasoning. Evaluating the performance of Large Language Models (LLMs) in this domain provides crucial insights into their practical utility in real-world scenarios. In some embodiments, language models are applied to generate comprehensive credit risk reports — a critical task in financial decision-making. Using language models, such as GPT-4, in this context provides effective reports for this complex task.

[0100] Automating the process of credit risk analysis can transform the finance industry. By leveraging LLMs can streamline processing vast amounts of data, enabling real-time analysis,surpassing traditional methods. Furthermore, automation can also contribute to more consistent and objective analyses. While human analysts might be influenced by biases or varying expertise levels, using LLMs for this task can maintain a standard level of analysis based on its prompt engineering infrastructure. The importance of this task is due to the growing influence of automation and artificial intelligence in the finance sector. Using language models in this domain can increase artificial intelligence use for credit risk management. Language model enhancements can translate to significant efficiency gains and better decision accuracy.

[0101] There are several challenges of using language models (e.g., large language models such as GPT-4). First, credit risk analysis itself can be multifactorial and can consider many features (e.g., variables, factors, etc. including personal credit history, current financial standing, broader economic conditions, and industry-specific factors. Second, in addition to the multifactorial nature of the analysis, financial data can have a dynamic nature that can frequently change, and sometimes in a volatile manner. Third, credit analysis is a task traditionally performed by credit analysts having a deep understanding of financial dynamics and risk. For a language model to perform credit analysis, that model should perform both generalized data analysis and employ specialized knowledge and intuition of credit analysts. While language models are usually employed for generalized data analysis, they are not employed for specialized knowledge, and especially in credit analysis. Current language models cannot generate complex comprehensive credit risk reports because doing so requires combining domain knowledge, contextual understanding, and multiple types of reasoning. Therefore, a prompt engineering strategy is generate a credit risk report.

[0102] In alternative methods, language models and prompt engineering strategies can yield unpredictable responses when handling unseen data, anomalies, or changing requirements. Thisunpredictability is a significant concern in financial decision-making because a consistent quality of response, insight, and analysis is needed. In some embodiments, a prompt engineering method standardizes the quality and content of the language model’s output. As a result, the output of the language model is more predictable and insightful, even for handling complex and dynamic tasks like credit risk analysis.

[0103] Language models can provide substantial advancements in architecture, training methods, and real-world applications. For example, a transformer model can be used for self-attention mechanisms, leading to significant improvements in many natural language processing (NLP) tasks. As another example, a BERT language model can employ bidirectional transformers. These bidirectional transformers can allow models to understand the context of a word based on all of its surroundings (e.g., tokens before and after, or left and right, of the word).

[0104] In some language models, such as GPT-3, scaling laws can confirm that as models increase in size, they continue to improve in performance. For example, GPT-3 demonstrated few-shot learning capabilities, enabling the model to generalize from a small number of examples. In another example, GPT-4 can extend these principles.

[0105] Language models, such as LLMs, have shown competitive performance compared to traditional machine translation models, even without specific training in translation tasks. In addition, these language models can be used for summarization tasks, generating concise summaries of lengthy documents, generating high-quality dialogue for conversational agents, and writing essays that can pass for human written.

[0106] Prompt engineering and few-shot learning can be employed for effective utilization of language models. Prompt engineering and few-shot learning guides language models towards more desirable and task-specific outputs. Prompt engineering can be the process of structuringan instruction that can be interpreted and understood by a generative Al model. Prompt engineering can involve structured inputs that provide context to and desired outcome of a task to the model. A quality prompt can narrow down the model’s broad knowledge to a specific task. Few-shot learning is a type of prompt engineering that can present the language with a small number of examples of a task, thereby assisting the model in identifying and applying the correct pattern for the given task.

[0107] As part of prompt engineering, a task for the language model can be formulated. Task formulation can include defining a clear objective, often expressed as a question or a statement for which the model is to complete, expand, or react.

[0108] Subsequently, the performance of the language model can be evaluated for that task. Evaluation can include matching the model’s output against a standard or specific criteria. For example, GLUE and SuperGLUE are accepted benchmarks that serve to evaluate a model's performance.

[0109] Abductive reasoning can include generating a plausible explanation (e.g., the most plausible or probable explanation) for a given set of observations. Prompt engineering can enable a language model to not only provide a prediction, but also to generate plausible explanation(s) of that prediction. Providing abductive reasoning from the language model allows generation of more informative and nuanced credit reports. For example, the language model can generate an explanation regarding how different features (e.g., financial indicators) interact and influence an entity’s credit risk score. This can make the assessment process more transparent and understandable.

[0110] In some embodiments, a prompt engineering method can be referred to as “labeled guide prompting.” Labeled guide prompting can be designed to guide the LLM to consider andrespond to multiple aspects of a task, including answering questions of the form “what?”, “why?” and “how?”. Labeled guide prompting in combination with a Bayesian network representation can promote abductive reasoning in Large Language Models (LLMs). This approach can cause the model to generate a hypothesis (e.g., a “best guesses”) that explains the observed phenomena according to the Bayesian network structure. The prompt can include explicit labeling of output elements, which thereby can provide control over responses. Such a prompt can cause the model to produce a specific, well-structured output. Consequently, such prompts can cause the model to provide logically coherent and insightful responses, enhancing language model’s efficacy in complex problem-solving tasks.

[0111] In some embodiments, labeled guide prompting can guide a language model (e.g., GPT- 4) to generate structured and insightful credit risk reports. Labeled guide prompting can increase frequency and quality of interactions in reports, thereby enhancing their insightfulness and usefulness in the context of credit risk assessment.

[0112] In some embodiments, language models such as LLMs can be employed with prompt engineering and Bayesian models to assess credit risk. In some embodiments, language models such as LLMs can be employed with prompt engineering and Bayesian models to perform other financial risk tasks.

[0113] In alternative methods, credit risk reports are created manually by human analysts. However, in some embodiments of the present disclosure, a language model can automate and enhance the result of this task. For example, in one experiment using language models to evaluate 100 credit applications, independent evaluators showed a statistically significant preference for the language model generated reports compared to manually generated report.

[0114] In some embodiments, graph models or Bayesian networks are employed as a prompt input to the language model. Employing graph models such as Bayesian networks can provide an understanding of an influence of each feature on the target feature, align with the network structure, and produce multiple plausible outcomes of the credit loan application using abductive reasoning.

[0115] Labeled guide prompting can cause the language model to generate a report that accurately addresses aspects of credit risk analysis. This not only improved the quality of the generated reports but also demonstrated the broad potential of this technique in other sophisticated tasks. The language, using engineered prompts, showed a statistically significant improvement in the insightfulness of the reports, as measured by well-defined metrics.

[0116] In some embodiments, a language model (e.g., an LLM such as GPT-4) is employed to generate credit risk reports that surpass the quality of traditional reports written by human analysts. A prompt engineering method can be designed to cause the model to generate a report having content specifications. The content specification can include generating a comprehensive credit risk report that quantify and explain credit risk in terms of a probability of delinquency and score corresponding to the same. The language model can receive a credit application as input and determine borrower’s information from that application. The report can provide support for credit decisions for a financial institution, such as approving the loan or rejecting the loan. In some embodiments, the financial institution can automatically or approve or reject the loan based on the reports. In some embodiments, the financial institution can automatically provide a provisional approval or provisional rejection of the loan and allow a human user to review that provisional decision. Accordingly, the report can provide an explanation of the impact of all risk factors to ensure completeness and can assess the borrower's risk profile.

[0117] In some embodiments, the language model can be trained on credit application data be obtained from a financial institution (e.g., a representative set of lenders, such as lenders within an agricultural supply chain). The individual credit applications of the credit application data can be filled out by farmer borrowers and can include information necessary for a thorough credit risk assessment in an agricultural context. This risk assessment task can be handled by the financial institution. The financial institution’s system can process a data from a new credit application and calculate a probability of delinquency alongside a corresponding credit score for each application.

[0118] To assess the utility and quality of the credit risk reports, an independent evaluation was conducted by a team of seven credit analysts external to a financial institution for example outputs of the model. These analysts, industry professionals from Brazil's agricultural finance sector, were blind to whether the reports were Al generated or human generated. This blindreview approach ensured an unbiased comparison of the reports based on their practical utility in credit risk assessment.

[0119] The evaluators were not aware that the comparison's purpose was to compare human and Al-generated reports. They believed they were comparing two different reports to determine which would be more useful in making a credit decision. Six evaluators reviewed reports in Portuguese, while one proficient in English reviewed reports in that language. They had three days to assess the assigned cases. The evaluation process consisted of a pair of credit reports for each credit application and a questionnaire for evaluators to complete. The questionnaire was composed of three questions: a. Which report was more helpful for you to assess the credit risk?b. Do any of the reports have any information that is not true or does not make sense? c. Why do you prefer the chosen report?

[0120] The design of this questionnaire ensured an objective evaluation of the reports, while the open-ended question provided insight into the evaluators' reasoning behind their choices. The evaluators preferred the language model generated reports by a statistically significant margins. Therefore, there are clear advantages for generating credit reports with language models.

[0121] Table 6: Example of a Bayesian Network Structure as a Prompt Component

[0122] Fig. 5 is a diagram 500 illustrating a prompt generation sever 206 according to embodiments of the present disclosure. A prompt generation server 206 receives dependencies 202 and a training data probability distribution 502. In some embodiments, the prompt generation server 206 can receive a directed graph resulting from the dependencies 202 (not shown by Fig. 5). The prompt generation server 206 can then generate an engineered prompt 506 including the structure of the dependencies 202, and one or more of a role, an instruction, model input(s), requested model output(s), items to be addressed, and few-shot learning instances.

[0123] In some embodiments, a probabilistic graphical model (e.g., a Bayesian network) can be used to estimate the probability of delinquency. Using a Bayesian network can be motivated bytwo primary considerations. First, Bayesian networks can provide robust predictive performance. Second, Bayesian networks have a structure that can allow computation of a joint probability, which can enhance the interpretability of the model output. Therefore, a Bayesian network can identify pathways of influence and interactions among variables. In some embodiments, the Bayesian network structure offers a framework conducive to engaging the LLM with complex tasks inherent in credit risk reports. The Bayesian Network can facilitates reasoning across multiple dimensions, including: a. Common-Sense Reasoning: Providing informative variable names and a sensible structure can cause the language model to generate easily accessible explanations. b. Logical-Deductive Reasoning: The language model can propose a general rule that prompts the language model to deduce specific cases that align with this rule. c. Logical-Abductive Reasoning: Factorization of the joint distribution according to the network can identify plausible scenarios (e.g., both pathways and interactions) given the information at hand. d. Implicit Reasoning: The network paths can offer a natural means of decomposing relationships into intermediate steps. e. Causal Reasoning: The structure of Bayesian networks can enable the identification and interpretation of potential causal relationships.

[0124] In some embodiments, the Bayesian network is defined so the relationships between each variable and the target are intuitive. When the network is provided to the language model, that language model can understand and predict the relationship between each variable and the target, predicated on the variable’s name, tag, or ID. The model can include setting appropriate priorsand analyze how the target estimate fluctuates in response to changes in the values of the features.

[0125] In some embodiments, prompt engineering provides a prompt with one or more components that guide the model to generate comprehensive and insightful credit risk reports. These components are listed below and can be used alone or in combination with the structure of the model. a. Role and Instruction: In one example, the role can provide that the language model is a data scientist expert in agricultural finance. This role provides the context and performance expectations for the language model. The instruction causes the language model to generate an in-depth credit risk report that explains the probability of delinquency and the corresponding credit risk score accounting for both the provided data inputs and the underlying network structure. b. Model Inputs: Model inputs can be the financial details given in the credit application. These inputs are the data for the language model to examine, interpret and on which to base its report. c. Model Structure: In some embodiments, the model structure is provided as shown in Table 6. The model structure provides a structure of the Bayesian network, including how nodes are connected to each other and which node is the output or target node. The language model can use the network structure to comprehend and reason about the dependencies between various factors, and how they contribute to the credit risk. In some embodiments, the model structure can comprise conditional joint probabilities for each node connection. In some embodiments, the model structure can comprise the connections between nodeswithout any conditional joint probabilities, and the weights to each edges are calculated by the language model based on training data also provided to the language model. d. Model Output: In some embodiments, the model outputs a computed credit risk score. The model output is part of the report. The language model provides a report that elaborates on the credit risk score based on the inputs and the structure of the Bayesian network. e. Items to be Addressed: In some embodiments, items to be addressed are additional requests that the language model address in the report. These aspects cam stem from the Bayesian network’s learning and inference process, thereby enabling the LLM to carry out reasoning tasks with greater accuracy. The items to be addressed can include: i. The direction of the impact of each model input on the output (e.g., does this model input increase or decrease the output?) ii. The network pathway from input to output (e.g., how does this model input impact the output, considering the network structure?) iii. Input interactions affecting the output (e.g., how does this model input interact with neighboring inputs to impact the output?) f. Few-shot Learning Instances: Few-shot learning instances can be samples of completed tasks that the LLM can learn from. By generalizing from these instances to the present task, the LLM can understand the structure, style, and content of an effective credit risk report. In a few-shot learning prompt, each example can be marked with an identifier, and the prompt can include responsesthat provide an expanded explanation in relation to one of the few-shot examples by providing one of those identifiers. In responses to a few-shot learning prompt, portions of the response that are responding to one of the examples of the fewshot learning prompt can be marked as corresponding to that example. As a result, the language model can provide more of a response to address the proposed item.

[0126] In some embodiments, the prompt generation server 206 can optionally receive other prompt tasks 504. Examples of the other prompt tasks 504 can include one or more of: a. Contextual Question- Answering: The language model can accurately interpret the comprehensive instruction, which includes explaining the credit risk score based on the model's structure, input, and output. b. Domain-Specific Tasks: The language model can demonstrate proficiency in one or more particular domains, such as agricultural finance and / or credit risk to generate a relevant and accurate report. c. Decomposition: The language model can explain the impact of each feature of the Bayesian model on the target feature. The language model can provide a granular explanation that aligns with the Bayesian network structure. d. Common-Sense Reasoning: The language model must output a report describing a plausible scenario given the inputted feature values to justify the estimated target feature. e. Logical Reasoning: The language model can calculate and describe an impact of the features on the target feature based on the network structure, and can engage in abductive reasoning to provide a plausible or probabilistic explanation of how the model inputs yield the model’s output.f. Implicit Reasoning: The language model can output implicit steps based on the network’s structure. For example, the language model can explain how a feature impacts the target through various pathways or interactions. g. Causal Reasoning: The language model can identify potential causal relationships between the features and the target variable as suggested by the network structure.

[0127] Fig. 6 is a flowchart 600 illustrating a method of training a model according to embodiments of the present disclosure. The method can include reading a plurality of dependencies (602). Each dependency can be between a pair of features of a plurality of features. The plurality of features can include a target feature. The method can include reading training data correlating each pair of features (604). The training data can comprise loan application data for a plurality of loans. The method can include determining, from the training data, a conditional probability for each pair of features (606). Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature. The method can include constructing a directed graph (608). The constructed directed graph can comprise a plurality of nodes and a plurality of edges. Each of the plurality of nodes can correspond to one of the plurality of features. Nodes corresponding to a dependency of a pair of features can be connected with one of the plurality of edges. The method can include assigning, for each pair of features, the determined conditional probability to the edge corresponding to that pair (610).

[0128] Fig. 7 is a flowchart 700 illustrating a method of using a model according to embodiments of the present disclosure. The method can comprise loading a directed graph(702). The directed graph can comprise a plurality of nodes and a plurality of edges. Each of thenodes can correspond to one of a plurality of features. Nodes corresponding to a dependency of a pair of features can be connected by one of the plurality of edges. Each edge can represent a conditional probability for each pair of features. Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature.

[0129] The method can further comprise reading a plurality of financial features of an entity (704). The method can further comprise applying the plurality of financial features to corresponding nodes of the directed graph (706). The directed graph can thereby generate a target feature. The method can further comprise outputting, based on the target feature, a recommendation to provide or deny lending to the entity (708).

[0130] Fig. 8 is a flowchart 800 illustrating an example embodiment of a method of using a language model to generate a risk score according to embodiments of the present disclosure. The method can comprise reading, by a language model, a text representation of a directed graph (802). The text representation can comprise a representation of plurality of dependencies. Each dependency can be between a pair of features of a plurality of features and a representation of a conditional probability for each pair of features. The plurality of features can include a target feature. Each pair of features can comprise a first feature and a second feature. The conditional probability can represent a probability of the first feature based on an occurrence of the second feature. The method can further comprise reading, at the language model, loan application data of an entity (804). The method can further comprise applying, by the language model, the loan application data to the directed graph (806). The method can further comprise generating, by the language model, a risk score on the loan application data of the entity (808).

[0131] In some embodiments, the above components alone or in any combination form a coherent and efficient framework that can guide the language model in generating comprehensive, insightful, and practical credit risk reports.

[0132] Fig. 9 is a diagram illustrating a computing node according to embodiments of the present disclosure. Computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, computing node 10 is capable of being implemented and / or performing any of the functionality set forth hereinabove.

[0133] In computing node 10 there is a computer system / server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0134] Computer system / server 12 may be described in the general context of computer systemexecutable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system / server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through acommunications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0135] As shown in Fig. 9, computer system / server 12 in computing node 10 is shown in the form of a general-purpose computing device. The components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0136] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0137] Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system / server 12, and it includes both volatile and non-volatile media, removable and non-removable media.

[0138] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown andtypically called a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

[0139] Program / utility 40, having a set (at least one) of program modules 42, may be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments as described herein.

[0140] Computer system / server 12 may also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 22. Still yet, computer system / server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer system / server 12 via bus 18. It should be understood that although notshown, other hardware and / or software components could be used in conjunction with computer system / server 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0141] The present disclosure may be embodied as a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0142] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0143] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0144] Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmablegate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0145] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0146] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0147] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce acomputer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0148] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0149] The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: reading a plurality of dependencies, each dependency being between a pair of features of a plurality of features, the plurality of features including a target feature; reading training data correlating each pair of features, the training data comprising loan application data for a plurality of loans; determining, from the training data, a conditional probability for each pair of features, wherein each pair of features comprises a first feature and a second feature, and wherein the conditional probability represents a probability of the first feature based on an occurrence of the second feature; constructing a directed graph, the directed graph comprising a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes correspond to one of the plurality of features, and wherein nodes corresponding to a dependency of a pair of features are connected by one of the plurality of edges; and assigning, for each pair of features, the conditional probability to the edge corresponding to that pair.

2. The computer-implemented method of Claim 1, wherein the loan application data comprises one or more of the plurality of features and the target feature, the target feature being an outcome of each loan.

3. The computer-implemented method of Claim 2, wherein the loan application data comprises agricultural loan application data.

4. The computer-implemented method of any one of Claims 1-3, wherein the directed graph is a Bayesian network, and the Bayesian network is configured to output a probability of the target variable, the probability representing a credit risk assessment based on financial data of an entity.

5. The computer-implemented method of any one of Claims 1-4, wherein plurality of dependencies includes (a) the target variable depending on a first variable of the plurality of variables, and (b) the first variable depending on a second variable of the plurality of variables.

6. The computer-implemented method of any one of Claims 1-5, wherein the plurality of variables comprises score features, the score features being derived from another computer-implemented model.

7. The computer-implemented method of Claim 6, wherein the score features include one or more of an agronomic score, a market score, a financial score, and a behavior score.

8. The computer-implemented method of any one of Claims 1-7, wherein the plurality of variables comprises data based features, the data based features loaded from a database.

9. The computer-implemented method of Claim 8, wherein the data based features include a ratio of short-term debt to total planted area (RSTDTPA), a ratio of long-term debt to the farm area (RLTDFA), a land lease cost, a credit history, and a main crop.

10. The computer-implemented method of any one of Claims 1-9, further comprising: converting the training data into discrete values.

11. The computer-implemented method of any one of Claims 1-10, further comprising: calculating a measure of concept drift for one or more of the plurality of features; and normalizing the features based on the measure of concept drift.

12. A computer-implemented method comprising: loading a directed graph, the directed graph comprising a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes correspond to one of a plurality of features, wherein nodes corresponding to a dependency of a pair of features are connected by one of the plurality of edges, wherein each edge represents a conditional probability for each pair of features, wherein each pair of features comprises a first feature and a second feature, and wherein the conditional probability represents a probability of the first feature based on an occurrence of the second feature; reading a plurality of financial features of an entity; applying the plurality of financial features to corresponding nodes of the directed graph, the directed graph thereby generating a target feature; and outputting, based on the target feature, a recommendation to provide or deny lending to the entity.

13. The computer-implemented method of Claim 12, further comprising: reading a loan application; and extracting the plurality of financial features from the loan application.

14. The computer-implemented method of Claim 13, wherein the loan application comprises an agricultural loan application.

15. The computer-implemented method of any one of Claims 12-14, wherein the generated target feature provides a credit risk assessment based on the financial features of the entity.

16. The computer-implemented method of any one of Claims 12-15, wherein plurality of dependencies includes (a) the target feature depending on a first feature of the plurality of features, and (b) the first feature depending on a second feature of the plurality of features.

17. The computer-implemented method of any one of Claims 12-16, wherein the plurality of features comprises score features, the score features being derived from another computer-implemented model.

18. The computer-implemented method of Claim 17, wherein the score features include one or more of an agronomic score, a market score, a financial score, and a behavior score.

19. The computer-implemented method of any one of Claims 12-18, wherein the plurality of features comprises data-based features, the data-based features loaded from a database.

20. The computer-implemented method of Claim 19, wherein the data based features include a ratio of short-term debt to total planted area (RSTDTPA), a ratio of long-term debt to the farm area (RLTDFA), a land lease cost, a credit history, and a main crop.

21. The computer-implemented method of any one of Claim 12-20, further comprising:calculating a measure of concept drift for one or more of the plurality of financial features; and normalizing the financial features based on the measure of concept drift.

22. The computer-implemented method of any one of Claims 12-21, wherein the directed graph is a Bayesian network.

23. A computer-implemented method comprising: reading, by a language model, a text representation of a directed graph, wherein the text representation comprises a representation of plurality of dependencies, each dependency being between a pair of features of a plurality of features, and a representation of a conditional probability for each pair of features, the plurality of features including a target feature, wherein each pair of features comprises a first feature and a second feature, and wherein the conditional probability represents a probability of the first feature based on an occurrence of the second feature; reading, at the language model, loan application data of an entity; applying, by the language model, the loan application data to the directed graph; and generating, by the language model, a risk score on the loan application data of the entity.

24. The computer-implemented method of Claim 23, wherein the loan application data includes one or more of the entity’s financial status and previous credit history.

25. The computer-implemented method of any one of Claims 23-24, further comprising:generating, by the language model, a description of a set of features that influenced generation of the risk score.

26. The computer-implemented method of any one of Claims 23-25, further comprising reading, at the language model, a prompt to provide a description of the set of features that influenced generation of the risk score.

27. The computer-implemented method of any one of Claims 23-26, wherein the directed graph is a Bayesian network.

Images