Ensemble of machine learning models to calculate the probability that an entity does not satisfy the target parameters

Abstract

A method for generating a machine learning (ML) model ensemble for calculating the probability that an entity does not satisfy a target parameter is provided, the method comprising: training a plurality of ML sub-models, each of which outputs a sub-value for an input of raw data elements; training a main ML model, using a training dataset including the ML sub-values ​​and corresponding entity parameters for sample entities, for which the main ML model outputs a value of the entity parameter corresponding to the target parameter for an input of the sub-values; inputting the raw data elements related to the entities into the ML sub-models to obtain their respective sub-values; calculating simulated adjustments to the sub-values ​​to generate adjusted sub-values ​​that are input into the main ML model in an iteration to obtain simulated values ​​of the entity parameter; and generating a risk classifier that generates the probability that the entity does not satisfy the target parameter according to an analysis of the simulated values ​​for the entity parameter.

Description

[Technical field]

[0001] [Related Applications] This application claims the benefit of priority to U.S. Provisional Patent Application No. 16 / 907,251, filed June 21, 2020, the contents of which are incorporated herein by reference.

[0002] The present invention, in some embodiments thereof, relates to machine learning (ML) and, more particularly, but not by way of limitation, to systems and methods for training and utilizing ensembles of ML models to calculate the risk that an entity fails to satisfy target parameters. [Background technology]

[0003] Predictions of whether a particular entity can satisfy a target value are usually based on an analysis of the entity's history. Past success in satisfying a target in the past indicates future success in satisfying the target, and a history of failures in satisfying a target history usually indicates predicted failure to satisfy the target in the future. Another approach is to observe other entities. The success or failure of other entities in satisfying their target history is used as an indication of the current entity's ability to satisfy its own target. Summary of the Invention

[0004] According to a first aspect, a method for generating a machine learning (ML) model ensemble for calculating a likelihood of an entity not satisfying a target parameter comprises: training a plurality of ML sub-models, each outputting a sub-value in response to input of a raw data element; training a main ML model for each of a plurality of sample entities, the main ML model outputting a value of an entity parameter corresponding to the target parameter in response to input of the plurality of sub-values ​​output by the plurality of ML sub-models using a training dataset including the plurality of ML sub-values ​​and corresponding entity parameters; inputting a plurality of raw data elements related to the entity to the plurality of ML sub-models to obtain respective sub-value outputs; calculating a plurality of simulated adjustments to the plurality of obtained sub-values ​​in a plurality of iterations to generate a plurality of adjusted sub-values, inputting the plurality of adjusted sub-values ​​to the main ML model, obtaining a plurality of simulated values ​​for the entity parameter from the main ML model in the plurality of iterations; and generating a risk classifier for generating a probability of the entity not satisfying the target parameter according to an analysis of the plurality of simulated values ​​for the entity parameter.

[0005] According to a second aspect, a method for dynamically and iteratively utilizing an ensemble of ML models for calculating a probability that an entity does not satisfy a target parameter includes receiving, in a first number of iterations, a request from the entity to obtain the target parameter; receiving a number of raw data elements extracted by code sensors installed on a number of network nodes and populated in real time to a server; inputting the raw data elements into a number of ML sub-models; obtaining a number of sub-values ​​from the number of ML sub-models; and, in a second number of iterations, performing a number of simulations on the number of obtained sub-values ​​to generate a number of adjusted sub-values. calculating an adjusted adjustment based on the entity parameter from the main ML model in the second iterations; inputting the adjusted sub-values ​​into a main ML model; obtaining a plurality of simulated values ​​of the entity parameter from the main ML model in the second iterations; inputting the plurality of simulated values ​​into a risk classifier that calculates a probability that the entity will not satisfy the target parameter according to an analysis of the plurality of simulated values ​​of the entity parameter; automatically providing the entity with the target parameter when the probability is below a threshold; and automatically denying a request for the target parameter when the probability is above the threshold.

[0006] According to a third aspect, a system for generating a machine learning (ML) model ensemble for calculating a probability that an entity does not satisfy a target parameter includes at least one hardware processor executing code for: training a plurality of ML sub-models, each outputting a sub-value in response to input of a raw data element; training a main ML model utilizing a training dataset including a plurality of ML sub-values ​​and corresponding entity parameters for each of a plurality of sample entities, for outputting a value of an entity parameter corresponding to the target parameter in response to input of the plurality of sub-values ​​output by the plurality of ML sub-models; inputting a plurality of raw data elements related to the entity to the plurality of ML sub-models to obtain an output of each sub-value; computing a plurality of simulated adjustments to the plurality of obtained sub-values ​​in a plurality of iterations to generate a plurality of adjusted sub-values; inputting the plurality of adjusted sub-values ​​to the main ML model; obtaining a plurality of simulated values ​​of the entity parameter from the main ML model in the plurality of iterations; and generating a risk classifier that generates a probability that the entity does not satisfy the target parameter according to an analysis of the plurality of simulated values ​​of the entity parameter.

[0007] According to a fourth aspect, a method of generating a machine learning (ML) model ensemble for computing a probability that an entity does not satisfy a target parameter includes training a plurality of ML sub-models, each outputting a sub-value in response to input of a raw data element; training a main ML model utilizing a training dataset including a plurality of ML sub-values ​​and corresponding entity parameters for each of a plurality of sample entities, for outputting a value of an entity parameter corresponding to the target parameter in response to input of the plurality of sub-values ​​output by the plurality of ML sub-models; inputting a plurality of raw data elements related to the entity to the plurality of ML sub-models to obtain an output of each sub-value; calculating, in a plurality of iterations, a plurality of simulated adjustments to the plurality of obtained sub-values ​​output by the plurality of ML sub-models based on a probabilistic simulation model and a set of prior distributions as provided or calculated from the raw data elements; and calculating a plurality of adjusted sub-values ​​for the plurality of obtained sub-values. applying each of the plurality of simulated adjustments to the plurality of obtained sub-values ​​output by the plurality of ML sub-models to generate a main ML model, inputting the plurality of adjusted sub-values ​​of the plurality of obtained sub-values ​​calculated by applying the plurality of simulated adjustments to the output of the plurality of ML sub-models, obtaining a plurality of simulated values ​​of the entity parameter from the main ML model in the plurality of iterations in response to inputting the plurality of adjusted sub-values ​​for the plurality of obtained sub-values ​​calculated by applying the plurality of simulated adjustments to the output of the plurality of ML sub-models, according to an analysis of the plurality of simulated values ​​of the entity parameter output by the main ML model in the plurality of iterations in response to inputting the plurality of adjusted sub-values ​​of the plurality of obtained sub-values ​​calculated by applying each of the plurality of simulated adjustments to the sub-values ​​output by the plurality of ML sub-models.and generating a risk classifier that generates a probability that the entity does not satisfy the target parameters.

[0008] According to a fifth aspect, a method of dynamically and repeatedly using an ML model ensemble to calculate the probability that an entity does not meet a target parameter includes, in a plurality of first iterations, receiving from the entity a request to obtain a target parameter, receiving a plurality of raw data elements extracted by code sensors installed on a plurality of network nodes and input into a server in real time, inputting the plurality of raw data elements into a plurality of ML submodels, obtaining a plurality of sub-values from the plurality of ML submodels, and in a plurality of second iterations, calculating a plurality of simulated adjustments for the plurality of obtained sub-values output by the plurality of ML submodels based on a probabilistic simulation model and a set of prior distributions when provided or calculated from the raw data elements, applying each of the plurality of simulated adjustments to the plurality of obtained sub-values output by the plurality of ML submodels to generate a plurality of adjusted sub-values of the plurality of obtained sub-values, inputting the plurality of adjusted sub-values of the plurality of obtained sub-values calculated by applying the plurality of simulated adjustments to the output of the plurality of ML submodels into a main ML model, obtaining a plurality of simulated values of entity parameters from the main ML model in the plurality of second iterations in response to the input of the plurality of adjusted sub-values of the plurality of obtained sub-values calculated by applying the plurality of simulated adjustments to the output of the plurality of ML submodels, inputting the plurality of simulated values output by the main ML model in the plurality of iterations into a risk classifier that calculates the probability that the entity does not meet the target parameter according to the analysis of the plurality of simulated values of the entity parameters, and when the probability is below a threshold,automatically providing the entity with the target parameters; and automatically rejecting a request for the target parameters when the probability exceeds the threshold.

[0009] In further implementations of the first, second, third, fourth and fifth aspects, when the plurality of raw data elements are provided, the risk classifier is generated based on a few shot or single shot training approach using the plurality of raw data elements as initial inputs by generating additional synthetic data for generating the risk classifier by calculating and using the plurality of simulated values ​​for the entity parameters output by the main ML model in the plurality of iterations in response to input of the plurality of adjusted sub-values ​​for the plurality of obtained sub-values ​​calculated by applying each of the plurality of simulated adjustments to the plurality of ML sub-models.

[0010] In further implementations of the first, second, third, fourth and fifth aspects, each of a plurality of simulated adjustments is calculated for a respective sub-value of the plurality of obtained sub-values, and each of the plurality of simulated adjustments is applied to each of the plurality of obtained sub-values ​​output by the plurality of ML sub-models to generate a plurality of adjusted sub-values ​​for each sub-value.

[0011] In further implementations of the first, second, third, fourth and fifth aspects, each of a set of adjusted sub-values ​​is calculated by calculating, for each of the plurality of sub-values, each of a set of a plurality of simulated adjustments for each of the plurality of sub-values ​​and applying each of the plurality of simulated adjustments to a corresponding one of the plurality of sub-values.

[0012] In further implementations of the first, second, third, fourth and fifth aspects, the multiple simulated adjustments to each sub-value of the multiple sub-values ​​are selected from the group consisting of increasing each sub-value, decreasing each sub-value and creating a new sub-value.

[0013] In further implementations of the first, second, third, fourth and fifth aspects, the method further comprises identifying at least one most influential raw data element of the plurality of raw data elements having the greatest impact on the entity parameter, and calculating the plurality of simulated adjustments is performed on sub-values ​​output by at least one ML sub-model of the plurality of ML sub-models that receives the at least one most influential raw data element as input.

[0014] In further implementations of the first, second, third, fourth and fifth aspects, the method further includes iteratively obtaining updates for the plurality of raw data elements, iteratively inputting the updates for the plurality of raw data elements into the plurality of ML sub-models to obtain updated sub-values, calculating the plurality of simulated adjustments for each of the plurality of updated sub-values ​​to obtain a plurality of adjusted updated sub-values ​​in the plurality of updated iterations, inputting the plurality of adjusted updated sub-values ​​into the main ML model, obtaining a plurality of simulated updated values ​​for the entity parameter, and generating an update for the risk classifier using an updated analysis of the plurality of simulated updated values ​​to generate an updated probability that the entity does not satisfy the target parameter.

[0015] In further implementations of the first, second, third, fourth and fifth aspects, the main ML model includes a plurality of parameters related to the sub-values ​​output by the plurality of ML sub-models, the parameters being functions of the plurality of parameters related to a plurality of weights, and training the main ML model includes learning the plurality of weights.

[0016] In a further implementation of the first, second, third, fourth and fifth aspects, the multiple weights of the function are calculated based on automated combinatorial deformation scenarios affecting the entity parameters.

[0017] In further implementations of the first, second, third, fourth and fifth aspects, the analysis further includes receiving a threshold value indicating a risk of the entity not satisfying the target parameter, and the analysis includes calculating the probability by calculating an entity score based on the simulated values ​​being above or below the threshold value for the entire set of simulated values.

[0018] In further implementations of the first, second, third, fourth and fifth aspects, the method further includes determining a probability that the entity satisfies the target parameter when the entity score exceeds the target parameter.

[0019] In further implementations of the first, second, third, fourth and fifth aspects, the plurality of raw data elements related to the entity are selected from the group consisting of structured data obtained from a structured data source that stores data for the entity, numerical data obtained from a data source that calculates numerical data for the entity, and unstructured data obtained from open source and / or social networks.

[0020] In a further realisation of the first, second, third, fourth and fifth aspects, at least one of the plurality of ML sub-models (i) outputs each metric based on a correlation window between values ​​of the raw data elements of a particular type obtained during a first time interval and values ​​of the raw data elements of the particular type obtained during at least one second time interval prior to the first time interval; (ii) calculates each metric when provided with values ​​of a set of raw data elements of a defined type; (iii) calculates each metric by a combination of a set of sub-sub-models selected according to the type of the entity; (iv) outputs a probability that the entity satisfies the target parameter when provided with a selected set of raw data elements, the sub-classifier for each of a plurality of sample entities comprising: (v) outputting a risk of failure of a component of the plurality of components of the entity relative to the entity parameters; (vi) outputting an indicator and / or predictive parameter based on analysis of raw data elements indicative of users accessing a website associated with the entity; (vii) outputting an indication of users within the entity based on analysis of users' profiles posted on a social network operated by an external social network server; and (viii) outputting an indicator and / or predictive parameter based on analysis of marketing elements indicative of users interacting with advertisements and / or promotions associated with the entity.

[0021] In further implementations of the first, second, third, fourth and fifth aspects, the multiple simulated adjustments are calculated based on a probabilistic simulation model and a set of prior distributions when calculated or provided from raw data.

[0022] In further implementations of the first, second, third, fourth and fifth aspects, features of the method are realized by at least one hardware processor running on a server, and the raw data elements are extracted by code sensors installed on a number of network nodes and populated in real time into the server.

[0023] In further implementations of the first, second, third, fourth and fifth aspects, the entity includes a company, the target parameters specified in the request include funds specified in the funding request to fund the company, and the entity parameters include financial assets of the entity that will be used to repay funds of the funding request.

[0024] In further implementations of the first, second, third, fourth and fifth aspects, the plurality of raw data elements related to the entity are selected from the group consisting of data elements indicative of customers of the company, data elements indicative of financial details of the company, and data elements obtained from third party sources related to the company.

[0025] In a further implementation of the first, second, third, fourth and fifth aspects, at least one of the plurality of ML sub-models comprises: (i) an indication of growth of the company calculated as a ratio between revenue during a first time interval and revenue obtained during at least one second time interval prior to the first time interval, the revenue calculated based on a plurality of first type raw data elements including invoices and / or payments obtained from a charging and / or billing system integration; (ii) an indication of customer churn calculated based on a plurality of second type raw data elements including invoices, payments and / or data obtained from a customer relationship management (CRM) system; (iii) an indication of gross margin calculated based on a plurality of third type raw data elements; (iv) an indication of unit economics calculated as a collection of combinations of a plurality of sub-sub-models selected according to a business type of the company; and (v) a sub-classifier that outputs a probability of the company raising a fund when provided with a plurality of fourth type raw data elements, wherein for each of a plurality of sample companies, the fourth type data elements are provided with a probability of the company raising a fund. a sub-classifier trained on a training dataset comprising values ​​of elements and an indication of funds raised by the respective company; (vi) when provided with a plurality of data elements of a fifth type, outputting a risk of failure of one of the plurality of revenue generating units of the company to the company based on a calculation of a statistical distribution of the plurality of revenue generating units; (vii) outputting indicators and / or forecast parameters based on an analysis of data elements of a sixth type indicative of users accessing websites related to the company and / or based on an analysis of data elements of a seventh type indicative of users interacting with advertisements provided; (viii) outputting instructions for redeployment of employees within the company based on an analysis of employee profiles posted on a social network operated by an external social network server and / or further external sources operated by network nodes; (ix) real-time short-term and / or long-term cost forecasts based on an analysis of commitments including contracts and / or purchase orders obtained via an enterprise resource planning (ERP) system integration;Output each respective metric and / or prediction parameter selected from the group consisting of: outputting the expected significant costs detected by comparison of external contractors, changes in the prices of services supplied, a bill of materials (BOM), and / or instructions for ongoing purchase orders.

[0026] In further embodiments of the first, second, third, fourth, and fifth aspects, the risk classifier utilizes the plurality of raw data elements as initial inputs to generate additional synthetic data for generating the risk classifier by calculating and utilizing, in the plurality of iterations, the plurality of simulated values for the entity parameters output by the main ML model in response to the input of the plurality of adjusted sub-values for the plurality of obtained sub-values calculated by applying each of the plurality of simulated adjustments to the plurality of ML sub-models. The additional synthetic data is generated based on a few-shot or single-shot training approach.

[0027] In further embodiments of the first, second, third, fourth, and fifth aspects, each of the plurality of simulated adjustments is calculated for each respective sub-value of the plurality of obtained sub-values, and each of the plurality of simulated adjustments is applied to each of the plurality of obtained sub-values output by the plurality of ML sub-models to generate a plurality of adjusted sub-values for each sub-value.

[0028] In further implementations of the first, second, third, fourth and fifth aspects, the method further comprises: iteratively obtaining updates for the plurality of raw data elements; iteratively inputting the updates for the plurality of raw data elements into the plurality of ML sub-models to obtain updated sub-values; in the plurality of updated iterations, calculating the plurality of simulated adjustments for each of the plurality of updated sub-values ​​to obtain a plurality of adjusted updated sub-values; inputting the plurality of adjusted updated sub-values ​​into the main ML model; obtaining a plurality of simulated updated values ​​for the entity parameter; and inputting the plurality of simulated updated values ​​into the risk classifier for generating an updated probability that the entity does not satisfy the target parameter according to an analysis of the plurality of simulated updated values ​​of the entity parameter.

[0029] In further implementations of the first, second, third, fourth and fifth aspects, the method further includes calculating a trend according to the iteratively calculated updated probability that the entity parameters do not satisfy the target parameters, and extrapolating the trend to a future time to predict a trend when the probability that the entity does not satisfy the target parameters exceeds a threshold.

[0030] In further implementations of the first, second, third, fourth and fifth aspects, the method further includes monitoring a change in at least one statistical quantity of the plurality of raw data elements and triggering an iteration in response to the change and / or when the change in the statistical quantity is significant.

[0031] In further implementations of the first, second, third, fourth and fifth aspects, the method further includes analyzing a plurality of weights of the function to identify a plurality of influential weights that most influence the value of the entity parameter, and calculating a set of sub-functions, each sub-function corresponding to a particular influential weight for correlating between at least one raw data element and the value of the entity parameter.

[0032] In further implementations of the first, second, third, fourth and fifth aspects, in response to repeatedly obtaining updates of the plurality of raw data elements, an interactive graphical user interface (GUI) presented on a display of a client terminal is dynamically updated with an indication of an entity score indicating the probability that the entity will not satisfy the target parameters and an amount of pre-approved funding available to the company determined based on a dynamically calculated probability of obtaining sufficient financial assets to repay the pre-approved funding.

[0033] In further implementations of the first, second, third, fourth and fifth aspects, the method further includes receiving a request via the GUI for a funding amount up to the pre-approved funding amount presented in the GUI, and automatically providing the funding amount to the company's account.

[0034] Unless otherwise specified, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the present patent specification, including definitions, will control. Additionally, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting. [Brief description of the drawings]

[0035] Some embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings. Referring now in more detail to the drawings, it is emphasized that the particulars shown are by way of example and are illustrative of embodiments of the invention. In this regard, the description taken with the drawings will make apparent to those skilled in the art how embodiments of the invention may be practiced.

[0036]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

[0037] The present invention in some embodiments thereof relates to machine learning (ML) and, more particularly, but not by way of limitation, to systems and methods for training and utilizing an ensemble of ML models to calculate the risk that an entity fails to satisfy target parameters.

[0038] Some embodiment aspects of the present invention relate to a system, method, apparatus and / or code instructions (stored in a memory and executable by one or more hardware processors) for generating a machine learning model ensemble for calculating a probability that an entity does not satisfy a target parameter (the term calculating a probability may be interchangeable with the term predicting a risk). The ML model ensemble includes the following components: (i) ML sub-models for generating sub-value results in response to input of raw data elements, (ii) a main ML model for generating value results for an entity parameter corresponding to a target parameter in response to input of the sub-values ​​output by the ML sub-models, and (iii) a risk classifier for generating a probability that an entity does not satisfy a target parameter according to an analysis of simulated values ​​of the entity parameters. The simulated values ​​are generated by calculating simulated adjustments to the obtained sub-values ​​to generate adjusted sub-values. The adjusted sub-values ​​are input to the ML model to obtain the simulated values. The ML model ensemble may be customized, i.e., calculated for each entity. The ML model ensemble is dynamically updated and utilized to dynamically calculate the probability that an entity will not satisfy the target parameters according to dynamic adjustments to the raw data elements, where the real-time probabilities are generated according to the real-time status of the raw data elements.

[0039] A plurality of simulated adjustments are calculated for the sub-values ​​output by the ML sub-model, optionally by a probabilistic simulation model and a set of prior distributions, as provided or calculated from the raw data elements, for example according to a Monte Carlo simulation model. The number of simulated adjustments is greater than the number of sub-values, for example by about 2-100, 10-50, 5-25 or other factors. The plurality of simulated adjustments may be calculated in a one-to-many and / or many-to-many approach, i.e., a plurality of simulated adjustments are calculated for each individual sub-value and / or each single sub-value, and / or a plurality of simulated adjustments are calculated for a sub-value. The plurality of simulated adjustments are applied to the plurality of sub-values ​​output by the ML sub-model to generate an adjusted sub-value for each respective sub-value. The adjusted sub-values ​​calculated by applying the simulated adjustments to the output of the ML sub-model are input to the main ML model. In response to the input of the adjusted sub-values ​​(calculated by applying the simulated adjustments to the output of the ML sub-model), a simulated value of the entity parameter is obtained from the main ML model. Using simulated tuning, the number of simulated values ​​of the entity parameter is significantly greater than the number of sub-values, e.g., by about 2-100, 10-50, 5-25, or other factor. The simulated values ​​increase the amount of data available to train the risk classifier, e.g., compared to training the risk classifier using only the available sub-values, i.e., unsimulated values.

[0040] The simulated adjustments of the entity parameters, the adjusted sub-values, and the simulated values, as well as single-shot or few-shot learning, allow for the computation of predictions that the entity will not satisfy the target parameters when no ground truth is available or when little ground truth is available for the entity. The ML model ensemble computes predictions without prior predictions and / or historical data of whether the target parameters were satisfied in the past. Multiple simulated values ​​representing a subsampled distribution "space" of infinite possible entity parameter outcomes may be analyzed to compute an aggregated overall probability that the entity will not satisfy the target parameters. The simulated adjustments to the sub-values ​​to obtain the adjusted sub-values ​​allow for the generation of sufficient data to compute a risk classifier that generates a probability that the entity will not satisfy the target parameters, even when no or little ground truth data is actually available for the entity for satisfying the target parameters.

[0041] The ML model ensemble may be generated using the following exemplary process: A number of ML sub-models are trained, each outputting a sub-value in response to input of a corresponding raw data element. A main ML model is trained, outputting a value of an entity parameter corresponding to a target parameter in response to input of the sub-values ​​output by the ML sub-models. The main ML model may be trained using a training dataset that includes, for each of the sample entities, the ML sub-values ​​(output by the ML sub-models in response to input of the raw data elements associated with each sample entity) and the corresponding entity parameters (i.e., serving as ground truth). The raw data elements associated with the entities are input to the trained ML sub-models to obtain an output for each sub-value. In multiple iterations, simulated adjustments to the obtained sub-values ​​are calculated to generate adjusted sub-values. The adjusted sub-values ​​are input to the main ML model. A set of multiple simulated values ​​for the entity parameters is obtained as a result of the main ML model over the iterations. A risk classifier that generates a probability that the entity does not satisfy the target parameter is generated as a process for analyzing the set of simulated values ​​of the entity parameters. An ML model ensemble is provided that includes an ML sub-model, a main ML model and a risk classifier.

[0042] The ML model ensemble may be used to calculate the probability that an entity does not satisfy a target parameter using the following exemplary process: A request from an entity to obtain the target parameter is received. Optionally, raw data elements are received, extracted by code sensors installed on network nodes and populated in real time to a server. The raw data elements are input to ML sub-models. Sub-values ​​are obtained as results of the ML sub-models.

[0043] Simulated adjustments to the obtained sub-values ​​may be calculated to generate adjusted sub-values, which are input to the main ML model to obtain simulated values ​​for the entity parameters. The process of calculating the simulated adjustments may be repeated to obtain a set of simulated values ​​for the entity parameters. The set of simulated values ​​is input to a risk classifier that calculates the probability that the entity does not satisfy the target parameters. The risk classifier calculates the probability according to an analysis of the set simulated values ​​for the entity parameters. The risk classifier may calculate an entity score indicating the probability that the entity does not satisfy the target parameters. The entity score may be calculated by a function applied to a subset of each simulated value above a threshold and to another subset of each simulated value below a threshold that defines a binary condition of satisfying or not satisfying the target parameters. The process features may be dynamically repeated to obtain dynamic updates to the probability that the entity does not satisfy the target parameters, for example in response to an updated request and / or in response to an update of the values ​​of the raw data elements. The target parameters may be automatically provided to the entity when the probability falls below a threshold. Alternatively, requests for target parameters may be automatically rejected when the probability exceeds a threshold.

[0044] In one example, the entity is a company, e.g., a sole proprietorship, startup, corporation, and / or other business-related organization. The target parameters relate to the monetary funds indicated in the funding request for funding the company. The entity parameters indicate the financial assets of the entity that will be used to repay the funds of the funding request. The ML model ensemble uses the existing and / or predicted financial assets to provide a real-time indication of the company's ability to repay the funds and the associated risk that the company will not be able to repay the given funds. For example, the ML model ensemble provides a real-time continuous underwriting process for providing funds to the company. The ML model ensemble determines the risk that the company will not be able to repay the current and future / additional funds. Optionally, a real-time prediction of the amount of funds the company can repay with an acceptable risk is presented in the GUI. The acceptable risk may be determined by the funding entity, e.g., as a threshold value as described herein. The GUI may be dynamically updated to present a pre-approved amount of funds based on ongoing changes to the prediction of the amount of funds available for the company to withdraw with the acceptable risk output by the ML model ensemble. A firm may request up to an advance amount of real-time funds to be presented in the GUI, which may be automatically deposited into the firm's account.

[0045] At least some embodiments of the systems, methods, apparatus and / or code instructions described herein relate to the technical problem of dynamically and efficiently computing predictions that an entity will not meet a target parameter when no ground truth or little ground truth is available for the entity, similar to the concept of single-shot or few-shot learning. At least some embodiments of the systems, methods, apparatus and / or code instructions described herein relate to the technical problem of dynamically and efficiently computing predictions for a target parameter of an entity in an environment where values ​​of raw data elements are dynamically adapted. At least some embodiments of the systems, methods, apparatus and / or code instructions described herein improve the field of machine learning by computing an ensemble of ML models that predict the probability of reaching a target value by an entity when ground truth is not available. The prediction may be for the probability that a company will accumulate sufficient assets to repay a fund in the future. In such a case, if the company is relatively new and has little or no history of obtaining and repaying a loan, ground truth may not be available. The ML model ensemble calculates predictions without historical data of past expectations and / or whether historical data was satisfied in the past, for example for new companies (e.g., startups) that have not received funding in the past and / or for which little or no historical financial data is available. The simulated adjustments, optionally Bayesian simulated adjustments, to the sub-values ​​output by the ML sub-models allow for the generation of multiple simulated values ​​for the entity parameter to generate adjusted sub-values ​​that are input to the main ML model (which may be customized for each entity). Each simulated value represents one possible prediction for the entity parameter based on the possible prediction scenarios represented by the simulated adjustments.A plurality of simulated values, representing a subsampled distribution "space" of infinitely possible entity parameter outcomes, may be analyzed to calculate an aggregate overall probability that an entity will not satisfy a target parameter. The plurality of simulated values ​​provides sufficient data to calculate a risk classifier, to calculate the probability that an entity will not satisfy a target parameter, and / or to calculate an ensemble of ML models (i.e., to predict the probability that an entity will satisfy a target value), even when no or little ground truth data for actually satisfying the target parameter is available for the entity.

[0046] At least some embodiments of the systems, methods, devices and / or code instructions described herein relate to the technical problem of improving the accuracy of predicting the probability that an entity will not satisfy a target value. At least some embodiments of the systems, methods, devices and / or code instructions described herein improve the field of machine learning by computing an ensemble of ML models customized to an entity. The simulated adjustments to the sub-values ​​to obtain adjusted sub-values ​​(as described herein) allow for generating sufficient data to compute a risk classifier that generates a probability that an entity will not satisfy a target parameter. The simulated adjustments to the sub-values ​​are customized to a particular entity and do not necessarily depend on data of other entities. Data of other entities may be utilized to create a main ML model, where multiple adjusted sub-values ​​are iteratively input to the main ML model to obtain a set of simulated values ​​of the entity parameter of the particular entity. In contrast, using standard approaches such as neural networks and / or other standard classifiers, predictions of values ​​are obtained directly by training on data of other sample entities that did and did not satisfy the target as ground truth in an attempt to use the data of the sample entities to make predictions for the particular entity. Simulated adjustment of the sub-values ​​of a particular entity is not performed in the standard approach. Because other entities are inherently different from the current entity being analyzed, such a standard classifier trained on other entities' data is inherently less accurate than the ML model ensemble described herein. Using only data on whether other entities have met the target provides a less accurate estimate of the probability that the current entity being analyzed will meet the target.The simulated adjustments to the sub-values ​​to generate adjusted sub-values ​​that are input to a main ML model to obtain a set of simulated values ​​for the entity parameter provide a more accurate set of simulated values ​​for the entity parameter when the set is analyzed as a whole. The set of simulated values ​​represents many possible predicted scenarios. Aggregation of the predicted scenarios provides a more accurate calculation of the overall probability that the entity will not satisfy the target parameter. In contrast, standard approaches use standard classifiers that do not consider multiple predicted scenarios but directly output a probability value.

[0047] At least some embodiments of the systems, methods, apparatus, and / or code instructions described herein improve a user's experience using a computer. The improvement is at least obtained by code sensors and / or other code that obtain data elements from network nodes for real-time updates of the probability that an entity will meet a target value predicted by the ML model ensemble using the obtained data elements when the data elements are provided to an ML model ensemble executing on a server. For example, the amount of additional funding that a company may obtain may be calculated in real-time based on updates of the data elements from the network node sources. The additional funding may be calculated by the ML model ensemble based on the probability that the company will have sufficient future assets to repay the additional funding. The real-time updates may be presented in a graphical user interface, e.g., a dashboard. The code sensors enable real-time provision of updated data elements to the ML model ensemble for real-time updates of the user interface. For example, the code sensors may stream dynamically collected data elements to the ML model ensemble for streaming updates of the user interface. The real-time updates reflect the substantially real-time state of the company based on real-time data acquired by the code sensors, e.g., real-time access by users to the company's website, real-time growth in revenue, etc. A prediction of future changes in financial repayment capacity may be calculated based on trends in past real-time predictions, allowing for future financial forecasting. For example, when a company experiences a monthly increase in repayment capacity that translates into a monthly increase in potential funds, a prediction can be made as to when the company is predicted to be likely to make a very large repayment. A user may choose to wait until a very large amount is available based on the prediction. A user may request and be approved for additional funds in real-time, without having to wait until sufficient historical results are available.

[0048] At least some embodiments of the systems, methods, apparatus and / or code instructions described herein relate to the technical problem of obtaining sufficient data to underwrite a company, and optionally how to obtain and use real-time data to provide ongoing underwriting, particularly real-time underwriting for the company, and / or how to monitor changes in the company's risk that may affect underwriting. Also, standard approaches are based on utilizing historical data for the company, which is not available for new companies and cannot accurately predict future scenarios that may affect the company. Other standard approaches based on using data for other companies are inaccurate because such data is not necessarily applicable to the company being evaluated. At least some embodiments of the systems, methods, apparatus and / or code instructions described herein provide a solution to the technical problem by using raw data elements related to the company provided to an ML sub-model to obtain a sub-value and calculating simulated adjustments to the sub-value to obtain an adjusted sub-value. The adjustments based on simulated distributions represent, for example, multiple different plausible forecast scenarios rather than attempting a single prediction. The adjusted sub-values ​​may be input into a main ML model trained using other companies' data or collaborative expert knowledge to provide predictions based on other companies' data and / or many subject matter expert analytical techniques (reinforcement learning). The multiple simulated values ​​obtained from the main ML model represent many different predictions for the company representing a plausible (local) subsampled space of the company's infinite scenario space. The generated risk classifier analyzes the multiple simulated values ​​to arrive at an overall most likely scenario representing an overall (e.g., single) predicted probability that the entity will not satisfy the target parameters. The ML model ensemble described herein provides increased accuracy of the probability that the entity will not satisfy the target parameters and / or increased accuracy of real-time monitoring of the probability that the entity will not satisfy the target parameters, for example, compared to standard underwriting approaches.

[0049] Before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited to the details of construction and the arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings and / or illustrative examples in this application, as the invention is capable of other embodiments or of being practiced or carried out in various ways.

[0050] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0051] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor memory, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, and any suitable combinations of the above. A computer-readable storage medium as used herein should not be interpreted as being a transitory signal in itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a wave guide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electric signal transmitted over a wire.

[0052] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within each respective computing / processing device.

[0053] The computer readable program instructions for carrying out the operations of the present invention may be either source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or conventional procedural programming languages, such as object oriented programming languages ​​such as Smalltalk, C++, and the like, and the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0054] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0055] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram blocks. These computer readable program instructions may be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, and a computer readable storage medium having instructions stored thereon includes an article of manufacture containing instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0056] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a sequence of processing steps to be executed on the computer, other programmable data processing apparatus, or other device, to generate a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram blocks.

[0057] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions described in the blocks may be executed out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be realized by a dedicated hardware-based system that executes the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0058] Reference is now made to Fig. 1A, which is a flowchart of a method for generating an ensemble of ML models for computing a probability that an entity does not satisfy a target parameter according to some embodiments of the present invention. Reference is also made to Fig. 1B, which is a flowchart of a method for using an ensemble of ML models for computing a probability that an entity does not satisfy a target parameter according to some embodiments of the present invention. Reference is also made to Fig. 2, which is a block diagram of a system 200 for generating and / or using an ensemble of ML models for computing a probability that an entity does not satisfy a target parameter according to some embodiments of the present invention. The system 200 may implement the operations of the methods described with reference to Fig. 1A and / or Fig. 1B by a processor 202 of a computing device 204 executing code instructions 206A stored in a storage device 206 (also referred to as a memory and / or program store).

[0059] Multiple architectures of the system 200 based on the computing device 204 may be realized. In an exemplary embodiment, the computing device 204 storing the code 206A may be implemented as one or more servers (e.g., network server, web server, computing cloud, virtual server) that provide services (e.g., one or more of the operations described with reference to FIG. 1A and / or FIG. 1B ) to one or more client terminals 212 via the network 214, such as, for example, providing software as a service (SaaS) to the client terminals 212, providing software services accessible using a software interface (e.g., application programming interface (API), software development kit (SDK)), providing applications for local download to the client terminals 212, and / or providing functionality using a remote access session to the client terminals 212 via a web browser or the like. For example, multiple users subscribe to a service centrally provided by the computing device 204 using their respective client terminals 212. Alerts and / or updates are provided to the respective client terminals 212 by the computing device 204. In other embodiments, computing device 204 may include locally stored software (e.g., code 206A) that performs one or more of the operations described with reference to FIG. 1A and / or FIG. 1B, for example as a self-contained client terminal designed to be used by a user of the client terminal.

[0060] In another embodiment, each client terminal 212 may obtain a respective ML model ensemble 216A, which may be customized, for local installation and use from the computing device 204 (which may compute and / or update the ML model ensemble 216A as described herein). Each client terminal 212 may store its own custom computed trained ML models 216A ensemble for local use.

[0061] Each ML model ensemble 216A may include one or more of an ML sub-model 216A-1, a main ML model 216A-2, and a risk classifier 216A-3 as described herein.

[0062] Each ML model ensemble 216A or its components may be custom created for each entity. In an exemplary embodiment, the ML sub-model 216A-1 of the ensemble 216A is not customized in the sense that the ML sub-model 216A-1 is created using data related to multiple different entities and is used for the different entities. The main ML model 216A-2 and the risk classifier 216A-3 may be customized for each entity. In such an embodiment, multiple customized ML model ensembles 216A may share a common ML sub-model 216A-1 and include a customized main ML model 216A-2 and a customized risk classifier 216A-3. The server 210 (also referred to herein as a network node) may be associated with a code sensor 210A (e.g., installed on a data storage device of the server 210 and executed by a hardware processor of the server 210) that extracts raw data elements. The raw data elements are transmitted to the computing device 204 over the network 214, for example, via an API and / or SDK, as described herein.

[0063] The processor 202 of the computing device 204 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), and an application specific integrated circuit (ASIC). The processor 202 may include a single processor or multiple processors (homogeneous or heterogeneous) configured for parallel processing, as a cluster and / or one or more multi-core processing devices.

[0064] The data storage device 206 stores code instructions executable by the processor 202 and may be, for example, a random access memory (RAM), a read only memory (ROM) and / or a storage device, for example, a non-volatile memory, a magnetic medium, a semiconductor storage device, a hard drive, a removable storage and an optical medium (e.g., DVD, CD-ROM), etc. The storage device 206 stores code 206A that, when executed by the processor 202, implements one or more functions and / or operations of the methods described with reference to FIGS. 1A-1B, such as, for example, calculating correlations, calculating simulated adjustments, calculating simulated values, training a customized ML model, presenting a GUI and / or generating instructions for calculating a GUI, and / or other functions described with reference to FIGS.

[0065] The computing device 204 may include a data repository 216 for storing data, such as one or more of an ML model ensemble 216A computed and / or updated as described herein, a training dataset 216C storing data for training the ML model ensemble 216A and / or for training associated ML sub-models 216A-2 (e.g., received raw data elements), and / or GUI code 216D for executing a GUI described herein (e.g., locally, remotely, and / or for download to each client terminal 212). The data repository 216 may be embodied as, for example, a memory, a local hard drive, virtual storage, a removable storage unit, an optical disk, a storage device, a remote server, and / or a computing cloud (e.g., accessed using a network connection).

[0066] The network 214 may be implemented as, for example, the Internet, a local area network, a virtual private network, a wireless network, a cellular network, a local bus, a point-to-point link (e.g., wired), and / or a combination of the above.

[0067] The computing device 204 may include a network interface 218 for connecting to the network 214, such as one or more of a network interface card, a wireless interface for connecting to a wireless network, a physical interface for connecting to a cable for a network connection, a virtual interface implemented in software, network communications software providing an upper layer of the network connection, and / or other embodiments.

[0068] The computing device 204 includes: A server 210 from which raw data elements (structured and / or unstructured data) are obtained A client terminal 212 that may be utilized by a user to remotely access a computing device 204 as described herein. using a network 214 (or other communication channel), such as via a direct link (e.g., cable, wireless) and / or a non-direct link (e.g., via an intermediary computing unit such as a server and / or storage device).

[0069] The computing device 204 and / or client terminal 212 include and / or communicate with one or more physical user interfaces 208 that optionally include mechanisms for a user to input data and / or view data (e.g., generated alerts, automated actions, and / or manually required approvals) within a GUI. Exemplary user interfaces 208 include, for example, one or more of a touch screen, a display, a keyboard, a mouse, and voice-activated software using a speaker and microphone.

[0070] Reference is made to Fig. 1A, which is a flowchart of a method for generating an ensemble of machine learning models for calculating the probability that an entity does not satisfy a target parameter according to some embodiments of the present invention, and to Fig. 1B, which is a flowchart of a method for using machine learning models for calculating the probability that an entity does not satisfy a target parameter according to some embodiments of the present invention.

[0071] In at least some embodiments, the ML model ensembles described herein are dynamically updated and dynamically used to provide a combined training and inference phase. Note that the generation of an ML model ensemble similar to FIG. 1A may be accomplished simultaneously, sequentially, in parallel, subsequent, and / or in combination with the use of an ML model ensemble similar to FIG. 1B. For example, some training features may be integrated (e.g., simultaneously, prior, subsequent, parallel) with inference features to provide a method for dynamic updating and use of an ML model ensemble.

[0072] 1A, a number of ML sub-models are trained and / or provided at 102. Each ML sub-model outputs a sub-value in response to the input of a respective raw data element.

[0073] The ML sub-models may be implemented, for example, as code that, when executed, calculates each of one or more sub-values ​​from the raw data elements. Exemplary embodiments of the ML sub-models include one or more or a combination of rule sets, functions, classifiers, neural networks of various architectures (e.g., artificial, deep, convolutional, fully connected), Markov chains, support vector machines (SVMs), regression (e.g., linear, ridge, rasons, isotonic, etc.), logistic regression, k-nearest neighbors, singular spectrum analysis (SSA), field aware decomposition machines (FFM), and decision trees.

[0074] Exemplary ML sub-models include one or more of the following: (i) ML sub-models that output respective metrics based on a correlation window between values ​​of raw data elements of a particular type obtained during a first time interval and values ​​of raw data elements of a particular type obtained during at least one second time interval prior to the first time interval, e.g., a value indicating how closely the values ​​within the window are correlated (e.g., 0 indicates no correlation and 1 indicates perfect correlation) and / or a value indicating the relationship of values ​​between windows, e.g., a percentage increase or decrease in values ​​between windows (e.g., a 30% increase in the current window over the previous window).

[0075] In an example where the entity is a company, the ML sub-model outputs an indication of the growth of the company calculated as a ratio between revenue during a first time interval and revenue obtained during at least one second time interval prior to the first time interval. The revenue may be calculated based on a plurality of raw data elements of a first type, such as revenue streams, such as invoices and / or payments (e.g., bank transactions, verified in an accounting system), which may be obtained from (e.g., via integration with) a billing and / or accounting system.

[0076] (ii) An ML sub-model that calculates a respective metric when provided with values ​​for a set of raw data elements of a defined type, e.g., using functions and / or other mathematical relationships.

[0077] In examples where the entities are businesses, the ML sub-model outputs an indication of customer churn calculated based on input of raw data elements of a second type, e.g., invoices, payments, and / or data from a customer relationship management (CRM) system. For example, the ML sub-model may be realized as a customer behavior model calculated by processing invoice data to group new customer interactions (e.g., registrations, purchases, etc.) into time periods (i.e., windows) for cohorts (e.g., sets of users). For each cohort, the ML sub-model is calculated to reflect the number of customers and revenue that returned from that cohort, and how revenue changed over time in subsequent windows. Changes in the time windows indicate churn / canceled customers for the number of returning customers per cohort, average customer lifespan values, and / or average and / or marginal lifespan values ​​for revenue.

[0078] In another example where the entity is a company, the ML sub-model outputs an indication of gross margin calculated based on the third type of raw data element. For example, the ML sub-model may be realized as a business-oriented model calculated from accounting and / or banking data segmented into revenue and / or expense streams (e.g., COGS, OPEX, etc., organic product sales revenue). The ML sub-model may indicate the change in gross margin over a period of time (i.e., correlation window). Gross margin may be calculated as the sum of sales revenues minus all relevant cost of goods sold for the window divided by sales revenues. Using the sub-model for revenue forecasting and the sub-model for cost of goods sold / cost forecasting results in a model for forecasting expected future gross margins over time.

[0079] (iii) ML sub-models that calculate their respective metrics by aggregation of a combination of outputs of multiple sub-sub-models selected according to the type of entity. A common set of sub-sub-models may be defined, from which a subset is selected according to the type of entity. Different subsets may be selected from different types of entities. The aggregation may for example be an average of the values ​​output by the members of the subset and / or a function that outputs a value given the outputs of the members of the subset.

[0080] In an example where the entity is a company, a subset of sub-sub-models may be selected according to company type, such as, for example, company structure (e.g., private owner, corporate, non-profile) and / or company industry (e.g., restaurant, high tech, services), etc. The ML sub-models may output an indication of unit economics calculated as an aggregate of a combination of outputs of the sub-sub-models selected according to the company industry.

[0081] For example, the ML sub-models may be realized as business-oriented models that measure the basic business offering. Unit economics vary between industries (e.g., product-oriented businesses vs. asset-oriented businesses, i.e., selling products to consumers vs. renting to consumers). The unit economics ML sub-models may be calculated for a selected business class (e.g., SaaS, E-commerce, Real Estate, etc.). The ML sub-models include multiple sub-sub-models, e.g., ARPA, ARPU, CAC over time, etc., to calculate an ML sub-model that represents the basic revenue generating units of the company. By evaluating the basic units and interactions between the sub-sub-models included in the generation of the unit economics ML sub-models, as described herein, it is possible to evaluate the core unit economics of the company and find boundaries where the business model unit economics cannot support the business goals (i.e., growth), and / or a large set of constrained selected random scenarios (Monte Carlo algorithms) that may cause the company to default before paying off a loan or debt (as described herein) may be simulated.

[0082] (iv) an ML sub-model implemented as a sub-classifier that, given a selected set of raw data elements, outputs a probability that an entity satisfies a target parameter. The sub-classifier may be trained on a training data set that includes, for each of a plurality of sample entities, sample data elements associated with each sample entity and an indication of satisfaction of the respective target parameter provided for each sample entity (e.g., whether the target parameter is satisfied or not).

[0083] In examples where the entities are companies, the sub-classifier may output a probability (e.g., a percentage or a binary value indicating likely or unlikely) indicating the likelihood that the company will raise funds when the sub-classifier is provided with an input of the fourth type of raw data element. The sub-classifier may be trained on a training dataset that includes, for each of a plurality of sample companies, values ​​of the fourth type of data element and an indication of the funds raised by each company. The sample companies may be companies that are similar to the company entity (e.g., of similar size, stage of development, industry, region) or may include companies that are not necessarily similar to the company entity.

[0084] For example, the ML sub-model may be implemented as an external type model. The fundability ML sub-model is evaluating the company's ability to raise funds. The ML sub-model is generated using training data by training the ML sub-model with inputs of detailed information about many other companies (e.g., tens of thousands or other values), including, for example, funding rounds, funding financial institutions, who defaulted, IPOs, purchases of the company, financing to debt ratio valuations, and other parameters. The ML sub-model takes company parameters as inputs and outputs the probability that the company will raise funds. The ML sub-model results may be updated when new pieces of relevant information about the company (e.g., raw data elements extracted by the code sensor) are obtained.

[0085] (v) An ML sub-model that outputs the risk of failure of one of the entity's multiple components with respect to the entity parameters.

[0086] In an example where the entity is a company, the ML sub-model outputs a risk of failure of one of a plurality of revenue generating units of the company or the entire company, the output being generated by the ML sub-model calculating a statistical distribution of the revenue generating units when provided with a plurality of fifth type data elements.

[0087] For example, the ML submodel is based on a Gini index implemented as a business-oriented ML submodel that measures the statistical distribution of the company's revenue-generating units (e.g., customers, assets in a particular location, or different types of assets). The ML submodel calculates the risk associated with the failure of one revenue-generating unit across the company by outputting the value of the index. The higher the value of the index output by the ML submodel, the lower the probability that the loss of a customer or asset will have a significant impact on revenue.

[0088] (vi) An ML submodel that outputs measured and / or predicted parameters based on the analysis of raw data elements indicating users accessing a website related to the entity.

[0089] In an example where the entity is a company, the ML submodel outputs measured and / or predicted parameters based on the analysis of the input of a sixth type of data element indicating users accessing a website related to the company (e.g., site analytics). For example, a website that offers online purchases of products provided by the company and / or a website that describes the products and / or services provided by the company.

[0090] In another example where the entity is a company, the ML submodel outputs measured and / or predicted parameters based on the analysis of a seventh type of data element indicating users interacting with presented advertisements (e.g., data analytics).

[0091] For example, the ML submodel may calculate the website analytics of the company's website. The ML submodel may be implemented as a customer behavior model. The following is an example of a set of measurements and / or predictions of customer behavior based on site analytics.

[0092] The Marketing CAC-ML sub-model is calculated to predict the impact of the marketing budget on new customer traffic to the company site. The CAC is calculated by dividing the online advertising marketing budget by the number of new unique customers at the website normalized by the sum of the segmented target audience of each marketing channel (e.g., ads on search engine results, ads on social networks, posts on social network pages). Using the above-mentioned points taken sequentially over the time interval, future CAC can be predicted by the ML sub-model in response to inputs of marketing budget (company P&L), CAC and / or audience fatigue.

[0093] Revenue Stability - An ML sub-model of revenue stability over time is calculated by analyzing the number of new unique daily users arriving at the company's website adjusted by costs associated with customer acquisition, such as bounce rate, return visits, time on site, page views, conversion rate, lifetime value, geography, device type, etc., and / or by tracking the website activity of new unique daily users. The ML sub-model may calculate the impact of CAC on revenue and detect the state of the company with respect to its ability to grow. Conceptually, the output of the ML sub-model provides an indication of whether the company has room to grow within its current market, what the CAC level is, and / or an understanding of whether the company has already exhausted its growth.

[0094] (vii) An ML sub-model that outputs user indications within the entity based on an analysis of the user's profile posted on a social network maintained by an external social network server.

[0095] (viii) The ML sub-model outputs measurement and / or predictive parameters based on an analysis of the marketing factor indicative of a user interacting with an advertisement and / or promotion associated with the entity.

[0096] In an example where the entity is a company, the ML sub-model outputs instructions for employee reassignment within the company based on an analysis of employee profiles posted on a social network operated by an external social network server. For example, the ML sub-model may calculate a job replacement rate for employees of the company. The ML sub-model may be implemented as an external type model. By finding and tracking the company's employee profiles (e.g., posted on a social network) and / or by monitoring the number of employees changing jobs and / or the average time it takes employees to find a new job, internal company changes may be detected that may indicate something bad is happening to the company. For example, the company is reducing its employee pool to cut costs and / or employees are leaving because they sense that the company is deteriorating. Both are negative signs, especially if the people who leave the company are talented people who will quickly find new jobs.

[0097] (ix) Another example relates to an ERP system from which sensors are collected, such as one or more of purchase orders, long term contracts and / or commitments, changes in supplies, etc. The extracted data is utilized to compute ML sub-models that output one or more indications of real-time short term and / or long term cost forecasts, detected and predicted significant cost changes due to external contractor contracts, changes in prices of services provided, bill of materials (BOM) and standing purchase orders.

[0098] A main ML model is provided and / or trained at 104. The main ML model outputs values ​​of entity parameters corresponding to target parameters in response to input of the sub-values ​​output by the ML sub-models.

[0099] The main ML model may be trained (e.g., using a supervised and / or unsupervised approach) using a training dataset that includes sub-values ​​and corresponding entity parameters (e.g., the entity parameters serve as ground truth labels for each sub-value) for a number of sample entities. The training dataset may be created by obtaining raw data elements for each sample entity (e.g., as described with reference to 106) and inputting the raw data elements for each sample entity into the ML sub-model (e.g., as described with reference to 108) to obtain corresponding sub-values. The set of sub-values ​​is labeled by ground truth labels of the entity parameters corresponding to each entity, e.g., obtained from available sources. For example, in the case of a company seeking financing, the entity parameters may be the company's financial assets indicating the company's ability to repay a loan and / or the company's ability to repay a gold loan, and may be obtained from sources such as annual reports and / or public financing events by the company.

[0100] The main ML model may be implemented as and / or include functionality of a number of parameters related to the sub-values ​​output by the ML sub-models, which may be implemented as and / or include one or more of a set of rules, neural networks of various architectures (e.g., artificial, deep, convolutional, fully connected), Markov chains, support vector machines (SVMs), logistic regression, k-nearest neighbors, decision trees, field-aware factorization machines (FFMs), singular spectrum analysis (SSA), and combinations of the above.

[0101] A parameter may be associated with each weight. The main ML model may be trained by learning the weights. The weights of the function may be calculated based on automated combinatorial distribution scenarios that affect the entity parameters. For example, the function may be a regression function where the weights are learned. In another example, the function may be a neural network where the weights of the neurons are learned.

[0102] The number of weights can be quite large, for example, when most or all of the relevant combinations between the raw data elements and the sub-values ​​are evaluated. The weights can be learned, for example, by evaluating causal relationships, internal interactions, mutual influences, internal influences, and / or combinations of the above.

[0103] At 106, raw data elements related to the entities are received from data sources of a plurality of network nodes. Optionally, the raw data elements are extracted for each entity. Example data elements include one or more of structured data obtained from a structured data source that stores data for the entities, numerical data obtained from a data source that calculates numerical data for the entities, and unstructured data obtained from open sources and / or social networks.

[0104] Optionally, the raw data elements are extracted by a code sensor, for example, installed on a network node, which searches for relevant data across the network, such as a crawling program that uses web links to crawl the network. The raw data elements may be received via an API and / or an SDK. The code sensor may be designed to monitor a data source for new raw data elements related to the entity and extract the raw data elements. The extraction of data elements may be customized for each entity, and the code sensor is designed and / or selected for each entity. The code sensor may be designed according to the type of data source, for example, to extract structured raw data elements from a structured data source (e.g., monitoring values ​​in a field for changes to the value), to extract unstructured raw data elements from an unstructured data source (e.g., monitoring a social network for new posts, analyzing the new posts to identify raw data elements, and extracting raw data elements from the new posts), and / or to extract numeric data elements from a data source that calculates numeric data.

[0105] The raw data elements may be extracted in real time (i.e., near real time) and / or may be populated to a server in real time (i.e., near real time). The terms real time and / or near real time may refer to a short period of time that may include delays in the transmission of data, e.g., network delays due to network congestion. Examples of near real time include, for example, less than 1, 10, 30, 60, 120, 180 seconds, or less than 1, 6, 12, 24 hours, or other values.

[0106] The raw data elements may be populated into the server via, for example, an API and / or SDK.

[0107] Note that the raw data elements may contain anomalies and / or some raw data elements may be missing (completely and / or occasionally). Such raw data elements, as used herein, represent an increased risk that the entity will not meet the target parameters.

[0108] For example, an entity includes a company. The term company may refer to one or more of a registered business, an individual operating a business, a legal entity, a non-profit organization, and a government-related organization. For a company, the following raw data elements related to the company may be extracted: data elements indicative of the company's customers, data elements indicative of the company's financial details, and data elements obtained from third-party sources related to the company. Exemplary data sources from which the raw data elements are extracted include web servers hosting the company's website or web application, online articles hosted by news sites discussing the company, social networks, and / or other servers where the company's employees and / or customers post content such as reviews and / or ratings. Other exemplary data sources include bank accounts, billing and billing systems, web analytics reports, human resource (HR) management systems, customer relationship management (CRM) and enterprise resource planning (ERP) systems.

[0109] In an example where the entity is a company, the raw data elements may be categorized into three types of data: customer behavior data (e.g., extracted from site analytics, CRM systems, churn analysis, etc.), business oriented data (e.g., extracted from accounting and / or billing systems, banking transactions, ERP systems, etc.), and open source (e.g., extracted from third party sources related to the company, such as specialized social networks, websites that rate companies, and websites that describe company finances).

[0110] In examples where the entity is a company, the target parameters described herein may be specified in the request and may include monetary funds indicated in the funding request to fund the company. The entity parameters described herein may include financial assets of the entity that will be utilized to repay the funds of the loan request.

[0111] At 108, the raw data elements associated with the entities are input to the ML sub-models to obtain outputs for each sub-value, e.g., metrics and / or prediction parameters. The ML sub-models may be implemented by a hardware processor in a server that receives the raw data elements.

[0112] Optionally, different raw data elements are directed to one or more ML submodels according to the input specifications of each ML submodel. Each raw data element may be provided to one or more ML submodels. The raw data elements may be provided to a corresponding ML submodel based on, for example, the type of the raw data element (e.g., structured, unstructured, numeric), a tag (e.g., metadata) associated with the raw data element that defines the source of the raw data element and / or the destination of the raw data element. In another example, virtual channels and / or tunnels may be set up to stream the raw elements from their source to the corresponding ML submodel.

[0113] At 110, adjusted sub-values ​​are generated by applying the calculated simulated adjustments to the obtained sub-values. Multiple simulated adjustments and / or adjusted sub-values ​​may be calculated for each obtained sub-value. Optionally, multiple simulated adjustments and / or adjusted sub-values ​​are calculated for individual (e.g., each individual) obtained sub-value. The sub-values ​​may be utilized to generate multiple simulated adjustments and / or adjusted sub-values ​​in a one-to-many (i.e., individual sub-values ​​generate multiple simulated adjustments and / or multiple adjusted sub-values) or many-to-many (i.e., multiple sub-values ​​generate multiple simulated adjustments and / or multiple adjusted sub-values) approach.

[0114] The number of simulated adjustments and / or adjusted sub-values ​​may be significantly greater than the number of sub-values, for example by a factor of 2 or more, 2 to 100, 5 to 50, 25 to 75, 10 to 50, 10 or more, 50 or more, 100 or more, or other factor.

[0115] The multiple simulated adjustments and / or adjusted sub-values ​​allow for a significant increase in the amount of data used to train a risk classifier (as described herein) that is much greater than would be available if simulated adjustments were not made. This allows for training a risk classifier using a zero-shot or few-shot approach when no or little data is available.

[0116] Optionally, the simulated adjustments are calculated based on a probabilistic simulation model (eg, a Monte Carlo simulation) and / or a set of prior distributions provided and / or calculated from the raw data.

[0117] Optionally, feature 172 of FIG. 1B is implemented before, after and / or in parallel with feature 110. Feature 172 of FIG. 1B may be implemented to identify the most influential raw data elements having the greatest impact on the entity parameters. The simulated adjustments may be made to sub-values ​​output by ML sub-models that receive input of the identified most influential raw data elements. Alternatively or additionally, the adjustments are made to all of the sub-values ​​or a subset of the sub-values.

[0118] Referring again now to 110 of FIG. 1A, the distribution used to calculate the simulated adjustment may be, for example, a possible range of values ​​of the raw data elements and / or sub-values, past changes in the values ​​of the raw data elements and / or sub-values ​​(e.g., historical analysis of different values ​​and / or changes in the values ​​of the raw data elements and / or sub-values), a prediction of possible future values ​​of the raw data elements and / or sub-values ​​(e.g., performed manually by an expert and / or predicted by a trained machine learning model), and / or fitting the raw data to a common distribution (e.g., normal, beta, uniform, exponential, chi-squared, etc.). The adjustment may be made by randomly selecting the raw data elements and / or sub-values ​​from a statistical distribution of possible values ​​for the raw data elements and / or sub-values, e.g., a normal distribution, a geometric distribution, a bimodal distribution, and / or other statistical distribution. The distribution may be based on estimated values ​​and / or historical values. Alternatively or additionally, the adjustment for the raw data elements and / or sub-values ​​is selected from the statistical distribution. The selected adjustment is applied to each raw data element and / or sub-value.

[0119] Examples of adjustments include an increase in raw data elements and / or sub-values, a decrease in raw data elements and / or sub-values, a change in type of raw data elements and / or sub-values, and the appearance of new types of raw data elements and / or sub-values.

[0120] At 112, the adjusted sub-values ​​are input into the main ML model.

[0121] At 114, a result of the main ML model is obtained. The result of the main ML model is a simulated value of the entity parameter based on the set of tuned sub-values.

[0122] Using simulated adjustments, a much larger number of simulated values ​​is obtained compared to using only the sub-values ​​without simulated adjustments. The number of simulated values ​​may be significantly greater than the number of sub-values, for example by a factor of 2 or more, 2-100, 5-50, 25-75, 10-50, 10 or more, 50 or more, 100 or more, or other factor.

[0123] Optionally, multiple simulated values ​​are calculated for each individual (e.g., each individual) sub-value. The sub-values ​​may be used to generate multiple simulated values ​​in a one-to-many (i.e., each sub-value generates multiple simulated values) or many-to-many (i.e., multiple sub-values ​​generate multiple simulated values) approach.

[0124] The multiple simulated values ​​allow for a significant increase in the amount of data used to train a risk classifier (as described herein) that is much greater than would be available if no simulated adjustments were made, making it possible to train a risk classifier using a zero-shot or few-shot approach when no or little data is available.

[0125] The process of simulating adjustments to the sub-values ​​and obtaining corresponding simulated values ​​for the entity parameters may be referred to as stress testing.

[0126] At 116, the features described with reference to 110-114 are repeated to generate multiple simulated values ​​for the entity parameter. During each iteration, another set of adjusted sub-values ​​is generated by applying another set of simulated adjustments to the sub-values. Each set of adjusted sub-values ​​is input to the main ML model to obtain a respective simulated value of the entity parameter. The multiple iterations generate a set of simulated values ​​of the entity parameter. Conceptually, the multiple simulated values ​​of the entity parameter represent a set of possible predicted values ​​of the entity parameter for a possible entity.

[0127] At 118, a risk classifier is generated according to an analysis of a set of simulated values ​​calculated for the entity parameters and / or as a mathematical formula and / or ML classifier. The risk classifier generates a probability that the entity will not meet the target parameters. For example, when the entity is a company, there is a possibility that the company will default on a full loan obligation (i.e., will not be able to repay the full loan amount). When the financial assets of the company are predicted to be less than the full loan amount, a probability that the company will not meet the full loan amount is determined.

[0128] The risk classifier is generated based on a few-shot or single-shot training approach that utilizes raw data elements as initial inputs by generating additional synthetic data for generating the risk classifier by calculating and utilizing simulated values ​​of entity parameters obtained by a main ML model in an iteration, where the simulated values ​​are obtained from the main ML model in response to input of adjusted sub-values ​​for the obtained sub-values ​​calculated by applying respective simulated adjustments to the ML sub-models when the raw data elements are provided.

[0129] Optionally, a threshold indicating a risk that the entity does not satisfy the target parameter is received, e.g., manually entered by a user, stored in memory as a pre-set value, and / or calculated automatically by the code (e.g., as an optimization). The analysis may refer to calculating the probability by using a risk classifier that receives a set of simulated values ​​calculated for the entity parameters and outputs an entity score. In one example, the risk classifier is realized as an over-under percentage classifier that evaluates simulated values ​​that are above (or below) a set threshold against the entire set of simulated values ​​to calculate the entity score, e.g., as described with reference to FIG. 4. Optionally, the probability that the entity satisfies the target parameter is determined when the entity score exceeds the target parameter.

[0130] The risk classifier may be implemented, for example, by a set of rules, neural networks of various architectures (e.g., artificial, deep, convolutional, fully connected), Markov chains, support vector machines (SVMs), logistic regression, k-nearest neighbors, decision trees, and combinations of the above.

[0131] At 120, one or more of 102-118, e.g., the ML features described with reference to 106-118, may be repeated. When new raw data is available, the iterations may be performed to generate updated ML sub-models, an updated main ML model, and / or an updated risk classifier. The iterations may be triggered, for example, when new raw data becomes available, by an event (e.g., daily, weekly, monthly), based on a set of rules (e.g., when new raw data is statistically significantly different from previous raw data), when a change in one or more monitored raw data elements, e.g., a change in its value, a new data element, and / or removal of a previous data element, is detected. For example, a data source may be scanned (e.g., in near real-time) by a code sensor to detect changes to the raw data elements. When a change is detected, an iteration may be triggered.

[0132] At 122, an ML model ensemble is provided, e.g., stored in a data storage device, transferred to another server, and / or provided for use by another process. The ML model ensemble includes ML sub-models, a main ML model, and a risk classifier. The ML model ensemble may be customized in the sense that the ML model ensemble is created for a particular entity (e.g., a company) to be used for the particular entity because it is based on a set of ML sub-models for the particular entity. The customized ML model ensemble may not be related to other entities for which the respective customized ML model ensemble may be created.

[0133] The ML model ensemble may be utilized to output predicted values ​​of entity parameters in response to being provided with input raw data elements, for example, as described with reference to FIG. 1B.

[0134] The ML model ensemble may be stored centrally and provided by a server, for example, for centrally computing entity parameters when provided with the raw data elements. In other embodiments, customized ML models may be provided to client terminals (e.g., smart phones, desktop computers) for local storage and / or execution, for local computation of entity parameters when provided with the raw data elements.

[0135] 1B, at 152, a request may be received. The request may be received from an entity. The request is to obtain target parameters. The request may be received via a user interface, such as, for example, a graphical user interface (GUI). The request may be entered manually and / or automatically.

[0136] In the examples described herein, the entity may be a business, such as a sole proprietor, a startup, a corporation, and / or other business-related organization. In such cases, the request may optionally be for a funding amount up to a pre-approved funding amount (e.g., presented in a GUI). The pre-approved funding amount may be dynamically calculated and updated as described herein.

[0137] A company may set up a user account for requesting funds (e.g., a loan) and / or be recognized by a funding service for requesting funds. The target parameters relate to monetary funds indicated in a funding request for funding of the company. The funding request may be submitted, for example, via a user interface of the user account and / or to the funding service. The entity parameters indicate, for example, the company's ability to repay funds based on current financial assets and / or projected future financial assets.

[0138] The features described with reference to Fig. 1A may be triggered in response to receiving a request. Alternatively, the initial features described with reference to Fig. 1A may be triggered in response to the entity itself prior to the request. For example, the entity sets up a user account and / or the entity is recognized. Optionally, the ML model ensemble is computed in response to receiving a request. The request for target parameters may be received, for example, prior to and / or in parallel with the implementation of the features described with reference to Fig. 1A. The request may be received, for example, via the entity's user account.

[0139] Optionally, in an initial stage, one or more features 154-168 are implemented as an initial assessment of the entity parameters. A requested target parameter may be provided as described herein, or values ​​corresponding to the entity parameters may be provided. In this example, the company requests funds. The company's ability to repay the funds is evaluated as described herein. The full amount of funds may be provided when it is determined that the company can repay the full amount, or when it is predicted that the company cannot repay the full amount, a partial value of the funds may be provided according to the company's determined ability to repay the funds. In one or more second stages, following the features described with reference to 170A-B, one or more features 152-178 are implemented as a continuous monitoring of the entity parameters. In this example, the company's ability to repay the funds is monitored. When the company improves and the ML model ensemble predicts that the company can repay more funds than the requested funds, additional funds may be automatically granted upon request. Alternatively, a company's automated request for additional funding may be denied (or manual intervention required for approval) if the company's ability to repay the full amount of funds requested (or the amount provided) declines.

[0140] At 154, raw data elements are received as described with reference to 106 in Figure 1A. The raw data elements are extracted by code sensors installed on the network nodes and populated in real time to a server.

[0141] Optionally, the raw data elements are new raw data elements that were not used to create the ML model ensemble. Alternatively or additionally, the new raw data elements are used to update the ML model ensemble, as described with reference to 120 in FIG. 1A.

[0142] At 156, the raw data elements are input to a ML sub-model, for example, trained as described with reference to 108 in FIG. 1A.

[0143] Optionally, the raw data elements are provided to the ML sub-models sequentially. The raw data elements may be dynamically extracted and streamed to the ML sub-models. In another example, a set of raw data elements is first collected, for example over a time interval, and the set is then provided to a set of ML sub-models. The set of raw data elements may be obtained sequentially over sequential time intervals, for example, hourly, daily, or weekly.

[0144] At 158, sub-values ​​are obtained as a result of the trained ML sub-model, for example as described with reference to 108 in FIG. 1A.

[0145] At 160, adjusted sub-values ​​are generated by applying a simulated adjustment to the obtained sub-values, for example as described with reference to 110 in FIG. 1A.

[0146] At 162, the adjusted sub-values ​​are input into a main ML model that has been trained, for example as described with reference to 112 in FIG. 1A.

[0147] At 164, simulated values ​​of the entity parameters are obtained as a result of the main ML model, for example as described with reference to 114 in FIG. 1A.

[0148] At 166, the features described with reference to 160-164 are repeated to obtain multiple simulated values ​​of the entity parameter, for example as described with reference to 116 in FIG. 1A.

[0149] At 168, the multiple simulated values ​​are input to a risk classifier that calculates the probability that the entity will not meet the target parameters, for example as described with reference to 118 in FIG. 1A.

[0150] At 170A, in response to the probability that the entity will not satisfy the target parameters falling below a threshold (e.g., a threshold as described with reference to 118 in FIG. 1A), the target parameters may be provided to the entity (i.e., upon request), optionally provided automatically. For example, an amount of funds may be automatically deposited into a company account.

[0151] Alternatively, at 170B, in response to the probability that the entity will not satisfy the target parameters exceeding a threshold, the request for the target parameters is denied, optionally automatically denied.

[0152] For example, for the case where the entity is a company, the predicted values ​​of the entity parameters include the financial assets of the entity. The target parameters include the monetary funds required in the loan. If the ML model predicts that the financial assets of the company are less than the required loan funds, then the risk of the company not meeting the loan obligation (i.e., the company will not be able to repay its loan) is determined.

[0153] Optionally, in such cases, the risk classifier may be utilized to evaluate the values ​​of the target parameters that the entity is likely to meet. The entity may be provided with the evaluated values ​​of the target parameters that are less than the requested other parameters. For example, when a company is predicted to be unable to meet the full loan obligation, the loan amount provided to the company may be determined according to the predicted amount that the company is likely to meet (e.g., the company's predicted financial assets). The loan amount provided to the company may be according to the company's predicted ability to repay, rather than the full amount required.

[0154] At 172, a trend may be calculated according to the iteratively calculated updated probability that an entity parameter does not satisfy a target parameter, for example, sequential output of a risk classifier.

[0155] Optionally, the trends are extrapolated to future times to predict when the probability of an entity not satisfying the target parameters will be above or below a threshold. Trends may be analyzed, for example, to predict when a company that may currently be able to repay a loan will be unable to repay the loan, and / or to predict when a company that may not currently be able to repay a requested loan will be able to repay the requested loan. Such companies may be monitored, for example, to recall loans and / or to offer larger loans.

[0156] Optionally, an alert is generated according to the trend, for example, when the company is predicted to be unable to repay the loan in the future based on the trend, etc. The alert may be, for example, a message in the GUI, a pop-up message on the screen, an email, a call and / or a message sent to a mobile device, etc.

[0157] At 174, the weights of the functions of the main ML model may be analyzed to identify influential weights that most affect the entity parameters. A set of sub-functions may be calculated, each sub-function corresponding to a particular influential weight that correlates between the raw data elements and the values ​​of the entity parameters.

[0158] The identified most influential raw data elements may be monitored as predictors of the probability of change to the entity parameters. Changes in the influential raw data elements may trigger a recalculation of risk, for example, by repeating 154-168.

[0159] At 176, the interactive GUI presented on the display of the client terminal may be dynamically updated.

[0160] The GUI may be dynamically updated with an indication of the risk calculated by the risk classifier, e.g., indicating the amount of pre-approved funds available to the company that the company may be able to repay, e.g., as determined based on a dynamically calculated probability of obtaining sufficient financial assets to repay the pre-approved funds.

[0161] The GUI is dynamically updated in response to triggered repetitions, for example as described with reference to 178.

[0162] If the entity is a company, the GUI may be updated to present an indication of the amount of pre-approved funds available to the company. The amount of pre-approved funds may be determined based on a dynamically calculated probability of obtaining sufficient financial assets to repay the pre-approved funds output by the ML model ensemble. The GUI may present real-time pre-approved funds based on the real-time raw data elements.

[0163] Optionally, a request for an amount of funds up to a pre-approved amount of funds presented in the GUI (e.g., as described with reference to 152) is received, for example, via the GUI. The pre-approved amount of funds may be automatically deposited into the company's account, as described with reference to 170A.

[0164] Optionally, an alert is generated in the user interface when the predicted probability that an entity does not meet the target parameter changes. The alert may be generated, for example, during repeated monitoring of the output of the risk classifier, as described with reference to 178. In the example, the company's ability to repay the full amount of the loan (because the predicted financial assets are expected to exceed the value of the loan) is changed, and the company is currently predicted to be unable to repay the full amount of the loan (because the predicted financial assets are expected to be less than the value of the loan). In such a case, one or more automated and / or manual actions may be triggered. For example, the company may be required to repay a portion of the loan corresponding to its future payment ability, and / or the company may be automatically rejected for additional financing requests.

[0165] In 178, one or more features described with reference to 152 - 176 are repeated.

[0166] The repetition may be performed, for example, in response to a new request by the entity for the target parameter. Optionally, a change in one or more statistical measures of the raw data elements, for example, a statistically significant change, is monitored (e.g., by a code sensor). The repetition may be triggered in response to the detected statistically significant change, i.e., the input of the changed value of the raw data element to the trained ML sub-model as in 156.

[0167] The repetition may be performed, for example, at intervals such as daily, weekly, monthly, quarterly, or annually, to dynamically recalculate the risk.

[0168] Optionally, the iterations are performed as follows: At 154, updates for the raw data elements are iteratively obtained; At 156, the updates for the raw data elements are input to the ML sub-model; At 158, updated sub-values ​​are obtained; At 160, simulated adjustments are applied to the updated sub-values ​​to obtain adjusted updated sub-values; At 162, the adjusted updated sub-values ​​are input to the main ML model; At 164, simulated updated values ​​for the entity parameters are obtained as a result of the main ML model; At 166, 160-164 are repeated to obtain multiple simulated updated values ​​for the entity parameters; At 168, the simulated updated values ​​are input to a risk classifier to generate an updated probability that the entity does not satisfy the target parameter (e.g., according to an analysis of the simulated updated values ​​of the entity parameters).

[0169] Optionally, the iterations are triggered in response to new requests, such as increases and / or additional target parameters, e.g., a company requesting additional funding. Reference is made to FIG. 3, which is a data flow diagram illustrating an example data flow for generating an ensemble of ML models for calculating the probability that an entity does not satisfy a target parameter, according to some embodiments of the present invention. One or more of the features of the data flow diagram described with reference to FIG. 3 may correspond to features of the method described with reference to FIG. 1A, for example, as described herein. The data flow described with reference to FIG. 3 may be realized by one or more components of the system described with reference to FIG. 2.

[0170] At 302, raw data elements relating to an entity are received from a number of network node data sources, for example as described with reference to 106 in FIG. 1A (also referred to in FIG. 3 as “raw data”).

[0171] At 304, the raw data elements are provided to a sub (also referred to in FIG. 3 as a “model generator”), e.g., as described with reference to 108 in FIG. 1A. Note that the ML sub-model may be trained as described with reference to 102 in FIG. 1A.

[0172] At 306, sub-values ​​are obtained as a result of the ML sub-model, for example as described with reference to 108 in Figure 1A. The sub-values ​​may also be referred to as "target predictors."

[0173] At 308, the sub-values ​​are input into a main ML model (also referred to as the "risk model" in FIG. 3), for example as described with reference to 112 in FIG. 1A.

[0174] The risk model may be a machine learning model and / or a mathematical model, e.g., a function that maps a combination of sub-values ​​output by the ML sub-model to an entity parameter. For example, the risk model may be a model based on a conceptual model of "profit = revenue - cost", where profit corresponds to the entity parameter and revenue and cost correspond to the sub-values ​​and / or adjustments to the raw data elements output by the ML sub-model when provided with raw data elements.

[0175] At 310, simulated adjustments to the sub-values ​​are calculated by a simulator code, such as Monte Carlo simulation and distribution, for example, as described with reference to 110 in FIG. 1A.

[0176] At 311A ​​and 311B, the features described with reference to 306, 308 and 310 may be repeated to generate multiple simulated values ​​of the entity parameter, for example as described with reference to 116 in FIG. 1A.

[0177] At 312, a risk classifier is generated utilizing a set of simulated values ​​calculated for the entity parameters, for example as described with reference to 118 in FIG. 1A.

[0178] At 314, the risk classifier generates a probability that the entity does not satisfy the target parameters (also referred to as a "risk score + risk analysis"), for example as described with reference to 118 in Figure 1A. The term risk score may be interchangeable with the term entity score.

[0179] An ML model ensemble including an ML sub-model, a main ML model and a risk classifier is provided to calculate the probability that an entity that may be received in a request does not satisfy a target parameter, for example as described with reference to FIG. 1B.

[0180] Reference is now made to Fig. 4, which is a schematic diagram illustrating the results of a stress test process used to generate a risk classifier according to some embodiments of the present invention (the adjusted sub-values ​​are input into the main ML model to obtain simulated values ​​of the entity parameters). The stress tests correspond, for example, to features 110, 112, 114 and 116 described with reference to Fig. 1A and / or features 160, 162, 164 and 166 described with reference to Fig. 1B.

[0181] The stress test is performed with raw data elements obtained at three time intervals, shown as t1, t2, and t3, depicted along a time axis 402. At each time interval, multiple simulated values ​​for the entity parameters 404 are calculated. The simulated values ​​404 are obtained by adjusting the sub-values ​​(obtained by inputting the raw data elements into ML sub-models) and inputting the sub-values ​​into a main ML model. The simulated values ​​are obtained as a result of the main ML model, as described herein.

[0182] The risk classifier may calculate an entity score according to the analysis of the simulated values ​​404. The entity score may be indicative of the probability that the entity will not meet the target parameters. The simulated values ​​of the entity parameters 404 may be normalized within a defined scale 406A-B, such as within a range of 0-800, 0-100, or other values. The entity score may be calculated against a threshold 408 set within the scale. The threshold 408 may represent a probability that indicates the likelihood that the target parameters will be met by the entity based on the entity parameters, independent of the actual value of the target parameters and / or without knowing what the target parameters actually are. For example, for an example where the entity is a company, when the calculated entity parameter represents available cash, the threshold 408 represents 3 times the operating expenses, and when the available cash falls below the threshold, the company may not be able to repay the loan regardless of the amount discussed (i.e., there is no alternative cash left). When the available cash is above the threshold, the company may be able to repay the loan (i.e., enough cash to cover the operating expenses and enough additional cash available). The entity score may be calculated using a formula 410 calculated from a number of simulated entity parameters 404 and may indicate the probability that the entity parameter is above or below a threshold 408 .

[0183] The generated risk classifier, which may output an entity score in response to input of simulated values ​​404 , may include an equation 410 and a threshold 408 .

[0184] The entity score may be calculated independently of other values ​​described herein. The entity score may be presented in a GUI on the display.

[0185] Reference is made to FIG. 5, which is a schematic diagram illustrating a process for a threshold 508 for determining whether an entity score of a particular entity represents a probability that the entity does not satisfy a target parameter according to some embodiments of the present invention. Entity scores 504 are obtained that are calculated for a number of entities (as described with reference to FIG. 1B and / or FIG. 4). Each entity score 504 is calculated with respect to the other entities by repeating for each entity the features described with reference to FIG. 1B and / or FIG. 4. The entity scores 504 may be normalized to fall within a range 506A-B (e.g., 0-800, 0-100, or other) and / or to fall within the range 506A-B. The threshold 508 may be set according to the distribution of the entity scores, e.g., the bottom 5% of the entity scores are below the threshold 508 and the top 95% are above the threshold 508. Other thresholds may be utilized, e.g., 25 / 75%, 50 / 50%, or other.

[0186] Each entity score 504 calculated for each entity may be evaluated against other entity scores of other entities to determine whether the entity score is above or below a threshold 508. For example, the status of each entity's entity score versus the entity scores of the other entities may be analyzed and / or presented in a GUI. For example, when the entity score 504 is below the threshold 508, the entity corresponding to the entity score may be determined to likely not meet the target parameters, and when the entity score is above the threshold, the entity may be determined to likely meet the target parameters.

[0187] The description of various embodiments of the present invention has been provided for illustrative purposes, but is not intended to be limiting or exhaustive with respect to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0188] It is expected that during the life of the patent resulting from this application, many relevant ML models will be developed, and the scope of the term ML model is intended to proactively include all such new technologies.

[0189] As used herein, the term "about" refers to ±10%.

[0190] The terms "comprising," "including," "having," and combinations thereof mean "including, without limitation." This term encompasses the terms "consisting of" and "consisting essentially of."

[0191] The phrase "consisting essentially of" means that the composition or method may contain additional components and / or steps, but only if the additional components and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0192] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. For example, "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0193] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the inclusion of features from other embodiments.

[0194] The term "optionally" is used herein to mean "provided in some embodiments and not in others." Unless such features are inconsistent, any particular embodiment of the invention may include a plurality of "optional" features.

[0195] Throughout this application, various embodiments of the invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Thus, the description of a range should be considered to specifically disclose all the possible sub-ranges as well as the individual numerical values within that range. For example, a description of a range such as 1-6 should be considered to specifically disclose sub-ranges such as 1-3, 1-4, 1-5, 2-4, 2-6, 3-6, as well as the individual numbers within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0196] Whenever a numerical range is indicated herein, it is meant to include any recited number (fractional or integral) within the indicated range. The phrase "range" between a first designating number and a second designating number, and the phrase "range" from the first designating number to the second designating number are used interchangeably herein and are meant to include the first designating number, the second designating number, and all fractions and integers therebetween.

[0197] It is understood that specific features of the invention that are described in the context of separate embodiments may be provided in combination in a single embodiment. Conversely, various features of the invention that are described in the context of a single embodiment for brevity may be provided separately in any other described embodiment of the invention or in any suitable sub-combination. Specific features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiment would be inoperative without those elements.

[0198] While the present invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, the present invention is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0199] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated herein by reference. Furthermore, citation or identification of any reference in this application should not be construed as an admission that such reference is available as prior art to the present invention. To the extent section headings are used, they should not be construed as necessarily limiting. Additionally, any priority documents of this application are incorporated herein by reference in their entirety.

Claims

1. 1. A method for generating an ensemble of machine learning (ML) models for computing a probability that an entity does not satisfy a target parameter, comprising: training a plurality of ML sub-models, each of which outputs a sub-value in response to an input of raw data elements; training a main ML model using a training data set including the ML sub-values ​​and corresponding entity parameters for each of a plurality of sample entities, the main ML model outputting a value of an entity parameter corresponding to the target parameter in response to input of the sub-values ​​output by the ML sub-models; inputting a plurality of raw data elements associated with said entity into said plurality of ML sub-models to obtain respective sub-value outputs; calculating a plurality of simulated adjustments to the plurality of obtained sub-values ​​to generate a plurality of adjusted sub-values ​​in a plurality of iterations, inputting the plurality of adjusted sub-values ​​into the main ML model, and obtaining a plurality of simulated values ​​for the entity parameter from the main ML model in the plurality of iterations; generating a risk classifier according to an analysis of the plurality of simulated values ​​for the entity parameters, the risk classifier generating a probability that the entity will not satisfy the target parameters; The method according to claim 1,

2. 2. The method of claim 1, wherein the risk classifier is generated based on a few-shot or single-shot training approach utilizing the raw data elements as initial inputs by generating additional synthetic data for generating the risk classifier by calculating and utilizing the simulated values ​​for the entity parameters output by the main ML model in the multiple iterations in response to input of the adjusted sub-values ​​for the multiple obtained sub-values ​​calculated by applying each of the simulated adjustments to the multiple ML sub-models when the multiple raw data elements are provided.

3. 2. The method of claim 1 , wherein each of a plurality of simulated adjustments is calculated for a respective sub-value of the plurality of obtained sub-values, and each of the plurality of simulated adjustments is applied to each sub-value of the plurality of obtained sub-values ​​output by the plurality of ML sub-models to generate a plurality of adjusted sub-values ​​for each sub-value.

4. 2. The method of claim 1 , wherein each of a set of adjusted sub-values ​​is calculated by calculating, for each of the plurality of sub-values, each of a set of a plurality of simulated adjustments for each of the plurality of sub-values ​​and applying each of the plurality of simulated adjustments to a corresponding one of the plurality of sub-values.

5. The method of claim 1 , wherein the plurality of simulated adjustments to each sub-value of the plurality of sub-values ​​are selected from the group consisting of increasing the each sub-value, decreasing the each sub-value, and creating a new sub-value.

6. identifying at least one most influential raw data element of the plurality of raw data elements having the greatest impact on the entity parameter; 2. The method of claim 1, wherein calculating the plurality of simulated adjustments is performed on sub-values ​​output by at least one ML sub-model of the plurality of ML sub-models that receives the at least one most influential raw data element as an input.

7. repeatedly obtaining updates to said plurality of raw data elements; iteratively inputting updates to the raw data elements into the ML sub-models to obtain updated sub-values; In multiple updated iterations, calculating the plurality of simulated adjustments for each of the plurality of updated sub-values ​​to obtain a plurality of adjusted updated sub-values; inputting the adjusted updated sub-values ​​into the main ML model; obtaining a plurality of simulated updated values ​​for the entity parameters; generating an update to the risk classifier utilizing an updated analysis of the plurality of simulated updated values ​​to generate an updated probability that the entity will not satisfy the target parameter; The method of claim 1 further comprising:

8. the main ML model includes a plurality of parameters associated with the sub-values ​​output by the plurality of ML sub-models, the plurality of parameters being functions of the plurality of parameters associated with a plurality of weights, and training the main ML model includes learning the plurality of weights; The method of claim 1 , wherein the weights of the function are calculated based on automated combinatorial deformation scenarios affecting the entity parameters.

9. receiving a threshold indicative of a risk that the entity will not satisfy the target parameter, the analysis including calculating the probability by calculating an entity score based on the simulated values ​​being above or below the threshold across a set of simulated values; determining a probability that the entity satisfies the target parameter when the entity score exceeds the target parameter; The method of claim 1 further comprising:

10. 2. The method of claim 1, wherein the plurality of raw data elements related to the entity are selected from the group consisting of structured data obtained from a structured data source that stores data for the entity, numerical data obtained from a data source that calculates numerical data for the entity, and unstructured data obtained from open sources and / or social networks.

11. At least one of the plurality of ML sub-models (i) outputting each metric based on a correlation window between values ​​of said raw data elements of a particular type obtained during a first time interval and values ​​of said raw data elements of said particular type obtained during at least one second time interval prior to said first time interval; (ii) calculating each metric given values ​​of a set of raw data elements of a defined type; (iii) calculating each metric by a set of combinations of a plurality of sub-sub-models selected according to the type of the entity; (iv) a sub-classifier that, when provided with a selected set of raw data elements, outputs a probability that the entity satisfies the target parameter, the sub-classifier being trained on a training data set that includes, for each of a plurality of sample entities, sample data elements associated with each sample entity and an indication of satisfaction of each target parameter provided for each sample entity; (v) outputting a risk of failure of a component of the plurality of components of the entity for the entity parameters; (vi) outputting indicators and / or predictive parameters based on an analysis of raw data elements indicative of users accessing websites associated with the entity; (vii) outputting an indication of a user within the entity based on an analysis of the user's profile posted on a social network operated by an external social network server; and (viii) outputting indicators and / or predictive parameters based on an analysis of the marketing elements indicative of users interacting with advertisements and / or promotions associated with the entity; The method of claim 1 , wherein the compound is selected from the group consisting of:

12. The method of claim 1 , wherein the plurality of simulated adjustments are calculated based on a probabilistic simulation model and a set of prior distributions as calculated or provided from raw data.

13. 2. The method of claim 1, wherein features of the method are implemented by at least one hardware processor executing on a server, and the raw data elements are extracted by code sensors installed on a plurality of network nodes and populated to the server in real time.

14. the entity includes a company, the target parameters specified in the request include funds specified in the funding request to fund the company, and the entity parameters include financial assets of the entity that will be utilized to repay funds for the funding request; 2. The method of claim 1, wherein the plurality of raw data elements related to the entity are selected from the group consisting of data elements indicative of customers of the company, data elements indicative of financial details of the company, and data elements obtained from third party sources related to the company.

15. At least one of the plurality of ML sub-models (i) an indication of the company's growth calculated as a ratio between revenue during a first time interval and revenue obtained during at least one second time interval prior to the first time interval, the revenue calculated based on a plurality of first type of raw data elements including invoices and / or payments obtained from a charging and / or billing system integration; (ii) an indication of customer churn calculated based on a plurality of second type raw data elements including invoices, payments, and / or data obtained from a customer relationship management (CRM) system; (iii) an indication of a gross margin calculated based on a plurality of the third type of raw data elements; (iv) an indication of unit economics calculated as a collection of combinations of a plurality of sub-sub-models selected according to a business type of the company; (v) a sub-classifier that, when provided with a plurality of raw data elements of a fourth type, outputs a probability that the company will raise a fund, the sub-classifier being trained on a training data set that includes, for each of a plurality of sample companies, values ​​of the data elements of the fourth type and an indication of the funds raised by the respective company; (vi) when provided with the plurality of fifth type data elements, outputting a risk of failure of a revenue generating unit of the plurality of revenue generating units of the company to the company based on a calculation of a statistical distribution of the plurality of revenue generating units; (vii) outputting indicators and / or predictive parameters based on an analysis of a sixth type of data element indicative of users accessing a website associated with the company and / or based on an analysis of a seventh type of data element indicative of users interacting with a served advertisement; (viii) outputting instructions for reassignment of employees within the company based on an analysis of employee profiles posted on a social network operated by an external social network server and / or a further external source operated by a network node; (ix) based on analysis of commitments including contracts and / or purchase orders obtained via enterprise resource planning (ERP) system integration, outputting real-time short-term and / or long-term cost forecasts, detected expected significant costs versus external contractors, changes in prices of services provided, bills of materials (BOMs) and / or ongoing purchase order instructions; 15. The method of claim 14, further comprising outputting respective indices and / or predictive parameters selected from the group consisting of:

16. 1. A method for dynamically and iteratively utilizing an ensemble of ML models to compute a probability that an entity does not satisfy a target parameter, comprising: In the first plurality of iterations, receiving a request from an entity to obtain target parameters; receiving a plurality of raw data elements extracted by code sensors installed on a plurality of network nodes and populating a server in real time; inputting the raw data elements into a plurality of ML sub-models; Obtaining a plurality of sub-values ​​from the plurality of ML sub-models; In the second plurality of iterations, calculating a plurality of simulated adjustments to the plurality of obtained sub-values ​​to generate a plurality of adjusted sub-values; inputting the plurality of adjusted sub-values ​​into a main ML model; obtaining a plurality of simulated values ​​of entity parameters from the main ML model in the second plurality of iterations; inputting the plurality of simulated values ​​of the entity parameters into a risk classifier that calculates a probability that the entity will not satisfy the target parameter according to an analysis of the plurality of simulated values ​​of the entity parameters; automatically providing the entity with the target parameters when the probability falls below a threshold; automatically denying a request for the target parameter when the probability exceeds the threshold; The method comprising:

17. 17. The method of claim 16, wherein the risk classifier is generated based on a few-shot or single-shot training approach utilizing the raw data elements as initial inputs by generating additional synthetic data for generating the risk classifier by calculating and utilizing the simulated values ​​for the entity parameters output by the main ML model in the multiple iterations in response to input of the adjusted sub-values ​​for the multiple obtained sub-values ​​calculated by applying each of the simulated adjustments to the multiple ML sub-models when the multiple raw data elements are provided.

18. repeatedly obtaining updates to said plurality of raw data elements; iteratively inputting updates to the raw data elements into the ML sub-models to obtain updated sub-values; In multiple updated iterations, calculating the plurality of simulated adjustments for each of the plurality of updated sub-values ​​to obtain a plurality of adjusted updated sub-values; inputting the adjusted updated sub-values ​​into the main ML model; obtaining a plurality of simulated updated values ​​for the entity parameters; inputting the plurality of simulated updated values ​​of the entity parameters into the risk classifier for generating an updated probability that the entity will not satisfy the target parameters according to an analysis of the plurality of simulated updated values ​​of the entity parameters; The method of claim 16 further comprising:

19. 17. The method of claim 16, further comprising: calculating a trend according to the iteratively calculated updated probability that the entity parameters do not satisfy the target parameters; and extrapolating the trend to a future time to predict a trend when the probability that the entity does not satisfy the target parameters exceeds a threshold.

20. monitoring a change in at least one statistical quantity of said plurality of raw data elements; triggering an iteration in response to said change and / or when the change in said statistic is significant; The method of claim 16 further comprising:

21. analyzing a plurality of weights of the function to identify a plurality of influential weights that most affect the value of the entity parameter; calculating a set of sub-functions, each sub-function corresponding to a specific influence weight for correlating between at least one raw data element and the value of said entity parameter; The method of claim 16 further comprising:

22. 17. The method of claim 16, wherein in response to repeatedly obtaining updates of the plurality of raw data elements, an interactive graphical user interface (GUI) presented on a display of a client terminal is dynamically updated with an indication of an entity score indicating the probability that the entity will not satisfy the target parameters and an amount of pre-approved funding available to the business determined based on a dynamically calculated probability of obtaining sufficient financial assets to repay the pre-approved funding.

23. receiving a request via the GUI for a funding amount up to the pre-approved funding amount presented in the GUI; automatically providing the funding amount to an account of the business; 23. The method of claim 22, further comprising:

24. 1. A system for generating an ensemble of machine learning (ML) models for computing a probability that an entity does not satisfy a target parameter, comprising: training a plurality of ML sub-models, each of which outputs a sub-value in response to an input of raw data elements; training a main ML model using a training data set including a plurality of ML sub-values ​​and corresponding entity parameters for each of a plurality of sample entities, the main ML model outputting a value of an entity parameter corresponding to the target parameter in response to input of the plurality of sub-values ​​output by the plurality of ML sub-models; inputting a plurality of raw data elements associated with the entity into the plurality of ML sub-models to obtain an output for each sub-value; In multiple iterations, calculating a plurality of simulated adjustments to the plurality of obtained sub-values ​​to generate a plurality of adjusted sub-values; inputting the adjusted sub-values ​​into the main ML model; obtaining a plurality of simulated values ​​of the entity parameters from the main ML model at the plurality of iterations; generating a risk classifier according to an analysis of the plurality of simulated values ​​of the entity parameters, the risk classifier generating a probability that the entity will not satisfy the target parameter; A system having at least one hardware processor that executes code for:

Images