Estimating the feasibility and effort of machine learning solutions

Abstract

A method, computer system, and computer program product for evaluating a likelihood of success associated with developing at least one machine learning (ML) solution is provided. The invention may include generating a set of questions based on an original training data set. The invention may also include calculating a feasibility score based on answers corresponding to each question in the generated set of questions. The invention may then include: in response to determining that the calculated feasibility score satisfies a threshold, calculating a level of effort associated with developing at least one ML solution to solve the problem. The invention may further include presenting to a user a plurality of results associated with evaluating the likelihood of success of the at least one ML solution.

Description

Background Art

[0001] The present invention relates generally to the field of computing, and more particularly to the development of machine learning (ML).

[0002] The success of an artificial intelligence (AI) project may depend on multiple conditions, such as the availability of labeled training data, labeling techniques, and lightweight domain knowledge. Summary of the invention

[0003] Embodiments of the present invention disclose a method, computer system, and computer program product for evaluating the likelihood of success associated with developing at least one machine learning (ML) solution. The present invention may include generating a set of questions based on an original training data set. The present invention may also include calculating a feasibility score based on answers corresponding to each question in the generated set of questions. The present invention may then include: in response to determining that the calculated feasibility score satisfies a threshold, calculating a level of effort associated with developing at least one ML solution to solve the problem. The present invention may further include presenting to a user a plurality of results associated with evaluating the likelihood of success of the at least one ML solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings. The various features of the accompanying drawings are not drawn to scale because the accompanying drawings are for the sake of clarity and facilitate the understanding of the present invention by those skilled in the art in conjunction with the specific embodiments. In the accompanying drawings:

[0005] Figure 1 illustrates a networked computer environment according to at least one embodiment;

[0006] Figure 2 is an operational flow diagram illustrating a process for evaluating a likelihood of success associated with developing at least one machine learning (ML) solution in accordance with at least one embodiment;

[0007] Figure 3 According to at least one embodiment Figure 1 block diagrams of the internal and external components of the computers and servers shown;

[0008] Figure 4 According to an embodiment of the present disclosure, Figure 1 A block diagram of an illustrative cloud computing environment of a computer system is shown; and

[0009] Figure 5 According to the embodiment of the present disclosure Figure 4 A block diagram of the functional layers of an illustrative cloud computing environment. DETAILED DESCRIPTION

[0010] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be embodied in various forms. However, the present invention may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Instead, these exemplary embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the scope of the invention to those skilled in the art. In the specification, details of well-known features and techniques may be omitted to avoid unnecessary obfuscation of the presented embodiments.

[0011] At any possible technical detail combination level, the present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0012] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0013] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0014] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" language, Python programming language, or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0015] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0016] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0017] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0018] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0019] The exemplary embodiments described below provide a system, method, and program product for developing at least one machine learning (ML) solution. Thus, the present embodiment has the ability to improve the technical field of ML solutions by estimating the feasibility score and effort level involved in the development of ML solutions. More specifically, a success assessment program may receive an original training data set. Next, the success assessment program may generate a series of questions, and corresponding answers to the series of questions, and then calculate a feasibility score and compare it to a threshold. If the feasibility score meets the threshold, the effort level is compared and the result is presented to the user.

[0020] As described earlier, the success of an artificial intelligence (AI) project may depend on multiple conditions, such as the availability of labeled training data, labeling techniques, and lightweight domain knowledge.

[0021] If anything, the worst experience may be developing a successful ML solution when any of these conditions break or do not exist. However, these conditions often break down in business application domains. For example, in business solution domains, where only a small amount of labeled data is available, labeling is not practical, or subject matter experts (SMEs) cannot perform labeling, training and modeling will include a high degree of domain knowledge. As a result, the likelihood of success associated with developing a successful ML solution is lower, especially for the aforementioned business solution domains.

[0022] Furthermore, when developing ML solutions, increased or significant amounts of human labor may result in minimal results (if any), thereby reducing the likelihood of success. Therefore, an effective way to increase the likelihood of success is to provide an early assessment of the feasibility of a solution, a means of early assessment of the level of effort that may be involved in a feasible solution.

[0023] Therefore, it may be advantageous to develop, among other things, a success likelihood assessment method for evaluating the success likelihood associated with developing at least one successful ML solution by estimating the feasibility score and effort level under challenging business domain conditions (e.g., when only a small amount of labeled data is available, labeling may not be practical (e.g., there are limited SMEs to perform labeling), and training and modeling involve domain knowledge).

[0024] According to at least one embodiment, the success assessment program can bootstrap the learning process from a set of rules and enhance learning based on rules defined by subject matter experts (SMEs). In this embodiment, the success assessment program can execute ML algorithms from a shared experience pool. Data sets can be collected from multiple projects and potential new ML solutions can be classified as feasible or infeasible, and then compared with the range of efforts based on similarity or distance to past solution categories or classifications.

[0025] According to at least one embodiment, the success assessment program can determine the feasibility of a particular ML solution. By determining feasibility, the success assessment program can assign a numerical score to each answer to a question generated by the assessment project (e.g., "yes" is assigned a numerical score of 1 and "no" is assigned a numerical score of 0). The sum of the assigned numerical scores associated with each generated answer to the question can be used to calculate the feasibility score. In another embodiment, the success assessment program can modify the numerical scores of the questions and add more values ​​and / or provide more complex functions (e.g., weighted sums) for calculating the feasibility score.

[0026] According to at least one embodiment, the success assessment program can estimate the level of effort. If the answer to the generated question is "yes" or affirmative, the success assessment program can provide more granular options for assessing the question's required effort (e.g., low, medium, or high). The success assessment program can then apply a function (e.g., a weighted sum) to determine the level of effort.

[0027] According to at least one embodiment, a success assessment program may include specific design elements, namely, including defined system business value metrics, the ability to obtain unused test sets associated with the ML solution, representation of real-life data in the original training data, approximately 30 pre-labeled examples, statistically estimated accuracy of annotations, the possibility of continuous annotation, and ten times more data than features (if a generalized model is obtained).

[0028] According to at least one embodiment, the success assessment procedure may include model development, i.e., automatic relearning from the original training data (e.g., the entire data pipeline can be automated). For an object, there may be a mapping that captures the features of each stage of the data pipeline, and a complete list of hyperparameters may be provided (e.g., for each hyperparameter, if the optimal value is automatically searched as part of an automated data pipeline, each hyperparameter that is not searched will reduce the likelihood of success). This embodiment may include models developed using techniques for advanced statistical estimation performance (at least k-fold cross validation). In addition, one or more non-parametric confidence intervals can be developed for all means, and explicit statistical assumptions can be made during data pipeline development and appropriate checks implemented at deployment. For example, two elements may be correlated and one of the elements is deleted. At deployment, the success assessment procedure can check whether the two elements are still correlated.

[0029] According to at least one embodiment, a success assessment procedure may include testing and deployment. Testing may be performed against one or more ground truth data sets that may never be used for learning using advanced statistical techniques such as bootstrapping and nonparametric confidence intervals. One or more ground truth data sets may be sharded to identify weaknesses (e.g., assets under development) and to develop statistical hypotheses during the development phase.

[0030] Reference Figure 1, shows an exemplary networked computer environment 100 according to one embodiment. The networked computer environment 100 may include a computer 102 with a processor 104 and a data storage device 106 that enables running a software program 108 and a success assessment program 110a. The networked computer environment 100 may also include a server 112 that is capable of running a success assessment program 110b that may interact with a database 114 and a communication network 116. The networked computer environment 100 may include multiple computers 102 and servers 112, only one of which is shown. The communication network 116 may include various types of communication networks, such as a wide area network (WAN), a local area network (LAN), a telecommunications network, a wireless network, a public switched network, and / or a satellite network. It should be noted that Figure 1 Only a description of one implementation is provided and no limitation is implied in the environments in which different embodiments may be implemented. Many modifications may be made to the described environment based on design and implementation requirements.

[0031] The client computer 102 can communicate with the server computer 112 via a communication network 116. The communication network 116 may include connections such as wired, wireless communication links, or fiber optic cables. Figure 3 As discussed, the server computer 112 may include internal components 902a and external components 904a, respectively, and the client computer 102 may include internal components 902b and external components 904b, respectively. The server computer 112 may also operate in a cloud computing service model, such as software as a service (SaaS), analysis as a service (AaaS), platform as a service (PaaS), or infrastructure as a service (IaaS). The server 112 may also be located in a cloud computing deployment model, such as a private cloud, a shared cloud, a public cloud, or a hybrid cloud. The client computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program or accessing a computer. According to various implementations of the present embodiment, the success assessment program 110a, 110b may interact with a database 114, which may be embedded in various storage devices, such as, but not limited to, a computer / mobile device 102, a network server 112, or a cloud storage service.

[0032] According to the present embodiment, a user using the client computer 102 or the server computer 112 can use the success assessment program 110a, 110b to develop at least one ML solution by estimating the feasibility score and the effort level, respectively. Figure 2 Explain the success assessment methodology in more detail.

[0033] Reference now Figure 2, shows an operational flow diagram illustrating an exemplary machine learning (ML) solution development process 200 used by the success assessment programs 110a, 110b according to at least one embodiment.

[0034] At 202, a raw training data set is received. Using a software program 108 on a user device (e.g., user computer 102), at least a raw training data set can be automatically received via a communication network 116 as an input to a success assessment program 110a, 110b. The raw training data set can include real-life data associated with reports, documents, execution logs, and / or source code, as well as other media for storage and / or transmission data associated with a specific task (e.g., health records and / or medical tests for identifying adverse events, X-rays for determining whether a radiologist and / or human expert should perform an examination and / or mammography, software test execution logs for determining whether a test failure is due to a problem in the execution environment or a software problem in the underlying test). In at least one embodiment, a user can manually upload the raw training data set into the success assessment program 110a, 110b.

[0035] For example, a clinical trial of a new drug for hypertension is currently being conducted on 600 patients. A research scientist for the clinical trial recommends using a machine learning (ML) solution to identify adverse events (or side effects) of the new drug based on the original training data, i.e., the medical reports associated with each patient in the clinical trial. The research scientist further decides to utilize a success assessment procedure 110a, 110b to assess the likelihood of success associated with the ML solution for identifying adverse events.

[0036] Next, at 204, a set of questions is generated. These questions may capture best practices and find shortfalls in the received raw training data set. In at least one embodiment, the questions may be manually defined, and in some other embodiments, the questions may be automatically extracted. The success assessment program 110a, 110b may train a machine learning (ML) model that takes input data associated with successful and unsuccessful ML models. The questions may then be aggregated through the stages of developing a specific ML model that is applied to the received raw training data set.

[0037] In at least one embodiment, the success assessment process 110a, 110b may include testing and deployment. Testing may be performed on one or more base real data sets that can be used to learn using advanced statistical techniques (e.g., bootstrapping and non-parametric confidence intervals). The one or more real data sets may be segmented to identify weaknesses (e.g., assets being developed), and statistical hypotheses may be developed during the development phase.

[0038] In at least one embodiment, the questions may be generated by an extractor. By utilizing natural language processing techniques (e.g., classification, sentiment, entity extraction, topic modeling), the extractor may parse the received raw training data set to identify and extract any best practices or deficiencies in the received raw training data set for generating questions for the user.

[0039] In this embodiment, the success assessment program 110a, 110b may generate at least one question. However, there may be a maximum limit on the questions generated (e.g., the default maximum limit of questions generated is 20). In some embodiments, the success assessment program 110a, 110b has no maximum limit on the number of questions generated. For example, the success assessment program 110a, 110b may generate as many questions as possible based on the identified deficiencies and best practices associated with the received original training data set.

[0040] In addition, the success assessment program 110a, 110b can present the generated questions to the user. For example, the success assessment program 110a, 110b can present the generated questions through a dialog box or a chatbot screen. The user can be prompted to answer each generated question with an affirmative answer (e.g., by writing "yes" in the command box below each question, by clicking the green or check mark button below each question) or a negative answer (e.g., by writing "no" in the command box below each question, by clicking the red or "X" mark button behind each question). In at least one embodiment, the success assessment program 110a, 110b can present the questions separately. Once the answer to the question is presented and answered by the user, the success assessment program 110a, 110b can generate another question for the user to respond.

[0041] In some embodiments, the success assessment program 110a, 110b can immediately present the generated list of questions to the user. Below each question, the success assessment program 110a, 110b can include a space (e.g., a command box, or a positive answer button and a negative answer button) for the user to provide an answer to each generated question.

[0042] In at least one embodiment, after the user answers the last generated question, the success assessment program 110a, 110b may present a message to the user (eg, "Question successfully answered," "Thank you for your answer") confirming receipt of the answer.

[0043] In at least one embodiment, the generated questions and corresponding answers may be saved and stored in database 114 .

[0044] In another embodiment, the success assessment program 110a, 110b may include a "Don't know" value or button (eg, represented by a question mark on the button or the phrase "Don't know" on the button) as a response to the generated question.

[0045] In at least one embodiment, a user can identify hyperparameters and automatically search for desired values ​​(i.e., feasibility scores), and a user can apply confidence intervals to ML performance metrics, which can be reflected in generated questions.

[0046] Continuing with the previous example, the medical professionals associated with the clinical trial, i.e., the research team consisting of scientists, doctors, nurses, dietitians, social workers, and psychiatrists, previously provided each patient with a list of 15 questions to answer. The success assessment program 110a, 110b then prompts each patient with each of the 15 questions previously provided, and further prompts each patient to provide a "yes" or "no" answer to each question. The success assessment program 110a, 110b then saves the questions and corresponding answers associated with each patient in the database 114. Once the patient completes the 15 questions, the success assessment program 110a, 110b is presented with a command "Thank you for your answer", indicating that the patient has successfully responded to the question.

[0047] Then, at 206, a feasibility score is calculated. The success assessment program 110a, 110b can generate a feasibility score based on the answers to the generated questions, where the feasibility score determines whether the data generated by the answers to the generated questions is sufficient or practical for developing an ML solution. The answers can be in binary form (e.g., "yes" or "no"). Each answer to the generated question can be assigned a numerical value (e.g., "yes" is assigned 1, "no" is assigned 0), and then the assigned numerical values ​​are added together, and the sum of the assigned numerical values ​​can be used as the feasibility score.

[0048] In at least one embodiment, the user of the success evaluation program 110a, 110b may be the person who developed the ML model.

[0049] In at least one embodiment, the success assessment procedures 110a, 110b may have a third answer option of "Don't know" (i.e., may include one or more variations of "Don't know", such as "Unsure", "Not sure"). The "Don't know" answer may be scored as "-1" or a similar value to indicate that further thought or analysis may be beneficial before continuing with a particular ML solution.

[0050] In another embodiment, the success assessment program 110a, 110b may vary the score for each question, where answers to certain questions may be weighted more heavily than other answers. Thus, the success assessment program 110a, 110b may calculate the feasibility score using a weighted sum.

[0051] In one embodiment, when calculating multiple feasibility scores, the success assessment program 110a, 110b can sort the feasibility scores in a previously determined order (e.g., from high to low, from low to high). In some embodiments, the success assessment program 110a, 110b can use other data associated with the users and / or subjects of the original training data set (which is associated with the feasibility scores) to sort the feasibility scores. For example, if the ML solution includes treatments for problems of different age groups, then the feasibility scores can be sorted in ascending or descending order by the age of the people associated with the original training data set used to calculate the feasibility scores.

[0052] In one embodiment, the adverse event classifier can be used to identify adverse events that occur in the original training data set received by the success assessment program 110a, 110b. By identifying one or more adverse events, the adverse event classifier can send the one or more adverse events to the success assessment program 110a, 110b via the communication network 116 to calculate a feasibility score.

[0053] Continuing with the previous example, for each adverse event, such as hallucinations, dizziness, numbness in the limbs, etc., experienced by the 600 patients in the clinical trial, a feasibility score is calculated. Each "yes" response is assigned a "1" and each "no" response is assigned a "0." Therefore, the highest feasibility score that an adverse event can obtain is "15." The success assessment procedures 110a, 110b calculate a feasibility score for each adverse event, ranging from 1 to 15. The success assessment procedures 110a, 110b further identify data related to each person in the original training data set, such as each person's age, gender, medical history and diseases, typical diet, and nationality.

[0054] The feasibility score is then compared to a threshold at 208. The success assessment program 110a, 110b may compare the feasibility score to a previously determined threshold level (e.g., some standardized value with a default level of 75% of 100%, 0.75 of 1, 7.5 of 10, 75 of 100, or the like).

[0055] In at least one embodiment, the thresholds may be predetermined by a person (e.g., an administrator or user), where the person may configure the threshold levels. For example, a user may click a "Settings" button located in the lower right corner of the main screen. The user is then prompted with a first dialog box with a list of different setting topics or options. When the user clicks the "Thresholds" button, the first dialog box may disappear and a second dialog box may appear. The second dialog box includes a list of previously selected thresholds, and a text box for the user at the bottom of the second dialog box to include a new threshold level. Each previously selected threshold level includes a button to the left of the previously selected threshold, in which the user can click to select the previously selected threshold level. After completing the threshold selection, the user can click a "Submit" button located below the text box at the bottom of the second dialog box. The second dialog box will disappear and the new threshold level will be implemented immediately.

[0056] In some embodiments, the threshold level can be automatically determined by the success assessment program 110a, 110b based on a learning method. The success assessment program 110a, 110b can utilize various threshold setting techniques (e.g., by using historical data to compare data and ratio ratios to similar time periods in the past, doubling or tripling the absolute number of cases in a given time period, crossing a predefined threshold, modeling the historical data through time series analysis techniques that can account for trends and seasonality in the data and notify any recent changes that may have affected the data). The threshold setting techniques can identify the most appropriate threshold for a particular task based on the data that the success assessment program 110a, 110b can implement.

[0057] Continuing with the previous example, the scientist research team previously selected "11" as the threshold level for the feasibility score of the clinical trial. Of the 800 adverse events experienced by patients, only 150 adverse events met the threshold level (or feasibility score was higher than 11).

[0058] Then, at 210, the success assessment program 110a, 110b determines whether the feasibility score satisfies a threshold. The success assessment program 110a, 110b can then evaluate the feasibility score to determine whether the feasibility score is less than, equal to, or greater than a previously determined threshold. Based on whether the feasibility score satisfies the threshold, the success assessment program 110a, 110b can utilize a classifier to identify the ML solution as successful (e.g., the feasibility score satisfies the threshold) or unsuccessful (e.g., the feasibility score does not satisfy the threshold).

[0059] In at least one embodiment, the success assessment program 110a, 110b can present to the user whether the feasibility score meets the threshold by utilizing visualization. For example, the success assessment program 110a, 110b presents whether the feasibility score meets the threshold with an indicator or a gauge indicator that immediately shows that the feasibility score is less than, equal to, or greater than the threshold. In another embodiment, the success assessment program 110a, 110b can incorporate color coding (e.g., red or green) and / or up and down arrows to indicate whether the feasibility score meets the threshold.

[0060] In some embodiments, the success assessment programs 110a, 110b may utilize graphs and / or charts (eg, bar graphs, pie charts, line graphs, column graphs, pivot tables, area graphs, scatter graphs, and / or bubble charts) to display the feasibility scores.

[0061] In at least one embodiment, a user can determine in advance whether a feasibility score visualization can be presented to the user by the success assessment program 110a, 110b after the feasibility score is compared to the threshold, or later. In this embodiment, the success assessment program 110a, 110b can determine how to present the feasibility score based on the visualization effects that the user can previously select. In some embodiments, the success assessment program 110a, 110b can automatically determine how to present whether the feasibility score meets the threshold based on various factors (e.g., the number of feasibility scores and the number of thresholds, the difference between the feasibility score and the threshold, and the user's preferences). The user's preferences can include the user's previous selections based on factors related to the feasibility score and the threshold.

[0062] In at least one embodiment, the success assessment programs 110a, 110b can utilize an ML model to learn how to visualize whether a feasibility score meets a threshold based on past training data, factors associated with the feasibility score and threshold, and whether a user has modified and / or changed any previous and consecutive feasibility scores and thresholds.

[0063] In at least one embodiment, the success assessment program 110a, 110b can identify the likelihood of successfully developing an ML model based on the data associated with each person or event associated with the received original training data set. The success assessment program 110a, 110b can also generate a graph and / or chart that can visualize the different likelihoods of success when developing ML solutions for different groups based on age, gender, nationality, typical diet, and medical history and diseases. For example, if most of the data is for teenagers and there is insufficient data for the elderly, then the likelihood of success in developing a solution for teenagers is high, and the likelihood of success for the elderly may be low.

[0064] Continuing with the previous example, each of the 150 adverse events that meet the threshold level will receive a green meter indicator showing that the calculated feasibility score is greater than the threshold level. However, for each of the 650 adverse events that do not meet the threshold level (or have a feasibility score equal to or less than 11), the user who developed the ML model will receive a red meter indicator showing that the calculated feasibility score is equal to or less than the threshold level.

[0065] At 210, if the success assessment program 110a, 110b determines that the feasibility score does not meet (or does not meet) the threshold, the success assessment program 110a, 110b ends. In at least one embodiment, the success assessment program 110a, 110b can notify the user (e.g., through a dialog box) that the feasibility score does not meet the previously determined threshold level. The success assessment program 110a, 110b can further determine a suggestion (e.g., the user can search for more original training data sets, redefine the learning objectives, or further examine the questions that generated the user provided "no" and / or "don't know" answers). The suggestion can analyze multiple factors (e.g., answers, learning objectives, original training data sets) that can affect the feasibility score and thus the success of the ML solution in solving the related problem. For example, each suggestion can include a hyperlink. If the user clicks on the hyperlink corresponding to the suggestion, the success assessment program 110a, 110b can expand the dialog box to include more details related to the selected hyperlink.

[0066] In at least one embodiment, the learning objective can be changed or redefined based on including fewer categories. For example, when imagining a task of classifying the severity of a disease, where there can be 10 levels of severity, the learning objective can be changed to classify the task into two categories, such as disease of any severity or no disease.

[0067] For example, if the user clicks on a hyperlink corresponding to Search for more original training data sets, the success assessment program 110a, 110b may provide additional sources from which the user may obtain more original training data sets. If the user clicks on a hyperlink corresponding to Redefine learning objectives, the success assessment program 110a, 110b may provide a list of recommended learning objectives for the user to review and select. If the user clicks on a hyperlink corresponding to Check answers to generated questions, the success assessment program 110a, 110b may provide a list of questions to which the user answered "no" and / or "don't know."

[0068] Continuing with the previous example, the success assessment procedures 110a, 110b examine the "no" answered questions corresponding to each of the 650 adverse events whose feasibility scores did not reach the threshold level. The success assessment procedures 110a, 110b further require the secondary extractor to analyze the medical reports associated with these 650 adverse event patients using NLP techniques to identify any data included in the medical reports that could explain the low feasibility scores.

[0069] However, at 210, if the success assessment program 110a, 110b determines that the feasibility score meets the threshold, then the effort level is calculated at 212. The effort level can be based on the estimated human time, domain expertise level, and / or software engineering expertise level to write the rules of the ML model based on the analysis of the original training data set. The user can then determine whether the effort level is high, medium, or low based on the estimated human time. In at least one embodiment, a baseline of estimated human time and / or domain expertise level and software engineering expertise level can be determined by a subject matter expert (SME) to write some rules and metrics and then make inferences.

[0070] In at least one embodiment, the level of effort may be based on the amount of computing tasks and / or resources performed to develop a successful ML solution or to increase the likelihood of success in developing an ML solution.

[0071] In at least one embodiment, the level of effort may be automatically determined based on the importance of developing the ML model and / or the average time to code an ML model in the domain and / or subject matter. The success assessment program 110a, 110b may then determine whether the level of effort is high, medium, or low.

[0072] In one embodiment, the level of effort can be estimated by utilizing a learning approach based on the effort invested in similar ML solutions (i.e., data about previous ML solutions can be utilized to determine the effort to be invested and other characteristics of the current ML solution). The level of effort in developing the ML model can also be based on the number of "no" and / or "don't know" responses to the generated questions.

[0073] In some embodiments, the level of effort for developing the rule-based portion may be given separately based on data labeling and validation by learning methods or human / subject matter experts (SMEs). Thus, the level of effort may be based on developing a rule-based portion for decomposing or classifying a problem to determine if the ML solution may work, or if the ML solution may not work, based on the answer to the problem, subject matter experts and / or humans may be utilized to provide rules for the problem.

[0074] In another embodiment, the success assessment programs 110a, 110b can use a minimum viable product (MVP) approach to estimate the effort level of different dimensions. The MVP approach can guide the user to provide answers to the generated questions. For example, the user can obtain and label a first original training data set, and then compare the original training data set with a second original training data set received by another person or automatically obtained to estimate the expected labeling accuracy of the first original training data set.

[0075] In another embodiment, the success assessment program 110a, 110b can utilize a learning approach, where the dimensions can be viewed as meta-features and classifiers can be trained to determine whether the ML solution is successful or unsuccessful. In some embodiments, the success assessment program 110a, 110b can collect historical data where non-learning parts (e.g., rule-based and logic-based) have been developed. For example, features such as the characteristics of the system can be defined, as well as, for example, the characteristics of the complexity of the rules based on software static analysis. The goal of the learning will be the human time and effort invested, which was available in past examples. The results can be used to better estimate the likelihood of success involving system development. For example, if the rules require a high level of domain expertise and several months of subject matter expert time, it may not be available and may result in a low likelihood of success.

[0076] Continuing with the previous example, a user developing an ML model for a success assessment program 110a, 110b determines 150 adverse events that meet a threshold based on the amount of time it takes to code the rules to develop the 150 ML models using three different SMEs. Depending on the SME, the estimated coding time ranges from 38 hours to 59 hours. The user determines that the level of effort is medium.

[0077] Then, at 214, the results are presented. The success assessment program 110a, 110b may present the user with the calculated feasibility score, the effort estimate (i.e., the calculated effort level), and the system decomposition associated with the compared effort level. In at least one embodiment, the results (e.g., data results) may be presented in a table and / or list. In another embodiment, the results may be presented with a hyperlink that can connect the results to the result source. In one embodiment, the success assessment program 110a, 110b may implement color coding or shading to confirm whether the calculated feasibility score meets a threshold. For example, the success assessment program 110a, 110b highlights the calculated feasibility score that meets the threshold in green, and highlights the calculated feasibility score that fails to meet the threshold in red.

[0078] Continuing with the previous example, the success assessment program 110a, 110b provides a table with a calculated feasibility score, effort estimate, and a systematic breakdown of the effort level associated with the comparison of each of the 800 adverse events in the 600 patients. For each adverse event where the calculated feasibility score fails to meet the threshold level, the success assessment program 110a, 110b displays the calculated feasibility score with light gray shading, and the calculated feasibility scores that meet the threshold level remain unchanged with no shading.

[0079] The success assessment program 110a, 110b can improve the function of the computer because the success assessment program 110a, 110b can assess the likelihood of success related to developing a machine learning (ML) solution before development. In addition, the success assessment program 110a, 110b can assess whether the original training data is suitable for the problem presented, and can associate a value with how well (or effectively) the original training data is suitable for the problem presented. In addition, the success assessment program 110a, 110b can improve the function of the computer regardless of whether the data has been labeled because the success assessment program 110a, 110b considers the decomposition of certain areas that may be suitable for a successful ML solution.

[0080] The success assessment procedures 110a, 110b may determine the chances of developing a successful ML solution by utilizing the feasibility score and effort level before implementing the ML solution.

[0081] Understandably, Figure 2 Only an illustration of one embodiment is provided and does not imply any limitations on how different embodiments may be implemented.Many modifications may be made to the depicted embodiments based on design and implementation requirements.

[0082] Figure 3 According to an illustrative embodiment of the present invention Figure 1 900 is a block diagram of the internal and external components of a computer. It should be understood that Figure 3 This only provides an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications may be made to the depicted environment based on design and implementation requirements.

[0083] Data processing systems 902, 904 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 902, 904 may represent a smartphone, a computer system, a PDA, or other electronic device. Examples of computing systems, environments, and / or configurations that data processing systems 902, 904 may represent include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, microcomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0084] The user client computer 102 and the network server 112 may include Figure 3 Respective internal component sets 902a, b and external components 904a, b are shown. Each internal component set 902a, b includes one or more processors 906, one or more computer-readable ROMs 908 and one or more computer-readable ROMs 912, one or more operating systems 914, and one or more computer-readable tangible storage devices 916. One or more operating systems 914, software programs 108, and success assessment programs 110a in client computers 102 and success assessment programs 110b in network servers 112 can be stored on one or more computer-readable tangible storage devices 916 for execution by one or more processors 906 through one or more RAMs 908 (typically including cache memory). Figure 3 In the illustrated embodiment, each computer-readable tangible storage device 916 is a disk storage device of an internal hard drive. Alternatively, each computer-readable tangible storage device 916 is a semiconductor memory device such as ROM 910, EPROM, flash memory or any other computer-readable tangible storage device that can store computer programs and digital information.

[0085] Each internal component set 902a, b also includes a R / W drive or interface 918 to read and write to one or more portable computer readable tangible storage devices 920, such as CD-ROMs, DVDs, memory sticks, tapes, disks, optical disks, or semiconductor storage devices. Software programs, such as software program 108 and success assessment program 110a, 110b, can be stored on one or more corresponding portable computer readable tangible storage devices 920, read through respective R / W drives or interfaces 918, and loaded into respective hard disk drives 916.

[0086] Each internal component set 902a, b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card or other wired or wireless communication link. The software program 108 and the success assessment program 110a of the client computer 102 and the success assessment program 110b of the network server computer 112 may be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network or other wide area network) and a corresponding network adapter or interface 922. From the network adapter (or switch port adapter) or interface 922, the software program 108 and the success assessment program 110a in the client computer 102 and the success assessment program 110b in the network server computer 112 are loaded into respective hard disk drives 916. The network may include copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0087] Each external component set 904a, b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. The external components 904a, b may also include human-machine interface devices such as a touch screen, a virtual keyboard, a touch pad, a pointing device, etc. Each internal component set 902a, b also includes a device driver 930 to interface to the computer display monitor 924, the keyboard 926, and the computer mouse 928. The device driver 930, the R / W driver or interface 918, and the network adapter or interface 922 include hardware and software (stored in the storage device 916 and / or ROM 910).

[0088] First, it should be understood that although the present disclosure includes a detailed description about cloud computing, the implementation of the technical solutions recorded therein is not limited to a cloud computing environment, but can be implemented in combination with any other type of computing environment now known or later developed.

[0089] Cloud computing is a service delivery model for convenient, on-demand network access to a shared pool of configurable computing resources. Configurable computing resources are resources that can be quickly deployed and released with minimal management cost or interaction with the service provider, such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0090] Features include:

[0091] On-demand self-service: Cloud consumers can unilaterally and automatically deploy computing capabilities such as server time and network storage on demand without human interaction with the service provider.

[0092] Broad network access: Computing power can be accessed over the network through standard mechanisms that facilitate the use of the cloud through different types of thin-client or thick-client platforms (e.g., mobile phones, laptops, personal digital assistants (PDAs)).

[0093] Resource pool: The computing resources of the provider are grouped into resource pools and serve multiple consumers through a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated on demand. Generally, consumers cannot control or even know the exact location of the provided resources, but can specify the location (such as country, state, or data center) at a higher level of abstraction, so it is location-independent.

[0094] Rapid elasticity: The ability to quickly and elastically (sometimes automatically) deploy computing power to scale up quickly, and quickly release it to scale down quickly. To consumers, the computing power available for deployment often appears to be unlimited, and any amount of computing power can be accessed at any time.

[0095] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0096] The service model is as follows:

[0097] Software as a Service (SaaS): The capability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities.

[0098] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers on the cloud infrastructure, which are created using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems or storage, but have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0099] Analytics as a Service (AaaS): The capability provided to the consumer is to access an analytics platform using a web-based or cloud-based network (i.e., infrastructure). The analytics platform may include access to analytics software resources, or may include access to related databases, corpora, servers, operating systems, or storage. The consumer does not manage or control the underlying web-based or cloud-based infrastructure including databases, corpora, servers, operating systems, or storage, but may control the deployed applications and possibly the configuration of the hosting environment.

[0100] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, network, and other basic computing resources on which consumers can deploy and run arbitrary software including operating systems and applications. Consumers neither manage nor control the underlying cloud infrastructure, but have control over the operating system, storage, and applications they deploy, and may have limited control over selected network components (such as host firewalls).

[0101] The deployment model is as follows:

[0102] Private Cloud: Cloud infrastructure is run solely for an organization. The cloud infrastructure can be managed by the organization or a third party and can exist inside or outside the organization.

[0103] Community Cloud: A cloud infrastructure is shared by several organizations and supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). A community cloud can be managed by multiple organizations within the community or by a third party and can exist inside or outside the community.

[0104] Public cloud: Cloud infrastructure is provided to the public or to large industry groups and is owned by an organization that sells cloud services.

[0105] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community or public) in a deployment model that remain distinct entities but are bound together by standardized or proprietary technologies (such as cloud burst sharing for load balancing between clouds) that enable data and application portability.

[0106] The cloud computing environment is service-oriented, with characteristics centered on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0107] Reference now Figure 4, in which an exemplary cloud computing environment 1000 is shown. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 100 with which a local computing device used by a cloud computing consumer can communicate. The local computing device may be, for example, a personal digital assistant (PDA) or a mobile phone 1000A, a desktop computer 1000B, a laptop computer 1000C, and / or an automobile computer system 1000N. The cloud computing nodes 100 can communicate with each other. The cloud computing nodes 100 can be physically or virtually grouped in one or more networks including but not limited to a private cloud, a community cloud, a public cloud, or a hybrid cloud, or a combination thereof as described above (not shown in the figure). In this way, cloud consumers can request infrastructure as a service (IaaS), platform as a service (PaaS), and / or software as a service (SaaS) provided by the cloud computing environment 1000 without having to maintain resources on local computing devices. It should be understood that Figure 4 The various types of computing devices 1000A-N shown are merely illustrative, and the cloud computing node 100 and the cloud computing environment 1000 can communicate with any type of computing device on any type of network and / or network addressable connection (eg, using a web browser).

[0108] Reference now Figure 5 , shows a set of functional abstraction layers 1100 provided by the cloud computing environment 1000. First, it should be understood that Figure 5 The components, layers and functions shown in are only exemplary, and the embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0109] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include: host 1104, server 1106 based on RISC (Reduced Instruction Set Computer) architecture, server 1108, blade server 1110, storage device 1112; network and network components 1114. In some embodiments, software components include network application server software 1116 and database software 1118.

[0110] Virtualization layer 1120 provides an abstraction layer that can provide examples of the following virtual entities: virtual servers 1122 , virtual storage 1124 , virtual networks 1126 (including virtual private networks), virtual applications and operating systems 1128 , and virtual clients 1130 .

[0111] In one example, the management layer 1132 may provide the following functions. Resource provisioning function 1134: Provide dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment; metering and pricing function 1136: Track the cost of resource usage within the cloud computing environment and provide bills and invoices for this purpose. In one example, the resources may include application software licenses. Security function: Provide identity authentication for cloud consumers and tasks, and provide protection for data and other resources. User portal function 1138: Provide access to the cloud computing environment for consumers and system administrators. Service level management function 1140: Provide allocation and management of cloud computing resources to meet the required service levels. Service level agreement (SLA) planning and fulfillment function 1142: Provide pre-scheduling and provisioning for future demand for cloud computing resources predicted according to the SLA.

[0112] The workload layer 1144 provides examples of functions that may be implemented by the cloud computing environment. In this layer, examples of workloads or functions that may be provided include: mapping and navigation 1146, software development and lifecycle management 1148, virtual classroom teaching provision 1150, data analysis processing 1152, transaction processing 1154, and success assessment related to developing ML solutions 1156. The success assessment program 110a, 110b provides a method for developing at least one ML solution by estimating a feasibility score and an effort level.

[0113] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for assessing a likelihood of success associated with developing at least one machine learning (ML) solution, comprising: Multiple adverse events in the original training dataset are identified by the classifier; generating a set of questions based on the original training data set, the set of questions being related to a plurality of defects associated with the original training data set; calculating a feasibility score based on the plurality of adverse events and the responses corresponding to each question in the generated question set; training a machine learning model based on an answer corresponding to each question in the generated question set, wherein the answer includes at least one original training data set related to the corresponding question; In response to determining that the calculated feasibility score satisfies a threshold, calculating a level of effort associated with developing at least one machine learning (ML) solution to solve the problem associated with the plurality of adverse events, Wherein calculating a level of effort associated with developing at least one ML solution to address a problem associated with a plurality of adverse events further comprises: decomposing a problem associated with the at least one ML solution based on a plurality of rule-based portions, wherein the plurality of rule-based portions determine whether the at least one ML solution solves the decomposed problem; classifying at least one ML solution, wherein one or more labels for classifying at least one ML solution include successful or unsuccessful; and A plurality of results associated with assessing the likelihood of success of the at least one ML solution are presented to the user.

2. The method of claim 1, wherein generating a question set based on the original training data set further comprises: Receive the original training data set; identifying one or more best practices from the received raw training data set and identifying one or more deficiencies from the received raw training data set; extracting a generated set of questions from the identified one or more best practices and the identified one or more deficiencies; as well as Aggregate the generated problem set with the stage of developing at least one ML solution.

3. The method of claim 1, further comprising: Retrieve the answer corresponding to each question from the generated question set, Among them, the answer corresponding to each question retrieved from the generated question set is binary.

4. The method of claim 3, further comprising: Assign a numerical value to each retrieved answer corresponding to each question; as well as A feasibility score is calculated based on the sum of the numerical values ​​assigned to the retrieved answers to the generated set of questions.

5. The method of claim 1, further comprising: In response to determining that the calculated feasibility score does not meet the threshold, prompting a user; as well as Identify at least one recommendation to analyze multiple factors related to the calculated feasibility score, Wherein the determined at least one suggestion is selected from a group consisting of searching at least one additional original training data set, redefining one or more learning objectives, and examining each question in the generated question set having a retrieved negative answer.

6. The method of claim 1, wherein the plurality of results presented include a calculated feasibility score, a calculated effort level, and a breakdown associated with the calculated effort level.

7. A computer system for evaluating the likelihood of success associated with developing at least one machine learning (ML) solution, comprising: A memory storing program codes; A processor coupled to the memory, configured to read program code from the memory to execute computer executable instructions in which the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer program product comprising a computer-readable storage medium, the storage medium comprising a computer program code, the computer program code comprising computer-executable instructions, the computer-executable instructions being configured to implement the steps of the method according to any one of claims 1 to 6.

9. An apparatus for evaluating the likelihood of success associated with developing at least one machine learning (ML) solution, comprising one or more modules configured to implement the functions of all steps of the method according to any one of claims 1 to 6.

Images