Automated training of machine learning classification for patient care opportunity misses or delays

By optimizing the training process of the machine learning classifier and removing feature categories with weak predictive power, the accuracy of predicting the presence and lateness of medical patients has been improved, solving the problem of low performance in existing technologies and achieving higher prediction accuracy and resource utilization efficiency.

CN118922892APending Publication Date: 2024-11-08GE PRECISION HEALTHCARE LLC +3
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202380018480.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-01-27
Filing Date
2023-01-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing machine learning classifiers exhibit low recall, precision, and area under the curve fraction when predicting whether medical patients will show up for scheduled medical appointments, and they cannot predict resource waste in the event of patient lateness.

Method used

By receiving and preprocessing annotated data candidates, a machine learning classifier is trained, internal parameters are optimized, feature class importance scores are generated, and weakly predictive feature classes are removed. The classifier is then retrained to improve performance.

Benefits of technology

It significantly improved the recall, precision, and area under the curve fraction of the machine learning classifier from 32%, 29%, and 0.78 to 96%, 49%, and 0.99, respectively, reducing the waste of medical resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922892A_ABST
    Figure CN118922892A_ABST
Patent Text Reader

Abstract

Systems / techniques are provided that facilitate automated training of machine learning classification for patient care opportunity misses or delays. In various embodiments, the system may access a set of annotated data candidates defined by two or more feature categories. In various aspects, the system may train a machine learning classifier according to the set of annotated data candidates, thereby iteratively updating internal parameters of the machine learning classifier. In various cases, the system ranks the two or more feature categories in an order of classification importance based on the iteratively updated internal parameters of the machine learning classifier. In various cases, the system may perform one or more electronic actions based on the two or more feature categories sorted in order of classification importance.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The subject application claims priority to U.S. non-provisional patent application serial number 17 / 585,903, filed on January 27, 2022, entitled “AUTOMATED TRAINING OF MACHINELEARNING CLASSIFICATION FOR PATIENT MISSED CARE OPPORTUNITIES OR LATEARRIVALS,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to machine learning classification, and more particularly to automated training of machine learning classification for missed or delayed patient care opportunities. Background Art

[0004] Medical patients often have scheduled medical appointments with medical professionals. When a patient fails to show up for such a scheduled medical appointment, the medical professional's time and / or resources that have been allocated to handle the scheduled medical appointment may be considered wasted. Therefore, it may be desirable to train a machine learning classifier to predict whether a given patient is likely to not show up for their scheduled medical appointment. Unfortunately, existing machine learning classifiers that are trained to make such predictions often exhibit overly inaccurate performance.

[0005] Therefore, systems and / or techniques that can address one or more of these technical problems may be desirable. Summary of the invention

[0006] The following presents a summary of the invention to provide a basic understanding of one or more embodiments of the invention. The summary of the invention is not intended to identify key or important elements, nor is it intended to delineate any scope of a specific embodiment or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to a more detailed description presented later. In one or more embodiments described herein, an apparatus, system, computer-implemented method, device, and / or computer program product that facilitates automated training of machine learning classifications for missed or delayed patient care opportunities is described.

[0007] According to one or more embodiments, a system is provided. The system may include a computer-readable memory that can store computer-executable components. The system may also include a processor that can be operably connected to the computer-readable memory and can execute the computer-executable components stored in the computer-readable memory. In various embodiments, the computer-executable component may include a receiver component. In various aspects, the receiver component can access a set of annotated data candidates defined by two or more feature categories. In various cases, the computer-executable component may also include a training component. In various cases, the training component can train a machine learning classifier based on the set of annotated data candidates, so that the internal parameters of the machine learning classifier are iteratively updated. In various aspects, the computer-executable component may also include a feature component. In various cases, the feature component can sort the two or more feature categories in the order of classification importance based on the iteratively updated internal parameters of the machine learning classifier. In various cases, the computer-executable component may also include an execution component. In various aspects, the execution component can perform one or more electronic actions based on sorting two or more feature categories in the order of classification importance.

[0008] According to one or more embodiments, the above-described system may be implemented as a computer-implemented method and / or a computer program product. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of an exemplary, non-limiting system that facilitates automated training of machine learning classifications for missed or delayed patient care opportunities in accordance with one or more embodiments described herein is shown.

[0010] Figure 2 An exemplary, non-limiting block diagram of a set of annotated data candidates defined by a set of feature categories according to one or more embodiments described herein is shown.

[0011] Figure 3 A block diagram of an exemplary, non-limiting system that facilitates automated training of machine learning classification for missed or delayed patient care opportunities including a set of updated internal parameters according to one or more embodiments described herein is shown.

[0012] Figure 4 An exemplary, non-limiting block diagram illustrating how a machine learning classifier may be trained is shown according to one or more embodiments described herein.

[0013] Figure 5A block diagram of an exemplary, non-limiting system that facilitates automated training of machine learning classification for missed or delayed patient care opportunities including a set of performance metrics according to one or more embodiments described herein is shown.

[0014] Figure 6 A block diagram of an exemplary, non-limiting system that facilitates automated training of machine learning classification for missed or delayed patient care opportunities including a set of feature class importance scores according to one or more embodiments described herein is shown.

[0015] Figure 7 An exemplary, non-limiting block diagram illustrating a set of feature category importance scores is shown, according to one or more embodiments described herein.

[0016] Figure 8 An exemplary, non-limiting block diagram illustrating how a set of feature category importance scores may be generated is shown in accordance with one or more embodiments described herein.

[0017] Fig. 9 A block diagram of an exemplary, non-limiting system that facilitates automated training of machine learning classification for missed or delayed patient care opportunities including an importance score training dataset according to one or more embodiments described herein is shown.

[0018] Fig.10 An exemplary, non-limiting block diagram illustrating an importance score training dataset is shown, according to one or more embodiments described herein.

[0019] Fig.11 An exemplary, non-limiting block diagram illustrating how an importance score generating machine learning model may be trained based on an importance score training dataset in accordance with one or more embodiments described herein is shown.

[0020] Fig.12 A block diagram of an exemplary, non-limiting system that facilitates automated training of machine learning classification for missed or delayed patient care opportunities including a threshold importance score and a set of updated annotated data candidates according to one or more embodiments described herein is shown.

[0021] Fig.13 An exemplary, non-limiting block diagram of an updated set of annotated data candidates is shown according to one or more embodiments described herein.

[0022] Fig.14 A flow chart is shown of an exemplary, non-limiting computer-implemented method for facilitating automated training of a machine learning classification for missed or delayed patient care opportunities in accordance with one or more embodiments described herein.

[0023] Fig.15 A block diagram illustrating an exemplary, non-limiting operating environment in which one or more embodiments described herein may be facilitated.

[0024] Fig.16 An exemplary networking environment is illustrated that is operable to perform various implementations described herein. DETAILED DESCRIPTION

[0025] The following specific embodiments are merely exemplary and are not intended to limit the application or use of the embodiments and / or the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" or "content of the invention" section or "specific embodiments" section.

[0026] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of illustration, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0027] A medical patient may have a scheduled medical appointment with a medical professional. When a patient fails to show up for such a scheduled medical appointment (e.g., fails to attend the appointment), the medical professional's time and / or resources that have been allocated to handle the scheduled medical appointment may be considered wasted. In fact, such no-shows may be particularly wasteful in radiology practices due to the high fixed costs associated with preparing radiological imaging equipment and / or paying for specially trained medical personnel. Therefore, it may be desirable to train a machine learning classifier to predict whether a given patient is likely to no-show for their scheduled medical appointment so that the wasted time and / or resources can be minimized.

[0028] Unfortunately, existing machine learning classifiers trained to make such predictions often exhibit overly inaccurate performance. Specifically, existing machine learning classifiers trained to predict whether a medical patient is likely to not show up for their scheduled medical appointments (e.g., existing machine learning classifiers trained to classify medical patients as likely to show up or, conversely, as likely not to show up) may exhibit recall rates as low as 32%, precision rates as low as 29%, and / or area under the curve scores as low as 0.78.

[0029] Furthermore, such prior machine learning classifiers only predict the likelihood of showing up versus the likelihood of not showing up for a given medical patient. In other words, such prior machine learning classifiers do not in any way address the fact that even when a medical patient shows up for a scheduled medical appointment, the medical professional's time and / or resources may still be wasted. For example, and as the inventors have recognized, if a medical patient is late for their scheduled medical appointment, the medical professional's time and / or resources may still be wasted by such tardiness, albeit to a lesser extent than if the medical patient did not show up at all. Unfortunately, such prior machine learning classifiers completely ignore this possibility of a medical patient being late for a scheduled medical appointment.

[0030] Therefore, systems and / or techniques that can address one or more of these technical problems may be desirable.

[0031] Various embodiments of the subject innovation can solve one or more of these technical problems. One or more embodiments described herein include systems, computer-implemented methods, devices, and / or computer program products that can facilitate automated training of machine learning classifications for missed or delayed patient care opportunities. In some aspects, the inventors of the various embodiments described herein recognize that existing machine learning classifiers trained to predict the appearance and non-appearance of medical patients suffer from too low performance metrics because such existing machine learning classifiers are often trained based on data containing irrelevant and / or weak feature categories. In other words, the inventors recognize that configuring a machine learning classifier to analyze a set of input features (where one or more of such a set of input features is irrelevant and / or otherwise weakly predictive relative to the desired output classification) can cause the machine learning classifier to have problems and / or be otherwise troubled, resulting in reduced accuracy and / or reduced precision. In addition, the inventors recognize that when such existing machine learning classifiers predict that a particular patient will appear at a particular scheduled medical appointment, such existing machine learning classifiers are not configured and / or trained in any way to predict whether the particular patient may appear late or appear on time at the particular scheduled medical appointment. Therefore, the present inventors have designed various embodiments described herein that can address these shortcomings of the prior art.

[0032] In various aspects, the various embodiments described herein may be viewed as a computerized tool (e.g., any suitable combination of computer-executable hardware and / or computer-executable software) that can facilitate automated training of machine learning classifications for missed or delayed patient care opportunities. In various cases, the computerized tool may include a receiver component, a preprocessing component, a training component, a validation component, a characterization component, and / or an execution component.

[0033] In various embodiments, there may be a set of annotated data candidates and a machine learning classifier that is expected to be trained according to the set of annotated data candidates. In various aspects, the set of annotated data candidates may include any suitable number of annotated data candidates. In various cases, each annotated data candidate may include a data candidate and annotations corresponding to the data candidate respectively. In various cases, each data candidate may be any suitable electronic information having any suitable format and defined by two or more feature categories (e.g., also referred to as attribute categories). That is, the data candidate may be any suitable data piece including text and numbers that includes, conveys and / or otherwise represents two or more features (e.g., one feature per feature category, one attribute per attribute category). As a non-limiting example, it is assumed that for any suitable positive integer l≥2, the data candidate is defined by l feature categories. In such cases, the data candidate may include l features, each of which may be represented by one or more scalars, one or more vectors, one or more matrices, one or more tensors and / or one or more strings. That is, all scalars, vectors, matrices, tensors and / or strings of such l scalars, vectors, matrices, tensors and / or strings may be collectively considered to form a data candidate.

[0034] In various aspects, two or more feature categories may belong to any suitable type of information. As a non-limiting example, two or more feature categories may include any suitable information related to medical professionals and / or medical patients who have scheduled medical appointments with a healthcare facility. For example, two or more feature categories may include: an appointment date category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating a time and / or date planned for a given scheduled medical appointment for a given patient); an assigned date category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating a time and / or date predetermined, set, tentatively scheduled, and / or otherwise assigned to a given scheduled medical appointment for a given patient); a hospitalization type category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating whether a given scheduled medical appointment for a given patient is an outpatient visit); department category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating which specific medical department (such as radiology, endocrinology, dermatology, audiology, obstetrics, and / or surgery) of a healthcare institution will perform a given scheduled medical appointment for a given patient); modality category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating whether a given scheduled medical appointment for a given patient is an outpatient or an inpatient visit); department category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating which specific medical department (such as radiology, endocrinology, dermatology, audiology, obstetrics, and / or surgery) of a healthcare institution will perform a given scheduled medical appointment for a given patient); modality category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating which specific medical equipment modality (such as computed tomography (CT) scanner, X-ray scanner, magnetic resonance imaging (MRI) scanner, ultrasound scanner, and / or positron emission tomography (PET) scanner); age category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the age and / or date of birth of a given patient); education category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the highest level of education and / or the highest level of education obtained by a given patient); employment category (e.g., features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the occupation and / or profession of a given patient); a plurality of scalars, vectors, matrices, tensors, and / or strings); a lesion category (e.g., the features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating an existing lesion, disease, ailment, diagnosis, and / or prognosis for a given patient); an ethnicity category (e.g., the features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating an ethnicity, citizenship, and / or nationality for a given patient); an insurance category (e.g., the features belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating whether and / or through which provider a given patient is provided with health insurance);A residence category (e.g., a feature belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the location of a given patient relative to a healthcare facility scheduled to host a scheduled medical appointment for the given patient and / or the distance at which the given patient resides relative to the healthcare facility); a traffic category (e.g., a feature belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the predicted amount of traffic expected to accumulate around the given patient's residence, around the healthcare facility scheduled to host the scheduled medical appointment, and / or along a route from the residence to the healthcare facility during any suitable time interval prior to the given patient's scheduled medical appointment); and / or a weather category (e.g., a feature belonging to this category may be one or more scalars, vectors, matrices, tensors, and / or strings indicating predicted weather events expected to occur around the given patient's residence, around the healthcare facility scheduled to host the scheduled medical appointment, and / or along a route from the residence to the healthcare facility during any suitable time interval prior to the given patient's scheduled medical appointment). These are merely non-limiting examples of potential feature categories. Thus, one of ordinary skill in the art will appreciate that any other suitable feature categories may be implemented as desired. ;

[0035] In various aspects, each annotation may correspond to a corresponding data candidate and may be a ground truth label (e.g., may be one or more ground truth scalars, vectors, matrices, tensors, and / or strings) indicating a correct and / or accurate classification known and / or believed to correspond to the corresponding data candidate. In various cases, such annotations may belong to any suitable type of classification as desired. As a non-limiting example, when two or more feature categories include information related to medical patients with scheduled medical appointments, each annotation may indicate whether a given medical patient is likely to show up and / or attend his / her scheduled medical appointment. In addition, in some cases, if a particular annotation indicates that a given medical patient is likely to show up and / or attend his / her scheduled medical appointment, the particular annotation may further indicate whether the given medical patient is likely to show up late (e.g., be late) for his / her scheduled medical appointment. Further, in some aspects, if a particular annotation indicates that a given medical patient is likely to show up late for his / her scheduled medical appointment, the particular annotation may further indicate the amount and / or duration of time that the given medical patient is likely to be late or be late for his / her scheduled medical appointment. Thus, in various aspects, possible classifications that an annotation may indicate may be considered as follows: no show, on-time show, and / or late show. As will be appreciated by one of ordinary skill in the art, such annotations may be manually made and / or may be obtained from historical / recorded data.

[0036] In any case, the set of annotated data candidates may include data candidates defined by two or more feature categories, and the set of annotated data candidates may also include annotations respectively corresponding to the data candidates.

[0037] In various aspects, it is expected that the machine learning classifier trained according to the set of annotated data candidates can present any suitable artificial intelligence architecture as needed. For example, the machine learning classifier can present a decision tree architecture (e.g., XGBoost). In such cases, the machine learning classifier may include any suitable number of decision nodes, any suitable decision threshold at such decision nodes, any suitable number of leaf nodes, and / or any suitable classification threshold at such leaf nodes. For example, the machine learning classifier can present a deep learning neural network architecture. In such cases, the machine learning classifier may include any suitable number of layers (e.g., input layer, one or more hidden layers, output layer), may include any suitable number of neurons in various layers (e.g., different layers may have the same and / or different numbers of neurons), may include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, corrected linear unit) in various neurons (e.g., different neurons may have the same and / or different activation functions), and / or may include any suitable inter-neuron connection mode (e.g., forward connection, skip connection, reincarnation connection). In various aspects, the machine learning classifier may represent any other suitable artificial intelligence architecture (e.g., logistic regression, linear regression, naive Bayes, support vector machine) as desired.

[0038] In any case, the machine learning classifier can be configured to receive as input a data candidate from the set of annotated data candidates and produce as output a classification having the same format as the annotations in the set of annotated data candidates. Thus, in various cases, it may be desirable to train a machine learning classifier based on the set of annotated data candidates (e.g., train a machine learning classifier to predict no show vs. show on time vs. show late for a medical patient represented by a given data candidate).

[0039] In various embodiments, a receiver component of a computerized tool may electronically receive and / or otherwise electronically access the set of annotated data candidates and / or machine learning classifiers. In some aspects, the receiver component may electronically retrieve the set of annotated data candidates and / or machine learning classifiers from any suitable centralized and / or decentralized data structure (e.g., a graph data structure, a relational data structure, a hybrid data structure), whether retrieved remotely and / or locally at the receiver component. In any case, the receiver component may electronically obtain and / or access the set of annotated data candidates and / or machine learning classifiers so that other components of the computerized tool may electronically interact with (e.g., read, write, edit, copy, regulate) the set of annotated data candidates and / or machine learning classifiers.

[0040] In various embodiments, the preprocessing component of the computerized tool can apply any suitable preprocessing, classification and / or filtering techniques to the set of annotated data candidates electronically. For example, in various aspects, the preprocessing component electronically reformats the set of annotated data candidates in any suitable manner as needed. For another example, in various cases, the preprocessing component can electronically remove and / or delete feature categories whose features do not show differences in the entire set of annotated data candidates (e.g., for a specific feature category, each data candidate in the set of annotated data candidates can have features belonging to the specific feature category; if the features belonging to the specific feature category are the same in all data candidates, the specific feature category can be deleted). As another example, in various cases, the preprocessing component can electronically remove and / or delete feature categories whose features show considerable null values ​​and / or unavailable values ​​in the entire set of annotated data candidates (e.g., for a specific feature category, each data candidate in the set of annotated data candidates can have features belonging to the specific feature category; if more than any suitable threshold ratio of data candidates have null values ​​and / or unavailable values ​​for a specific feature category, the specific feature category can be deleted). As yet another example, in various aspects, the pre-processing component can electronically remove and / or delete data candidates whose features exhibit a substantial number of null values ​​and / or unusable values ​​(e.g., each data candidate in the set of annotated data candidates can have one feature per feature category; if any data candidate has more than any suitable threshold proportion of features having null values ​​and / or unusable values, the data candidate can be deleted).

[0041] In various embodiments, the training component of the computerized tool can electronically train the machine learning classifier according to the group of annotated data candidates (e.g., after the group of annotated data candidates has been preprocessed by the preprocessing component). For example, it is assumed that the machine learning classifier is a decision tree model. In such cases, the training component can fit the machine learning classifier to the group of annotated data candidates via any suitable sample segmentation technique (e.g., segmenting the data candidates according to the annotation based on the estimate of positive correctness, segmenting the data candidates according to the annotation based on the Gini impurity, segmenting the data candidates according to the annotation based on the information gain, segmenting the data candidates according to the annotation based on the variance reduction, segmenting the data candidates according to the annotation based on the measure of "goodness". In addition, in various cases, the training component can perform any suitable pruning technique (e.g., reducing error pruning, cost complexity pruning) on ​​the machine learning classifier after such sample segmentation. In any case, the final result can be that the machine learning classifier now has internal parameters (e.g., decision node position, decision node threshold, leaf node position and / or leaf node threshold), which have been optimized to accurately classify the input data candidates.

[0042] As another example, suppose that the machine learning classifier is changed to a deep learning neural network. In such cases, the internal parameters (e.g., weights, biases) of the machine learning classifier can be randomly initialized. In various aspects, the training component can select a data candidate and an annotation corresponding to the data candidate from the set of annotated data candidates. In various cases, the training component can feed the selected data candidate as an input to the machine learning classifier, which will cause the machine learning classifier to produce a certain output. More specifically, in various cases, the input layer of the machine learning classifier can receive the selected data candidate, the selected data candidate can complete the forward transfer through one or more hidden layers of the machine learning classifier, and the output layer of the machine learning classifier can calculate the output based on the activation provided by one or more hidden layers of the machine learning classifier. In various cases, the output can be regarded as the inferred classification that the machine learning classifier believes should correspond to the selected data candidate, and the selected annotation can be regarded as the basic fact classification known to correspond to the selected data candidate. It should be noted that if the machine learning classifier has not been trained so far and / or has undergone little training, the output may be extremely inaccurate (e.g., the output may be very different from the selected annotation). In any case, the training component can calculate the error and / or loss between the output and the selected annotation, and the training component can update the internal parameters of the machine learning classifier by performing back propagation based on the calculated error and / or loss. In various cases, the training component can repeat the training process for each (and / or in some cases fewer) data candidate in the set of annotated data candidates, with the end result that the internal parameters (e.g., weights, biases) of the machine learning classifier can be iteratively optimized to accurately classify the input data candidates. Those of ordinary skill in the art will appreciate that any suitable training batch size, any suitable training termination criteria, and / or any suitable error / loss function can be implemented by the training component as desired.

[0043] In any case, the training component can electronically train the machine learning classifier based on the set of annotated data candidates, with the result that internal parameters (e.g., weights and / or biases for a neural network; node positions and / or node thresholds for a decision tree model; regression coefficients for a regression model) can be updated and / or optimized to accurately classify the input data candidates. In other words, regardless of the specific structure and / or architecture of the machine learning classifier, the machine learning classifier can have configurable internal parameters that are updated and / or optimized during training.

[0044] In various embodiments, the verification component of the computerized tool can electronically calculate one or more performance metrics of the machine learning classifier after the training component completes the training. In some cases, the training component can avoid training the machine learning classifier according to any suitable subset of the group of annotated data candidates. Therefore, the verification component can use such a subset to test and / or verify the machine learning classifier. As will be understood by those of ordinary skill in the art, in various cases, one or more performance metrics may include the accuracy grade of the machine learning classifier, the precision grade of the machine learning classifier, the recall grade of the machine learning classifier, the area under the curve grade of the machine learning classifier, the true positive rate of the machine learning classifier, the false positive rate of the machine learning classifier, the true negative rate of the machine learning classifier, the false negative rate of the machine learning classifier, and / or any other suitable desired performance metric.

[0045] In various embodiments, the feature component of the computerized tool can generate a set of feature category importance scores electronically based on the internal parameters of the update / optimization of the machine learning classifier. In various aspects, the set of feature category importance scores can include an importance score for each feature category. For example, if two or more feature categories include a total of l categories (for example, the first feature category to the lth feature category) for any suitable positive integer l≥2, then the set of feature category importance scores can include l scores (for example, the first importance score to the lth importance score). In various cases, each importance score can be a scalar (and / or vector, matrix, tensor and / or string in other cases) representing how important the corresponding feature category is relative to the desired classification. For example, it is assumed that a specific importance score corresponds to a specific feature category. If a specific importance score is small (for example, if the magnitude of a specific importance score is less than any suitable threshold), this can indicate and / or convey that a specific feature category is not very important relative to the desired classification (for example, it can be indicated that the predictability of a specific feature category to the desired classification is weak). On the other hand, if a particular importance score is large (e.g., if the magnitude of the particular importance score is greater than any suitable threshold), this may indicate and / or communicate that a particular feature class is important relative to the desired classification (e.g., may indicate that the particular feature class is highly predictive of the desired classification). By way of non-limiting example only, the importance score may be a real scalar whose value ranges between 0 and 1, where 0 indicates weak predictability relative to the desired classification, and where 1 indicates strong predictability relative to the desired classification. However, in other cases, 0 may indicate strong predictability, while 1 may indicate weak predictability. In still other cases, a different range of values ​​may be implemented (e.g., the value and / or magnitude of the importance score may range from any suitable first real value to any suitable second real value). As another non-limiting example, the importance score may be a positive integer value indicating a ranking between two or more feature classes. In some cases, for any suitable positive integer q, a rank and / or importance score of 1 may be considered to indicate the feature class with the strongest / important predictive power relative to the desired classification, a rank and / or importance score of 2 may be considered to indicate the feature class with the second strongest / important predictive power relative to the desired classification, and / or a rank and / or importance score of q may be considered to indicate the feature class with the qth strongest / important predictive power relative to the desired classification.

[0046] In any case, the feature component can electronically calculate the group of feature category importance scores based on the updated / optimized internal parameters of the machine learning classifier. As a non-limiting example, in some cases, the feature component can implement the calculation in an analytical manner. For example, if the machine learning classifier is a decision tree model, each decision node of the machine learning classifier can correspond to a corresponding feature category in two or more feature categories (for example, each decision node can represent a decision split based on a corresponding feature category). Therefore, for any given decision node in the machine learning classifier, the importance score of the decision node can be calculated based on the position of the decision node. For example, a specific feature category corresponding to the root node (for example, the first decision node) of the machine learning classifier can be regarded as the most important and / or the most predictive of two or more feature categories. Therefore, the feature component can assign appropriate importance scores to the specific feature categories corresponding to the root node (for example, the specific feature categories corresponding to the root node can be given the best importance scores and / or rankings). In addition, for any other given decision nodes in the machine learning classifier, importance scores and / or rankings can be assigned to the feature categories corresponding to the given decision node based on the distance between the given decision node and the root node. In other words, decision nodes that are closer to the root node (e.g., decision nodes that are higher in the decision tree) can be considered more important and / or more predictive than decision nodes that are farther from the root node (e.g., decision nodes that are lower in the decision tree).

[0047] As another non-limiting example, the feature component can calculate the set of feature category importance scores via the execution of artificial intelligence, rather than implementing an analysis technique. For example, in various cases, the feature component can electronically store, electronically maintain, electronically control, and / or otherwise electronically access the importance score generation machine learning model. In various aspects, the importance score generation machine learning model can present any suitable artificial intelligence architecture as needed. For example, the importance score generation machine learning model can present a deep learning neural network architecture. In such cases, the importance score generation machine learning model may include any suitable number of layers (e.g., input layer, one or more hidden layers, output layer), may include any suitable number of neurons in various layers (e.g., different layers may have the same and / or different numbers of neurons), may include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, corrected linear unit) in various neurons (e.g., different neurons may have the same and / or different activation functions), and / or may include any suitable inter-neuron connection pattern (e.g., forward connection, skip connection, recurrent connection). In any case, the importance score generating machine learning model can be configured to receive updated / optimized internal parameters of the machine learning classifier and / or indications of two or more feature categories as input, and the importance score generating machine learning model can be further configured to generate the set of feature category importance scores as output. More specifically, there can be one or more first scalars, vectors, matrices, tensors, and / or strings representing the updated / optimized internal parameters of the machine learning classifier. In addition, there can be one or more second scalars, vectors, matrices, tensors, and / or strings representing two or more feature categories. In various aspects, one or more first scalars, vectors, matrices, tensors, and / or strings can be cascaded with one or more second scalars, vectors, matrices, and / or strings, such cascades can be received by the input layer of the importance score generating machine learning model, such cascades can complete forward propagation through one or more hidden layers of the importance score generating machine learning model, and the output layer of the importance score generating machine learning model can calculate the set of feature category importance scores based on the activations provided by one or more hidden layers of the importance score generating machine learning model. In various aspects, as explained in more detail herein, such importance score generating machine learning models may be trained in any suitable manner in order to accurately infer feature category importance scores based on the input internal parameters of the machine learning classifier.

[0048] Thus, in various aspects, the feature component can electronically execute the importance score generating machine learning model based on the updated / optimized internal parameters to produce the set of feature category importance scores.

[0049] In various embodiments, the execution component of the computerized tool can electronically initiate any suitable electronic action based on the set of feature category importance scores. As a non-limiting example, the execution component can electronically transmit one or more feature category importance scores in the set of feature category importance scores to any suitable computing device as needed. As another non-limiting example, the execution component can electronically present one or more feature category importance scores in the set of feature category importance scores on any suitable computer screen, display and / or monitor. As another non-limiting example, the execution component can electronically identify which feature category importance scores (if any) in the set of feature category importance scores meet any suitable threshold value, can electronically identify feature categories corresponding to such identified importance scores and / or can electronically transmit indications of such identified feature categories to any suitable computing device. As yet another non-limiting example, the execution component may electronically identify which feature class importance scores, if any, of the set of feature class importance scores satisfy any suitable threshold, may electronically identify feature classes corresponding to such identified importance scores, and / or may electronically present an indication of such identified feature classes on any suitable computer screen, display, and / or monitor. As another non-limiting example, the execution component may electronically identify which feature class importance scores, if any, of the set of feature class importance scores fail to satisfy any suitable threshold, may electronically identify feature classes corresponding to such identified importance scores, and / or may electronically transmit an indication of such identified feature classes to any suitable computing device. As another non-limiting example, the execution component may electronically identify which feature class importance scores, if any, of the set of feature class importance scores fail to satisfy any suitable threshold, may electronically identify feature classes corresponding to such identified importance scores, and / or may electronically present an indication of such identified feature classes on any suitable computer screen, display, and / or monitor.

[0050] In some cases, when the execution component identifies a feature category whose importance score fails to meet any suitable threshold, the execution component may electronically delete, remove, erase and / or otherwise eliminate such identified feature categories from the set of annotated data candidates. In other words, the execution component may determine which feature categories are considered to be weakly predictive of the desired classification, and the execution component may remove such weakly predictive feature categories from the set of annotated data candidates accordingly. For example, if the i-th feature category is considered to be too weakly predictive for any suitable positive integer i, the i-th feature category may be removed from the set of annotated data candidates (e.g., the i-th feature of each data candidate may be deleted). In such cases, the execution component may be considered to generate a set of updated annotated data candidates. In various aspects, the training component may reconfigure and / or retrain the machine learning classifier based on the set of updated annotated data candidates. As verified by the inventors experimentally, retraining the machine learning classifier after the weakly predictive and / or irrelevant feature categories have been removed from the set of annotated data candidates results in the machine learning classifier achieving better performance metrics (e.g., achieving better accuracy, better precision, better area under the curve). After such retraining, the execution component can electronically deploy the machine learning classifier in any suitable operational context (e.g., the retrained machine learning classifier can be electronically executed according to any suitable data candidate encountered in the field).

[0051] Thus, various embodiments described herein include computerized tools that facilitate automated training of machine learning classifiers for missed or delayed patient care opportunities. More specifically, it may be desirable to train a machine learning classifier to classify medical patients with scheduled medical appointments as no-shows, on-time shows, or late shows. In various cases, the computerized tool may train the machine learning classifier based on a set of annotated data candidates defined by two or more feature categories, the computerized tool may rank the two or more feature categories based on classification importance and / or predictive relevance, the computerized tool may remove and / or delete feature categories with poor predictiveness from the set of annotated data candidates, and the computerized tool may retrain the machine learning classifier based on the set of annotated data candidates after such removal / deletion of feature categories with poor predictiveness. The end result may be an improvement in the performance of the machine learning classifier.

[0052] Various embodiments of the subject innovations may be used to solve problems that are highly technical in nature (e.g., to facilitate automated training of machine learning classifications for missed or delayed patient care opportunities) using hardware and / or software that are not abstract and cannot be performed as a set of psychological behaviors of a human. Further, some of the processes performed may be performed by a dedicated computer (e.g., a machine learning classifier, a decision tree model, a deep learning neural network) for performing defined tasks associated with automated training of machine learning classifications for missed or delayed patient care opportunities. For example, such defined tasks may include: accessing, by a device operatively coupled to a processor, a set of annotated data candidates defined by two or more feature categories; training, by the device, a machine learning classifier based on the set of annotated data candidates, thereby iteratively updating internal parameters of the machine learning classifier; sorting, by the device, two or more feature categories in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; and performing, by the device, one or more electronic actions based on the two or more feature categories sorted in order of classification importance. In various cases, the one or more electronic actions may include: identifying, by the device, which of the two or more feature categories are ranked below a threshold; removing, by the device, such feature categories from the set of annotated data candidates, thereby generating an updated set of annotated data candidates; and retraining, by the device, a machine learning classifier based on the updated set of annotated data candidates.

[0053] Such defined tasks are not performed manually by humans. In fact, neither the human mind nor a human with a pen and paper can electronically receive training data defined by various attribute categories, electronically train a machine learning classifier based on the training data (e.g., via backpropagation and / or sample segmentation), electronically rank the attribute categories based on the trained, updated, and / or optimized internal parameters of the machine learning classifier, electronically delete attribute categories ranked below a threshold from the training data, and / or electronically retrain the machine learning classifier based on the training data after such deletion. In contrast, various embodiments of the subject innovations are inherently and unavoidably dependent on computer technology, and cannot be implemented outside of a computing environment (e.g., machine learning classifiers such as decision tree classifiers and / or neural network classifiers are inherently computerized constructs that cannot be implemented in any way simply by the human mind without a computer; thus, a computerized tool for training a machine learning classifier based on a data set, determining which features of the data set are weakly predictive, removing such weak features from the data set, and retraining the machine learning classifier based on the data set after such removal is likewise inherently computerized and cannot be implemented in any sensible, practical, or reasonable manner without a computer).

[0054] In addition, various embodiments of the subject innovation can integrate various teachings described herein into practical applications, which are related to automated training of machine learning classification for missed or delayed patient care opportunities. As described above, the prior art can cause machine learning classifiers to achieve relatively low performance metrics. The inventors realize that such low performance metrics are usually caused by the prior art training machine learning classifiers based on irrelevant and / or weakly predictive features (e.g., such machine learning classifiers may be troubled by irrelevant and / or weakly predictive features). Therefore, the inventors have designed various embodiments described herein. Specifically, the various embodiments described herein can train machine learning classifiers based on the set of annotated data candidates. In addition, after such training, the various embodiments described herein can analyze (e.g., analytically and / or via artificial intelligence) the trained, updated and / or optimized internal parameters of the machine learning classifier so as to sort the feature categories defining the set of annotated data candidates in order of classification importance. In various cases, the various embodiments described herein can delete any feature category whose ranking (e.g., its classification importance score) fails to meet any suitable threshold from the set of annotated data candidates. Thus, the various embodiments described herein may retrain a machine learning classifier based on the set of annotated data candidates after such deletion, which may result in the machine learning classifier achieving significantly improved performance (e.g., poorly predictive features may no longer be present to plague and / or cause problems for the machine learning classifier). In fact, during their experiments (e.g., using a "test" dataset and a separate "production" dataset), the inventors implemented the embodiments described herein: after initial training, the machine learning classifier exhibited a recall rate of 32%, a precision rate of 29%, and an area under the curve score of 0.78; however, after deleting the poorly predictive feature classes and retraining, the machine learning classifier exhibited a recall rate of 96%, a precision rate of 49%, and an area under the curve score of 0.99. Clearly, the various embodiments described herein constitute concrete and tangible technical improvements in the field of machine learning classification, and therefore such embodiments certainly qualify as useful and practical applications of computers.

[0055] In addition, various embodiments of the subject innovation can control real-world tangible devices based on the disclosed teachings. For example, various embodiments of the subject innovation can electronically train, retrain and / or execute real-world machine learning models (e.g., decision trees, neural networks).

[0056] It should be understood that the drawings and description herein provide non-limiting examples of the subject innovation and are not necessarily drawn to scale.

[0057] Figure 1A block diagram of an exemplary, non-limiting system 100 that can facilitate automated training of machine learning classifiers for missed or delayed patient care opportunities in accordance with one or more embodiments described herein is shown. As shown, an automated classifier training system 102 can be electronically integrated with a set of annotated data candidates 104 and / or with a machine learning classifier 122 via any suitable wired and / or wireless electronic connection.

[0058] In various embodiments, the set of annotated data candidates 104 may include any suitable number of data candidates. In various aspects, each data candidate may be viewed as a set of features that respectively correspond to a certain basic fact classification annotation. Figure 2 This is shown in more detail.

[0059] Figure 2 An exemplary, non-limiting block diagram 200 of a set of annotated data candidates defined by a set of feature categories according to one or more embodiments described herein is shown. That is, Figure 2 A non-limiting, exemplary embodiment of the set of annotated data candidates 104 is shown.

[0060] As shown, the set of annotated data candidates 104 may include a set of data candidates 202 , a set of annotations 204 corresponding to the set of data candidates 202 , and / or a set of feature categories 206 that collectively define the content of each data candidate in the set of data candidates 202 .

[0061] In various aspects, as shown, the set of data candidates 202 may include n candidates for any suitable positive integer n: data candidate 1 to data candidate n. In various cases, each data candidate in the set of data candidates 202 may represent a different sample of data. As a non-limiting example, each data candidate in the set of data candidates 202 may represent a corresponding medical patient who has a scheduled medical appointment with a medical professional and / or with a healthcare institution.

[0062] As shown, each data candidate in the set of data candidates 202 can be defined by the set of feature categories 206. In various cases, the set of feature categories 206 can include m categories: feature category 1 to feature category m for any suitable positive integer m≥2. In various cases, the feature category can be any suitable type, kind, class, and / or category of measurable information (e.g., measurable features and / or attributes) desired, and the data candidate can be considered as a collection of features / attributes that respectively belong to the set of feature categories 206.

[0063] For example, if each data candidate in the set of data candidates 202 represents a corresponding medical patient having a scheduled medical appointment with a medical professional and / or with a healthcare facility, each feature category in the set of feature categories 206 may be a certain type / kind of measurable characteristic of a medical patient having a scheduled medical appointment. For example, a feature category may be an appointment date category, and features / attributes belonging to such a category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate a time and / or date when a given scheduled medical appointment is planned and / or scheduled for a given patient.

[0064] As another example, a feature category may be an assigned date category, and features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate a time and / or date that is predetermined, set, tentatively scheduled, and / or otherwise assigned to a scheduled medical appointment for a given patient.

[0065] As yet another example, a feature category may be a hospitalization type category, and the features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate whether a given scheduled medical appointment for a given patient is an outpatient visit or an inpatient visit.

[0066] As yet another example, a feature category may be a department category, and the features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate which specific medical department of a healthcare institution (such as radiology, endocrinology, dermatology, audiology, obstetrics, and / or surgery) will perform a given scheduled medical appointment for a given patient.

[0067] As another example, a feature category may be a modality category, and a feature / attribute belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate which specific medical equipment modality (such as a CT scanner, X-ray scanner, MRI scanner, ultrasound scanner, and / or PET scanner) will be used during a given scheduled medical appointment for a given patient.

[0068] As another example, a feature category may be an age category, and features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof indicating the age and / or date of birth of a given patient.

[0069] As yet another example, a feature category may be an education category, and the features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof indicating the highest level of education and / or the highest degree of education obtained by a given patient.

[0070] As yet another example, a feature category may be an occupation category, and the features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate the occupation and / or profession of a given patient.

[0071] As another example, a feature category may be a pathology category, and the features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate an existing pathology, disease, ailment, diagnosis, and / or prognosis for a given patient.

[0072] As another example, a feature category may be an ethnicity category, and features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate the ethnicity, citizenship, and / or nationality of a given patient.

[0073] As yet another example, a feature category may be an insurance category, and features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that indicate whether and / or through which provider a given patient is provided with health insurance.

[0074] As yet another example, a feature category may be a residence category, and the features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof indicating the location at which a given patient resides relative to a healthcare facility that is scheduled to host the given patient's scheduled medical appointments and / or indicating the distance at which the given patient resides from the healthcare facility.

[0075] As another example, a feature category may be a traffic category, and features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof indicating the predicted amount of traffic expected to accumulate around a given patient's residence, around a healthcare facility scheduled to host the scheduled medical appointment, and / or along a route from the residence to the healthcare facility during any suitable time window prior to the given patient's scheduled medical appointment.

[0076] As yet another example, a feature category may be a weather category, and features / attributes belonging to such category may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof indicating predicted weather events expected to occur around a given patient's residence, around a healthcare facility scheduled to host the scheduled medical appointment, and / or along a route from the residence to the healthcare facility during any suitable time interval prior to the given patient's scheduled medical appointment.

[0077] These are merely non-limiting examples of potential feature categories. One of ordinary skill in the art will appreciate that any other suitable feature categories may be implemented as desired in various circumstances.

[0078] Because the set of feature categories 206 may include m categories, each data candidate in the set of data candidates 202 may include m features. For example, data candidate 1 may include m different features: feature 1(1) to feature 1(m), where feature 1(1) may belong to feature category 1, and where feature 1(m) may belong to feature category m. Similarly, data candidate n may include m different features: feature n(1) to feature n(m), where feature n(1) may belong to feature category 1, and where feature n(m) may belong to feature category m. Therefore, assume that data candidate 1 represents a specific medical patient with a specific scheduled medical appointment, assume that data candidate n represents different medical patients with different scheduled medical appointments, assume that feature category 1 is a patient age category, and assume that feature category m is an appointment date category. In such cases, feature 1(1) may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the age and / or date of birth of a particular medical patient, feature 1(m) may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the time and / or date at which a particular scheduled medical appointment is scheduled to occur, feature n(1) may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the ages and / or dates of birth of different medical patients, and feature n(m) may be one or more scalars, vectors, matrices, tensors, and / or strings indicating the times and / or dates at which different scheduled medical appointments are scheduled to occur.

[0079] In various aspects, the set of annotations 204 can correspond to the set of data candidates 202, respectively. Therefore, because the set of data candidates 202 can include n candidates, the set of annotations 204 can include n annotations: annotation 1 to annotation n. In various cases, an annotation can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof that represent and / or indicate a basic fact classification label that is known and / or otherwise considered to correspond to the corresponding data candidate. For example, data candidate 1 can correspond to annotation 1, which means that annotation 1 can be regarded as the basic fact classification label for data candidate 1. Similarly, data candidate n can correspond to annotation n, which means that annotation n can be regarded as the basic fact classification label for data candidate n.

[0080] In various cases, any suitable classification may be represented by the set of annotations 204. As a non-limiting example, when each data candidate in the set of data candidates 202 represents a respective medical patient with a scheduled medical appointment, the classification represented by the set of annotations 204 may be whether the respective medical patient is likely to arrive on time, arrive late, and / or not arrive at all for their scheduled medical appointment. In such cases, annotation 1 may indicate whether the medical patient represented by data candidate 1 is likely to show up on time, likely to show up late, and / or may not show up at all for his / her scheduled medical appointment. Similarly, annotation n may indicate whether the medical patient represented by data candidate n is likely to show up on time, likely to show up late, and / or may not show up at all for his / her scheduled medical appointment. In some aspects, if the annotation indicates that the medical patient is likely to show up late for their scheduled medical appointment, the annotation may further indicate the amount of time (e.g., measured in minutes and / or hours) that the medical patient is likely to be late.

[0081] As will be appreciated by one of ordinary skill in the art, in some cases the set of annotated data candidates 104 may be obtained and / or derived from historical data regarding real-world patients visiting real-world healthcare facilities.

[0082] Back to Figure 1 , the machine learning classifier 122 may present any suitable type of artificial intelligence classification architecture as needed. As a non-limiting example, the machine learning classifier 122 may present a decision tree architecture. In such cases, the machine learning classifier 122 may include any suitable number and / or arrangement of decision nodes, any suitable decision threshold percentage in such decision nodes, any suitable number and / or arrangement of leaf nodes, and / or any suitable classification threshold percentage in such leaf nodes. As another non-limiting example, the machine learning classifier 122 may present a neural network architecture. In such cases, the machine learning classifier 122 may include any suitable number of layers, any suitable number of neurons in each layer, any suitable activation function in each neuron, and / or any suitable inter-neuron connection. As will be appreciated by those of ordinary skill in the art, the machine learning classifier 122 may present any other suitable architecture as needed, such as a support vector machine architecture, a linear and / or logistic regression architecture, and / or a naive Bayes architecture.

[0083] In any case, it may be desirable to train a machine learning classifier 122 based on the set of annotated data candidates 104. As described herein, an automated classifier training system 102 can facilitate such training.

[0084] In various embodiments, the automated classifier training system 102 may include a processor 106 (e.g., a computer processing unit, a microprocessor) and a computer readable memory 108 operatively and / or operatively and / or communicatively connected / coupled to the processor 106. The computer readable memory 108 may store computer executable instructions that, when executed by the processor 106, cause the processor 106 and / or other components (e.g., a receiver component 110, a preprocessing component 112, a training component 114, a verification component 116, a feature component 118, an execution component 120) of the automated classifier training system 102 to perform one or more actions. In various embodiments, the computer readable memory 108 may store computer executable components (e.g., a receiver component 110, a preprocessing component 112, a training component 114, a verification component 116, a feature component 118, an execution component 120), and the processor 106 may execute the computer executable components.

[0085] In various embodiments, the automated classifier training system 102 may include a receiver component 110. In various aspects, the receiver component 110 may electronically receive and / or otherwise electronically access the set of annotated data candidates 104 and / or the machine learning classifier 122. In various cases, the receiver component 110 may electronically retrieve the set of annotated data candidates 104 and / or the machine learning classifier 122 from any suitable centralized and / or decentralized data structure (not shown). In any case, the receiver component 110 may electronically obtain the set of annotated data candidates 104 and / or the machine learning classifier 122 so that other components of the automated classifier training system 102 can electronically interact with the annotated data candidates 104 and / or with the machine learning classifier 122.

[0086] In various embodiments, the automated classifier training system 102 may include a pre-processing component 112. In various aspects, the pre-processing component 112 may electronically apply any suitable formatting changes, any suitable filtering changes, and / or any suitable classification changes to the set of annotated data candidates 104. As a non-limiting example, in various cases, the pre-processing component 112 may electronically remove, erase, and / or delete any feature categories that are uniform and / or otherwise non-different from the set of feature categories 206, and therefore from the set of annotated data candidates 104. For example, assume that for any suitable positive integer 1≤i≤m, feature category i does not exhibit differences. In other words, the i-th feature of each data candidate in the set of data candidates 202 may all be the same (e.g., may not all change). In such cases, the i-th feature may be considered to provide no information that can be used to successfully classify the data candidate. Therefore, the pre-processing component 112 may electronically delete the i-th feature category from the set of feature categories 206 (e.g., the i-th feature of each data candidate in the set of data candidates 202 may be electronically removed and / or eliminated).

[0087] As another non-limiting example, in various aspects, the pre-processing component 112 can electronically remove, erase, and / or delete any feature category that has a significant number of null values ​​and / or unavailable values ​​from the set of feature categories 206, and therefore from the set of annotated data candidates 104. For example, assume that for any suitable positive integer 1≤j≤m, feature category j exhibits null values ​​and / or unavailable values ​​greater than a threshold ratio. In other words, the ratio of the number of data candidates in the set of data candidates 202 that have the jth null feature and / or the jth unavailable feature (e.g., the numerator, which can be less than or equal to n) to the total number of data candidates in the set of data candidates 202 (e.g., the denominator, which can be equal to n) can be greater than the threshold ratio. In such cases, the jth feature category can be considered to have too many null values ​​and / or unavailable values. Therefore, the pre-processing component 112 can electronically delete the jth feature category from the set of feature categories 206 (e.g., the jth feature of each data candidate in the set of data candidates 202 can be electronically removed and / or eliminated).

[0088] As yet another non-limiting example, in various cases, the pre-processing component 112 may electronically remove, erase, and / or delete any data candidate having a substantial number of empty features and / or unavailable features from the set of data candidates 202, and therefore from the set of annotated data candidates 104. For example, assume that for any suitable positive integer 1≤k≤n, the kth data candidate exhibits a greater than a threshold ratio of empty features and / or unavailable features. In other words, the ratio of the number of features having empty and / or unavailable values ​​in the kth data candidate (e.g., the numerator, which may be less than or equal to m) to the total number of features in the kth data candidate (e.g., the denominator, which may be m) may be greater than a threshold ratio. In such cases, the kth data candidate may be deemed to have too many empty features and / or unavailable features. Therefore, the pre-processing component 112 may electronically delete the kth data candidate from the set of data candidates 202.

[0089] In various embodiments, the automated classifier training system 102 can include a training component 114. In various aspects, the training component 114 can electronically train a machine learning classifier 122 based on the set of annotated data candidates 104, as described herein.

[0090] In various embodiments, the automated classifier training system 102 can include a validation component 116. In various cases, the validation component 116 can electronically calculate one or more performance metrics of the machine learning classifier 122, as described herein.

[0091] In various embodiments, the automated classifier training system 102 can include a feature component 118. In various cases, the feature component 118 can electronically sort the set of feature categories 206 in order of their classification importance, as described herein.

[0092] In various embodiments, the automated classifier training system 102 can include an execution component 120. In various aspects, the execution component 120 can electronically execute any suitable electronic action based on ranking the set of feature categories 206 in order of classification importance, as described herein.

[0093] Figure 3 A block diagram of an exemplary, non-limiting system 300 that can facilitate automated training of machine learning classification for missed or delayed patient care opportunities including a set of updated internal parameters according to one or more embodiments described herein is shown. As shown, in some cases, system 300 can include the same components as system 100, and can also include a set of updated internal parameters 302.

[0094] In various embodiments, the training component 114 can electronically train the machine learning classifier 122 based on the set of annotated data candidates 104 (e.g., after the set of annotated data candidates 104 has been formatted, classified, and / or filtered by the pre-processing component 112). In various cases, such training can produce updated internal parameters 302. More specifically, regardless of the specific artificial intelligence architecture of the machine learning classifier 122, the machine learning classifier can have internal parameters that govern how the machine learning classifier 122 determines and / or infers classifications. For example, if the machine learning classifier 122 exhibits a decision tree architecture, the internal parameters of the machine learning classifier 122 can be considered as decision node locations, decision node thresholds, leaf node locations, and / or leaf node thresholds of the machine learning classifier 122. For another example, if the machine learning classifier 122 exhibits a neural network architecture, the internal parameters of the machine learning classifier 122 can be considered as weight matrices and / or bias values ​​of neurons of the machine learning classifier 122. In any case, the machine learning classifier 122 may have internal parameters, and the internal parameters of the machine learning classifier 122 may be updated and / or optimized via training so that the machine learning classifier 122 may accurately classify the input data candidates. The updated internal parameters 302 may be considered as the values ​​of the internal parameters of the machine learning classifier 122 after the training component 114 completes the training.

[0095] As one of ordinary skill in the art will appreciate, the type of training that the machine learning classifier 122 undergoes may depend on the artificial intelligence architecture of the machine learning classifier 122 .

[0096] As a non-limiting example, assume that the machine learning classifier 122 exhibits a decision tree architecture, such as XGBoost. In such cases, the training component 114 can electronically fit the machine learning classifier 122 to the set of annotated data candidates 104 by applying any suitable sample segmentation technique. For example, in some cases, the training component 114 can segment the set of annotated data candidates 104 according to annotations based on estimates of positive correctness. In other cases, the training component 114 can segment the set of annotated data candidates 104 according to annotations based on Gini impurity. In other cases, the training component 114 can segment the set of annotated data candidates 104 according to annotations based on information gain. In other cases, the training component 114 can segment the set of annotated data candidates 104 according to annotations based on variance reduction. In still other cases, the training component 114 can segment the set of annotated data candidates 104 according to annotations based on a measure of "goodness". Furthermore, in various aspects, the training component 114 can perform any suitable pruning techniques, such as error reduction pruning and / or cost complexity pruning, on the machine learning classifier 122. In any case, the end result can be that the machine learning classifier now has updated internal parameters 302 (e.g., the end result can be that the decision node positions, decision node thresholds, leaf node positions, and / or leaf node thresholds have been optimized to accurately classify the input data candidates).

[0097] As another non-limiting example, assume that the machine learning classifier is instead a deep learning neural network. In such a case, the training component 114 can train the machine learning classifier 122 based on the set of annotated data candidates 104 in a supervised manner. Figure 4 This is shown in more detail.

[0098] Figure 4 An exemplary, non-limiting block diagram 400 illustrating how a machine learning classifier 122 may be trained is shown according to one or more embodiments described herein. In various aspects, the training component 114 may randomly initialize (and / or may initialize in any other suitable manner) internal parameters (e.g., weights, biases) of the machine learning classifier 122.

[0099] In various cases, the training component 114 can select a data candidate 402 (e.g., one of the data candidates in 202) and an annotation 404 (e.g., one of the annotations in 204) corresponding to the data candidate 402 from the set of annotated data candidates 104. As shown, in various cases, the data candidate 402 can include m features: feature 402(1) to feature 402(m). In various aspects, the training component 114 can feed the data candidate 402 as input to the machine learning classifier 122, and this can cause the machine learning classifier 122 to generate an output 406. More specifically, in various cases, an input layer of the machine learning classifier 122 can receive the data candidate 402, the data candidate 402 can complete a forward pass through one or more hidden layers of the machine learning classifier 122, and an output layer of the machine learning classifier 122 can calculate the output 406 based on the activations generated by the one or more hidden layers of the machine learning classifier 122. In various cases, the output 406 can be considered to represent the inferred classification that the machine learning classifier 122 believes should correspond to the data candidate 402. In contrast, annotation 404 can be considered to represent a ground truth classification known to correspond to data candidate 402. Note that if the machine learning classifier 122 has not undergone and / or has undergone little training to date, the output 406 may be extremely inaccurate. That is, the output 406 may be very different from the annotation 404. In any case, the training component 114 can calculate the error and / or loss between the output 406 and the annotation 404. In various aspects, the training component 114 can then update the internal parameters of the machine learning classifier 122 by performing back propagation, where such back propagation is driven by the calculated error / loss. In various cases, the training component 114 can repeat the training process for each (and / or in some cases less) data candidate in the set of annotated data candidates 104. This will ultimately cause the internal parameters of the machine learning classifier 122 to be iteratively optimized to accurately classify the input data candidates. That is, such training can produce updated internal parameters 302. Those of ordinary skill in the art will appreciate that any suitable training batch size, any suitable training termination criteria, and / or any suitable error / loss function can be implemented by the training component 114 as needed.

[0100] Thus, in various cases, the training component 114 can electronically train the machine learning classifier 122 based on the set of annotated data candidates 104, thereby generating updated internal parameters 302. As will be appreciated by one of ordinary skill in the art, regardless of the specific structure and / or architecture of the machine learning classifier 122, the machine learning classifier 122 can have configurable internal parameters that are updated and / or optimized during training (e.g., node positions and / or node thresholds can be configurable internal parameters for tree-based models; weight vectors and / or bias values ​​can be configurable internal parameters for neural networks, and scalar coefficients can be configurable internal parameters for linear and / or logistic regression models).

[0101] Figure 5 A block diagram of an exemplary, non-limiting system 500 that can facilitate automated training of machine learning classifications for missed or delayed patient care opportunities, including a set of performance metrics, according to one or more embodiments described herein is shown. As shown, in some cases, system 500 can include the same components as system 300, and can also include a set of performance metrics 502.

[0102] In various embodiments, the validation component 116 can electronically calculate the set of performance metrics 502 by evaluating the performance of the machine learning classifier 122 after the training component 114 has trained the machine learning classifier 122. For example, in some aspects, the training component 114 can train the machine learning classifier 122 based on all subsets of the set of annotated data candidates 104 except any suitable subset. In such cases, the validation component 116 can utilize such subsets to generate the performance metrics 502. In other words, because such subsets may include various data candidates (e.g., some of the data candidates in 202) and corresponding ground truth annotations (e.g., some of the annotations in 204), the machine learning classifier 122 can be executed based on such various data candidates, and the output generated by the machine learning classifier 122 can be compared with the corresponding annotations to quantify the performance of the machine learning classifier 122. As will be appreciated by one of ordinary skill in the art, the set of performance metrics 502 may include any suitable metrics for quantifying the performance of the machine learning classifier 122. For example, the set of performance metrics 502 may include an accuracy score for the machine learning classifier 122. For another example, the set of performance metrics 502 may include a recall score for the machine learning classifier 122. As yet another example, the set of performance metrics 502 may include an area under the curve score for the machine learning classifier 122. As yet another example, the set of performance metrics 502 may include a true positive score, a false positive score, a true negative score, and / or a false negative score for the machine learning classifier 122.

[0103] Figure 6A block diagram of an exemplary, non-limiting system 600 that can facilitate automated training of machine learning classification for missed or delayed patient care opportunities including a set of feature category importance scores according to one or more embodiments described herein is shown. As shown, in some cases, system 600 can include the same components as system 500, and can also include a set of feature category importance scores 602.

[0104] In various embodiments, the feature component 118 can electronically generate the set of feature category importance scores 602 based on the updated internal parameters 302. In various aspects, the set of feature category importance scores 602 can correspond to the set of feature categories 206, respectively. In other words, each feature category can have an importance score. In various cases, the set of feature category importance scores 602 can be viewed as indicating and / or representing how strongly and / or weakly predictive each feature category in the set of feature categories 206 is relative to the desired classification. Figure 7 This is explained in more detail

[0105] Figure 7 An exemplary, non-limiting block diagram 700 illustrating a set of feature category importance scores is shown according to one or more embodiments described herein. That is, Figure 7 A non-limiting, exemplary embodiment of the set of feature category importance scores 602 is shown.

[0106] In various embodiments, as shown, the set of feature category importance scores 602 can correspond to the set of feature categories 206 in a one-to-one manner. Therefore, since the set of feature categories 206 can include m categories, the set of feature category importance scores 602 can include m scores: importance score 1 to importance score m. In various aspects, importance score 1 can correspond to feature category 1. Therefore, importance score 1 can be regarded as indicating the degree of prediction, importance, and / or otherwise relevance of feature category 1 to the desired classification (e.g., the degree of relevance of features / attributes belonging to feature category 1 to the desired classification). Similarly, in various cases, importance score m can correspond to feature category m. Therefore, importance score m can be regarded as indicating the degree of prediction, importance, and / or otherwise relevance of feature category m to the desired classification (e.g., the degree of relevance of features / attributes belonging to feature category m to the desired classification).

[0107] In various aspects, the importance score can be a real-valued scalar with a magnitude range between any two suitable real-valued numbers. For example, the real-valued importance score can be a scalar with a magnitude range between 0 and 1, where 0 indicates the lowest amount of classification importance, where 1 indicates the highest amount of classification importance, and where intermediate decimal values ​​indicate commensurate intermediate amounts of classification importance. However, in some cases, 0 may instead indicate the highest amount of classification importance, and 1 may instead represent the lowest amount of classification importance. One of ordinary skill in the art will appreciate that any other suitable magnitude range may be implemented (e.g., real-valued importance scores need not be limited to a range between 0 and 1).

[0108] In various other aspects, the importance score may be an integer-valued scalar with a magnitude ranging between 1 and m. In such cases, the importance score may be viewed as a ranking assigned to the set of feature categories 206. For example, in some cases, an integer-valued importance score of 1 may indicate the most important and / or most predictive feature category among m total feature categories (e.g., 1 may represent the first position among the m total feature categories), while an integer-valued importance score of m may indicate the least important and / or least predictive feature category among the m total feature categories (e.g., m may represent the last position among the m total feature categories). However, in other cases, an integer-valued importance score of 1 may indicate the least important and / or least predictive feature category among the m total feature categories (e.g., 1 may represent the lowest ranking among the m total feature categories), while an integer-valued importance score of m may indicate the most important and / or most predictive feature category among the m total feature categories (e.g., m may represent the highest ranking among the m total feature categories).

[0109] In any case, the set of feature category importance scores 602 can respectively indicate the relevance and / or importance of each feature category in the set of feature categories 206 relative to the desired classification that the machine learning classifier 122 is trained to perform.

[0110] Figure 8 An exemplary, non-limiting block diagram illustrating how a set of feature category importance scores may be generated according to one or more embodiments described herein is shown. Figure 8 It is shown how the features component 118 can electronically generate the set of feature category importance scores 602 .

[0111] As shown in the figure, Figure 8 Two scenarios are shown: scenario 802 and scenario 804 . As shown in scenario 802 , in various aspects, the features component 118 can electronically generate the set of feature category importance scores 602 by applying any suitable analysis technique to the updated internal parameters 302 and / or the set of feature categories 206 .

[0112] As a non-limiting example, assume that the machine learning classifier 122 presents a decision tree architecture. In such cases, the updated internal parameters 302 can be considered to represent the decision node position, decision node threshold percentage, leaf node position, and / or leaf node threshold percentage achieved by the machine learning classifier 122 after training by the training component 114. In various cases, each decision node of the machine learning classifier 122 can be considered to segment the input data candidate according to one of the feature categories in the set of feature categories 206. Therefore, each decision node of the machine learning classifier 122 can be considered to correspond to a corresponding feature category. In various aspects, the feature component 118 can assign importance scores to the feature categories based on the position of the decision node corresponding to each feature category. For example, the root node of the machine learning classifier 122 can be considered to segment the input data candidate based on the most important and / or most predictive feature category. Therefore, the feature component 118 can assign the best importance score (e.g., the best ranking) to the feature category corresponding to the root node. In addition, in various cases, the feature component 118 can assign importance scores to the feature categories based on how far the decision node corresponding to each remaining feature category is from the root node. For example, assume that a given decision node is located one level below the root node in the machine learning classifier 122 (e.g., one level away from the root node), and assume that a different decision node is located two levels below the root node in the machine learning classifier 122 (e.g., two levels away from the root node). In such cases, the feature component 118 can assign better importance scores (e.g., better rankings) to feature categories corresponding to the given decision node, and can assign worse importance scores (e.g., worse rankings) to feature categories corresponding to different decision nodes. In this manner, the feature component 118 can analytically assign importance scores to the set of feature categories 206 based on the updated internal parameters 302 (e.g., based on the trained decision node positions in the machine learning classifier 122 when the machine learning classifier 122 is a tree-based model).

[0113] As another non-limiting example, assume that the machine learning classifier 122 exhibits a linear and / or logistic regression architecture. In such cases, the updated internal parameters 302 can be considered to represent the regression coefficients achieved by the machine learning classifier 122 after training by the training component 114. In various cases, each regression coefficient of the machine learning classifier 122 can be multiplied by the corresponding feature of the input data candidate (and / or applied exponentially to the corresponding feature of the input data candidate), and such features can belong to a corresponding one of the set of feature categories 206. Therefore, each regression coefficient of the machine learning classifier 122 can be considered to correspond to a corresponding feature category (e.g., a corresponding one of 206). In various aspects, the feature component 118 can assign an importance score to each feature category based on the magnitude of the regression coefficient corresponding to the feature category. For example, the maximum regression coefficient of the machine learning classifier 122 can be considered to be multiplied by the most important and / or most predictive feature of the input data candidate (and / or applied exponentially to the most important and / or most predictive feature of the input data candidate). Therefore, the feature component 118 can assign the best importance score (e.g., best ranking) to the feature category corresponding to the maximum regression coefficient. In addition, in various cases, the feature component 118 can assign importance scores to the feature category based on the magnitude of the regression coefficient corresponding to each remaining feature category. For example, it is assumed that the first regression coefficient is greater than the second regression coefficient. In such cases, the feature component 118 can assign better importance scores (e.g., better ranking) to the feature category corresponding to the first regression coefficient, and can assign worse importance scores (e.g., worse ranking) to the feature category corresponding to the second regression coefficient. In this way, the feature component 118 can assign importance scores to the group of feature categories 206 in an analytical manner based on the updated internal parameters 302 (e.g., when the machine learning classifier 122 is a regression-based model, based on the trained regression coefficients in the machine learning classifier 122).

[0114] In various aspects, analysis techniques for assigning / calculating the set of feature category importance scores 602 (such as the analysis techniques described in the previous two paragraphs) are useful when the updated internal parameters 302 are easily interpreted, such as in the case of decision tree architectures and / or linear / logistic regression architectures. However, in some cases, the updated internal parameters 302 may be more complex and / or less easily interpreted. In fact, this may be the case when the machine learning classifier 122 is a deep learning neural network (e.g., a deep learning neural network may have hundreds and / or thousands of internal weight values ​​and / or bias values). In such cases, scenario 804 can be implemented.

[0115] As shown in scenario 804, in various aspects, the feature component 118 can electronically generate the set of feature category importance scores 602 by executing the importance score generating machine learning model 806 according to the updated internal parameters 302 and / or according to the set of feature categories 206. In such cases, the feature component 118 can be considered to electronically store, electronically maintain, electronically control and / or otherwise electronically access the importance score generating machine learning model 806. In various aspects, the importance score generating machine learning model 806 can represent any suitable artificial intelligence architecture as desired. For example, the importance score generating machine learning model 806 can represent a deep learning neural network architecture. In such cases, the importance score generating machine learning model 806 may have any suitable number of layers (e.g., an input layer, one or more hidden layers, an output layer), any suitable number of neurons in the various layers (e.g., different layers may have the same and / or different numbers of neurons from each other), any suitable activation functions (e.g., sigmoid, softmax, hyperbolic tangent, rectified linear unit) in the various neurons (e.g., different neurons may have the same and / or different activation functions from each other), and / or any suitable pattern of connections between neurons (e.g., forward connections, recurrent connections, skip connections).

[0116] In any case, the importance score generating machine learning model 806 can be configured to receive the updated internal parameters 302 and / or the set of feature categories 206 as input, and the importance score generating machine learning model 806 can be further configured to generate the set of feature category importance scores 602 as output. More specifically, the updated internal parameters 302 can be represented by one or more first scalars, vectors, matrices, tensors, and / or strings. Similarly, the set of feature categories 206 can be represented by one or more second scalars, vectors, matrices, tensors, and / or strings. Thus, in various cases, a first scalar, vector, matrix, tensor, and / or string can be concatenated with a second scalar, vector, matrix, tensor, and / or string, an input layer of the importance score generating machine learning model 806 can receive such a cascade, such a cascade can complete a forward pass through one or more hidden layers of the importance score generating machine learning model 806, and / or an output layer of the importance score generating machine learning model 806 can calculate the set of feature category importance scores 602 based on the activations generated by one or more hidden layers of the importance score generating machine learning model 806.

[0117] In order for the importance score generating machine learning model 806 to accurately perform its above-mentioned functions, it should first be trained. In various cases, such training can be as described above with respect to Figures 9 to 11 to achieve as described.

[0118] Fig. 9 A block diagram of an exemplary, non-limiting system 900 that can facilitate automated training of machine learning classification for missed or delayed patient care opportunities including an importance score training dataset according to one or more embodiments described herein is shown. As shown, in various cases, the system 900 can include the same components as the system 600, and can also include an importance score training dataset 902.

[0119] In various aspects, receiver component 110 can electronically receive, retrieve, obtain, and / or otherwise access importance score training dataset 902 from any suitable source, and training component 114 can electronically train importance score generation machine learning model 806 based on importance score training dataset 902. Figure 10 to Figure 11 This is described further.

[0120] Fig.10 An exemplary, non-limiting block diagram 1000 illustrating an importance score training data set is shown according to one or more embodiments described herein. That is, Fig.10 A non-limiting, exemplary embodiment of an importance score training data set 902 is shown.

[0121] In various embodiments, as shown, the importance score training data set 902 may include a set of training cascades 1002 and / or a set of basic fact importance scores 1004. In various aspects, the set of training cascades 1002 may include x cascades for any suitable positive integer x: the first training cascade to the xth training cascade. In addition, in various cases, each training cascade in the set of training cascades may include training internal parameters cascaded with a set of training feature categories. For example, the first training cascade may include training internal parameters 1 cascaded with a set of training feature categories 1. In various cases, the training internal parameters 1 may have the same data format and / or data dimension as the updated internal parameters 302 and / or the set of training feature categories 1 may have the same data format and / or data dimension as the set of feature categories 206. Similarly, the xth training cascade may include training internal parameters x cascaded with a set of training feature categories x. As described above, the training intrinsic parameters x may have the same data format and / or data dimension as the updated intrinsic parameters 302 and / or the set of training feature categories x may have the same data format and / or data dimension as the set of feature categories 206 .

[0122] In various aspects, as shown in the figure, the group of basic fact importance scores 1004 can correspond to the group of training cascades 1002 respectively. Therefore, since the group of training cascades 1002 can have x cascades, the group of basic fact importance scores 1004 can have x groups of importance scores: a group of basic fact importance scores 1 to a group of basic fact importance scores x. In various cases, the group of basic fact importance scores 1 can have the same data format and / or data dimension as the group of feature category importance scores 602. In addition, in various cases, the group of basic fact importance scores 1 can correspond to the first training cascade. In other words, when given training internal parameters 1, the group of basic fact importance scores 1 can be regarded as known and / or considered to be assigned to the group of training feature categories 1 The accurate / correct importance score (e.g., accurate / correct ranking). Similarly, in various aspects, the group of basic fact importance scores x can have the same data format and / or data dimension as the group of feature category importance scores 602. Furthermore, in various cases, the set of ground truth importance scores x can correspond to the xth training cascade. That is, when given the training internal parameters x, the set of ground truth importance scores x can be considered as the accurate / correct importance scores (e.g., accurate / correct ranking) assigned to the set of training feature categories x.

[0123] Fig.11 An exemplary, non-limiting block diagram 1100 illustrating how an importance score generating machine learning model may be trained based on an importance score training dataset according to one or more embodiments described herein is shown. That is, Fig.11 It is shown how an importance score generation machine learning model 806 can be trained based on an importance score training dataset 902.

[0124] In various embodiments, the internal parameters (e.g., weights, biases) of the importance score generating machine learning model 806 can be randomly initialized. In various aspects, the training component 114 can select any suitable cascade from the importance score training data set 902. Fig.11 As shown, the selected cascade can include training internal parameters 1102 and / or a set of training feature categories 1104. In addition, the training component 114 can identify a set of ground truth importance scores 1106 in the importance score training dataset 902 that correspond to the selected cascade.

[0125] In various cases, the training component 114 can feed the selected cascade to the importance score generating machine learning model 806, which can cause the importance score generating machine learning model 806 to produce a certain output 1108. More specifically, the training internal parameters 1102 and the set of training feature categories 1104 can be cascaded together, the input layer of the importance score generating machine learning model 806 can receive such cascades, such cascades can be passed forward through one or more hidden layers of the importance score generating machine learning model 806, and the output layer of the importance score generating machine learning model 806 can calculate the output 1108 based on the activations generated by the one or more hidden layers of the importance score generating machine learning model 806. In various cases, the output 1108 can be regarded as representing the set of importance scores that the importance score generating machine learning model 806 believes should correspond to the set of training feature categories 1104 given the training internal parameters 1102. In contrast, given the training internal parameters 1102, the set of ground truth importance scores 1106 can be viewed as representing the importance scores and / or rankings known and / or believed to correspond to the set of training feature categories 1104. Note that if the importance score generating machine learning model 806 has not been trained to date and / or has been trained very little, then the output 1108 may be very inaccurate (e.g., the output 1108 may be significantly different from the set of ground truth importance scores 1106).

[0126] In any case, the training component 114 can calculate the error / loss (e.g., Euclidean distance, cross entropy) between the output 1108 and / or the set of ground truth importance scores 1106. Accordingly, the training component 114 can update the internal parameters (e.g., weights, biases) of the importance score generating machine learning model 806 by performing back-propagation, where such back-propagation is driven by the calculated error and / or loss.

[0127] In various aspects, the training component 114 can repeat the training process for each training cascade in the importance score training data set 902, with the end result that the internal parameters of the importance score generating machine learning model 806 can be iteratively optimized to accurately infer importance scores from the input cascades. As will be appreciated by those of ordinary skill in the art, any suitable training batch size, any suitable training termination criteria, and / or any suitable error, loss, and / or objective function can be implemented.

[0128] In addition, those skilled in the art will understand that Figures 9 to 11 The “internal parameters” being updated during training in question belong to the importance score generating machine learning model 806, not to the machine learning classifier 122. Figure 4 The training of the machine learning classifier 122 is described.

[0129] In any case, the feature component 118 can electronically generate the set of feature category importance scores 602 based on the updated internal parameters 302 .

[0130] Fig.12 A block diagram of an exemplary, non-limiting system 1200 that can facilitate automated training of machine learning classification for missed or delayed patient care opportunities including a threshold importance score and a set of updated annotated data candidates according to one or more embodiments described herein is shown. As shown, in some cases, system 1200 can include the same components as system 900, and can also include a threshold importance score 1202 and / or a set of updated annotated data candidates 1204.

[0131] In various embodiments, the execution component 120 can electronically facilitate any suitable electronic action based on the set of feature category importance scores 602. For example, in some cases, the execution component 120 can electronically transmit any feature category importance score in the set of feature category importance scores 602 to any suitable computing device (not shown). For another example, in some cases, the execution component 120 can electronically present any feature category importance score in the set of feature category importance scores 602 on any suitable computer screen / monitor (not shown).

[0132] However, in some aspects, the execution component 120 may electronically store, maintain, and / or otherwise access the threshold importance score 1202. In various cases, the threshold importance score 1202 may be viewed as a delineation of a marker that separates a feature class that is sufficiently important and / or sufficiently predictive from a feature class that is not sufficiently important and / or not sufficiently predictive. As will be appreciated by one of ordinary skill in the art, the magnitude of the threshold importance score 1202 may be any suitable value desired. In any case, the execution component 120 may compare the set of feature class importance scores 602 to the threshold importance score 1202. If any importance score from the set of feature class importance scores 602 fails to meet the threshold importance score 1202, then the feature class corresponding to the importance score (e.g., one of the feature classes in 206) may be viewed as too irrelevant and / or too weakly predictive with respect to the desired classification that the machine learning classifier 122 is trained / configured to perform. In various aspects, if the execution component 120 identifies one or more feature categories that are too irrelevant and / or too weakly predictive, the execution component 120 can electronically delete, erase, remove, and / or otherwise eliminate such one or more feature categories from the set of feature categories 206, and thus electronically delete, erase, remove, and / or otherwise eliminate such one or more feature categories from the set of annotated data candidates 104. In various cases, the version of the set of annotated data candidates 104 resulting from such deletion, erasure, removal, and / or elimination can be considered the updated set of annotated data candidates 1204.

[0133] Fig.13 An exemplary, non-limiting block diagram 1300 of a set of updated annotated data candidates according to one or more embodiments described herein is shown. Fig.13 A non-limiting, exemplary embodiment of the updated set of annotated data candidates 1204 is shown.

[0134] exist Fig.13 In a non-limiting example, the execution component 120 can determine that the importance score 1 corresponding to feature category 1 fails to meet the threshold importance score 1202. Therefore, it can be inferred and / or concluded that feature category 1 is not important enough for the desired classification that the machine learning classifier 122 is configured to perform and / or is not predictive enough for the desired classification. In such a case, the execution component 120 can delete and / or remove feature category 1 from the set of feature categories 206. In other words, the execution component 120 can delete and / or remove the first feature from each data candidate in the set of data candidates 202, such as Fig.13Thus, after such deletion / removal, the set of feature categories 206 can be viewed as including a total of m-1 feature categories, rather than a total of m feature categories (e.g., each data candidate in the set of data candidates 202 can be viewed as now having a total of m-1 features, rather than a total of m features).

[0135] More generally, if the execution component 120 determines that the i-th importance score of the set of feature category importance scores 602 fails to meet the threshold importance score 1202, the execution component 120 can delete / remove the i-th feature category from the set of feature categories 206 (e.g., the i-th feature can be deleted / removed from each data candidate in the set of data candidates 202).

[0136] In various aspects, once the execution component 120 generates the updated set of annotated data candidates 1204, the training component 114 can reconfigure and / or retrain the machine learning classifier 122 based on the updated set of annotated data candidates 1204. As verified by the inventors experimentally, retraining the machine learning classifier 122 after irrelevant and / or weakly predictive feature categories are deleted / removed can significantly improve the performance (e.g., accuracy, recall, area under the curve) exhibited by the machine learning classifier 122. In fact, the inventors experimentally implemented the embodiments described herein. After initial training but before retraining, the machine learning classifier 122 exhibited a recall rate of 32%, a precision rate of 29%, and an area under the curve score of 0.78. However, after retraining based on the training data set, once the weakly predictive feature categories were deleted from the training data set, the machine learning classifier 122 exhibited a recall rate of 96%, an accuracy rate of 49%, and an area under the curve score of 0.99. This certainly constitutes a concrete and tangible technical improvement.

[0137] In various embodiments, after retraining has been completed, the execution component 120 can electronically deploy the machine learning classifier 122 in any suitable operational context as needed (e.g., the machine learning classifier 122 can be executed based on any suitable data candidates encountered in the field).

[0138] Fig.14 A flow chart of an exemplary, non-limiting computer-implemented method 1400 that can facilitate automated training of a machine learning classifier for missed or delayed patient care opportunities according to one or more embodiments described herein is shown. In various cases, the automated classifier training system 102 can facilitate the computer-implemented method 1400.

[0139] In various embodiments, act 1402 may include accessing, by a device operatively coupled to a processor (eg, via 110 ), a set of annotated data candidates (eg, 104 ) defined by two or more feature categories (eg, 206 ).

[0140] In various aspects, action 1404 may include training a machine learning classifier (e.g., 122) based on the set of annotated data candidates by a device (e.g., via 114), thereby iteratively updating internal parameters of the machine learning classifier (e.g., thereby generating 302).

[0141] In various cases, action 1406 may include ranking, by the device (e.g., via 118), two or more feature categories in order of classification importance based on iteratively updated internal parameters of the machine learning classifier (e.g., 602 may be considered to represent and / or communicate such ranking).

[0142] In various cases, action 1408 may include performing, by the device (eg, via 120 ), one or more electronic actions based on ranking the two or more feature categories in order of classification importance.

[0143] Despite Fig.14 It is not explicitly shown in , but the machine learning classifier can be an XGBoost model, and the internal parameters can include the decision tree threshold of the XGBoost model or the decision tree position of the XGBoost model.

[0144] Despite Fig.14 Not explicitly shown, but the one or more electronic actions may include visually presenting, by a device (eg, via 120 ) and on an electronic display, a ranking of two or more feature categories (eg, 602 ).

[0145] Despite Fig.14 Not explicitly shown, but one or more electronic actions may include visually presenting, by a device (e.g., via 120) and on an electronic display, an indication of which of the two or more feature categories are ranked below a threshold value (e.g., 1202).

[0146] Despite Fig.14 Not explicitly shown, but the one or more electronic actions may include: identifying, by a device (e.g., via 120), which of two or more feature categories are ranked below a threshold (e.g., 1202); removing, by a device (e.g., via 120), such feature categories from the set of annotated data candidates, thereby generating an updated set of annotated data candidates (e.g., 1204); and retraining, by a device (e.g., via 114), a machine learning classifier based on the updated set of annotated data candidates.

[0147] Despite Fig.14 Not explicitly shown in the figure, but the machine learning classifier can be configured to receive as input data candidates associated with medical patients with scheduled medical appointments, the machine learning classifier can be configured to generate as output a classification label based on the data candidates, and the classification label can indicate whether the medical patient is likely to not show up for the scheduled medical appointment.

[0148] Despite Fig.14 Not explicitly shown in the figure, but the machine learning classifier can be configured to receive as input data candidates associated with medical patients with scheduled medical appointments, the machine learning classifier can be configured to generate as output a classification label based on the data candidates, and the classification label can indicate whether the medical patient is likely to be late for the scheduled medical appointment.

[0149] Despite Fig.14 Not explicitly shown in , but each annotated data candidate in the set of annotated data candidates may belong to a respectively corresponding medical patient having a respectively corresponding scheduled medical appointment, and the two or more feature categories may include a patient demographic feature category, a patient medical record feature category, a weather feature category, and / or a traffic feature category.

[0150] Thus, various embodiments described herein include computerized tools that can automatically train a machine learning classifier based on a set of data candidates defined by a set of feature categories, can rank the set of feature categories based on optimized / updated internal parameters implemented by the machine learning classifier after training, can eliminate weakly predictive feature categories from the set of data candidates, and / or can retrain the machine learning classifier based on the set of data candidates after the weakly predictive feature categories have been removed. Such computerized tools can significantly improve the accuracy, precision, and / or recall of a machine learning classifier. Thus, such computerized tools certainly constitute a useful and practical application of computers.

[0151] In various cases, the machine learning algorithm and / or model may be implemented in any suitable manner to facilitate any suitable aspect described herein. In order to facilitate some of the above-mentioned machine learning aspects of various embodiments of the subject innovation, consider the following discussion of artificial intelligence (AI). Various embodiments of the present innovation herein may employ artificial intelligence to facilitate the automation of one or more features of the present innovation. These components may employ various AI-based schemes to perform various embodiments / examples disclosed herein. In order to provide or contribute to the numerous determinations (e.g., determination, detection, inference, calculation, prediction, prognosis, estimation, derivation, forecast, detection, estimation) of the present innovation, the components of the present innovation may examine the entirety or subgroup of the data to which it is granted access rights, and may provide reasoning about or determine the state of the system and / or environment from a set of observations captured via events and / or data. For example, determination may be used to identify a specific context or action, or a probability distribution of a state may be generated. These determinations may be probabilistic; that is, the calculation of the probability distribution of the state of interest is based on consideration of data and events. Determination may also refer to a technique for composing a higher-level event from a set of events and / or data.

[0152] Such determination can result in constructing new events or actions from a set of observed events and / or stored event data, whether or not the events are closely related in time, and whether or not the events and data come from one or more event and data sources. The components disclosed herein can employ various classification (explicitly trained (e.g., via training data) and implicitly trained (e.g., via observed behavior, preferences, historical information, receiving external information, etc.)) schemes and / or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) in conjunction with performing automatic and / or determined actions associated with the claimed subject matter. Thus, classification schemes and / or systems can be used to automatically learn and perform a variety of functions, actions, and / or determinations.

[0153] The classifier can take the input attribute vector z=(z1,z2,z3,z4,z n) is mapped to the confidence that the input belongs to a certain category, such as according to f(z) = confidence(category). Such classification can use probability and / or statistical based analysis (e.g., analyzing utility and cost considerations) to determine the actions to be automatically performed. Support vector machine (SVM) can be an example of a classifier that can be used. SVM operates by finding a hypersurface in the space of possible inputs, where the hypersurface attempts to separate triggering criteria from non-triggering events. Intuitively, this makes the classification correct for testing data that is close to but different from the training data. Other directed and undirected model classification methods include, for example, naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, and any of the probabilistic classification models that provide different independent patterns can be used. Classification as used herein also includes statistical regression for developing priority models.

[0154] Those of ordinary skill in the art will appreciate that the disclosure herein describes non-limiting examples of various embodiments of this subject innovation. For ease of description and / or illustration, when discussing various embodiments of this subject innovation, the various parts of this disclosure utilize the term "each". Those of ordinary skill in the art will appreciate that such uses of the term "each" are non-limiting examples. In other words, when the disclosure herein provides a description of "each" specific object and / or parts applied to some specific objects and / or parts, it should be understood that this is a non-limiting example of various embodiments of this subject innovation, and it should also be understood that in various other embodiments of this subject innovation, it may be that such description is applicable to less than the situation of "each" specific object and / or parts in this specific object and / or parts.

[0155] In order to provide additional context for the various embodiments described herein, Fig.15 The following discussion is intended to provide a brief, general description of a suitable computing environment 1500 in which various embodiments of the embodiments described herein may be implemented. Although the embodiments have been described above in the general context of computer-executable instructions that may be executed on one or more computers, those skilled in the art will recognize that the embodiments may also be implemented in conjunction with other program modules and / or as a combination of hardware and software.

[0156] Generally, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In addition, those skilled in the art will appreciate that the methods of the present invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc., each of which is operatively connected to one or more associated devices.

[0157] The illustrated embodiments of the embodiments herein may also be practiced in a distributed computing environment where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0158] Computing devices typically include various media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, where the two terms are used differently in this article, as described below. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by a computer, and include volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable storage media or machine-readable storage media can be implemented in conjunction with any method or technology for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0159] Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disk (BD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store the desired information. In this regard, the terms "tangible" or "non-transitory" as applied to storage, memory, or computer-readable media herein should be understood as excluding only the propagation of transient signals themselves as a modifier, and not disclaiming all standard storage, memory, or computer-readable media that are not only the propagation of transient signals themselves.

[0160] Computer-readable storage media may be accessed by one or more local or remote computing devices, eg, via access requests, queries, or other data retrieval protocols, to perform various operations on the information stored by the media.

[0161] Communication media typically embodies computer readable instructions, data structures, program modules, or other structured or unstructured data in a data signal, which may be, for example, a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery or transmission media. The term "modulated data signal" or "signal" refers to a signal that has one or more of its characteristics set or changed to encode information in one or more signals. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media.

[0162] Reference again Fig.15 , an exemplary environment 1500 for implementing various embodiments of aspects described herein includes a computer 1502, which includes a processing unit 1504, a system memory 1506, and a system bus 1508. The system bus 1508 couples system components including, but not limited to, the system memory 1506 to the processing unit 1504. The processing unit 1504 can be any of a variety of commercially available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 1504.

[0163] The system bus 1508 may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory 1506 includes ROM 1510 and RAM 1512. A basic input / output system (BIOS) may be stored in a non-volatile memory such as ROM, an erasable programmable read-only memory (EPROM), an EEPROM, wherein the BIOS contains basic routines that help to transfer information between elements within the computer 1502, such as during startup. The RAM 1512 may also include high-speed RAM, such as static RAM for caching data.

[0164] The computer 1502 also includes an internal hard disk drive (HDD) 1514 (e.g., EIDE, SATA), one or more external storage devices 1516 (e.g., a magnetic floppy disk drive (FDD) 1516, a memory stick or flash drive reader, a memory card reader, etc.), and a drive 1520, such as a solid-state drive, an optical drive, which can read or write from a disk 1522 (such as a CD-ROM disk, a DVD, a BD, etc.). Alternatively, in the case of a solid-state drive, the disk 1522 will not be included unless it is separated. Although the internal HDD 1514 is illustrated as being located within the computer 1502, the internal HDD 1514 can also be configured to be used externally in a suitable infrastructure (not shown). In addition, although not shown in the environment 1500, a solid-state drive (SSD) can be used in addition to or instead of the HDD 1514. The HDD 1514, external storage device 1516, and drive 1520 may be connected to the system bus 1508 via a HDD interface 1524, an external storage interface 1526, and a drive interface 1528, respectively. The interface 1524 for an external drive implementation may include at least one or both of a Universal Serial Bus (USB) and an Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technology. Other external drive connection technologies are within the contemplation of the embodiments described herein.

[0165] The drives and their associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, etc. For the computer 1502, the drives and storage media accommodate the storage of any data in a suitable digital format. Although the above description of computer-readable storage media refers to corresponding types of storage devices, it should be understood by those skilled in the art that other types of computer-readable storage media (whether currently existing or developed in the future) can also be used in the exemplary operating environment, and further, any such storage media can contain computer-executable instructions for performing the methods described herein.

[0166] A number of program modules may be stored in the drives and RAM 1512, including an operating system 1530, one or more application programs 1532, other program modules 1534, and program data 1536. All or portions of the operating system, application programs, modules, and / or data may also be cached in RAM 1512. The systems and methods described herein may be implemented using various commercially available operating systems or combinations of operating systems.

[0167] Computer 1502 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate the hardware environment for operating system 1530, and the emulated hardware may optionally be different from the hardware of the operating system 1530. Fig.15 1502. In such embodiments, operating system 1530 may include one of a plurality of virtual machines (VMs) hosted at computer 1502. In addition, operating system 1530 may provide an operating environment, such as a Java operating environment or a .NET framework, to application 1532. The operating environment is a consistent execution environment that allows application 1532 to run on any operating system that includes the operating environment. Similarly, operating system 1530 may support containers, and application 1532 may be in the form of a container that is a lightweight, standalone, executable software package that includes, for example, the application's code, runtime, system tools, system libraries, and settings.

[0168] In addition, the computer 1502 can be enabled with a security module such as a trusted processing module (TPM). For example, in the case of a TPM, the boot component is hashed in the next boot component, and the result is waited for to match the security value before loading the next boot component. This process can occur at any layer in the code execution stack of the computer 1502, such as applied to the application execution level or the operating system (OS) kernel level, thereby achieving security at any code execution level.

[0169] A user may enter commands and information into the computer 1502 through one or more wired / wireless input devices (e.g., a keyboard 1538, a touch screen 1540, and a pointing device such as a mouse 1542). Other input devices (not shown) may include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control or other remote control, a joystick, a virtual reality controller and / or a virtual reality headset, a game pad, a stylus, an image input device (e.g., camera(s)), a gesture sensor input device, a visual movement sensor input device, an emotion or facial detection device, a biometric input device (e.g., a fingerprint or iris scanner), and the like. These and other input devices are often connected to the processing unit 1504 through an input device interface 1544, which may be coupled to the system bus 1508, but these and other input devices may be connected through other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR port, a 32-bit 4K optical drive, a 4K optical drive, a 24-bit ... Interfaces, etc.

[0170] A monitor 1546 or other type of display device may also be connected to the system bus 1508 via an interface, such as a video adapter 1548. In addition to the monitor 1546, computers typically include other peripheral output devices (not shown), such as speakers, printers, and the like.

[0171] The computer 1502 can operate in a networked environment using logical connections to one or more remote computers such as (one or more) remote computers 1550 via wired and / or wireless communications. The remote computer 1550 can be a workstation, a server computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment appliance, a peer device, or other public network node, and typically includes many or all of the elements described with respect to the computer 1502, although only a memory / storage device 1552 is illustrated for simplicity. The depicted logical connections include wired / wireless connections to a local area network (LAN) 1554 and / or a larger network (e.g., a wide area network (WAN) 1556). Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global communication network, such as the Internet.

[0172] When used in a LAN networking environment, the computer 1502 can be connected to the local network 1554 through a wired and / or wireless communication network interface or adapter 1558. The adapter 1558 can facilitate wired or wireless communication with the LAN 1554, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter 1558 in a wireless mode.

[0173] When used in a WAN networking environment, the computer 1502 may include a modem 1560, or may be connected to a communications server on the WAN 1556 via other means for establishing communications over the WAN 1556, such as through the Internet. The modem 1560, which may be an internal or external device and a wired or wireless device, may be connected to the system bus 1508 via the input device interface 1544. In a networked environment, program modules depicted relative to the computer 1502, or portions thereof, may be stored in the remote memory / storage device 1552. It should be appreciated that the network connections shown are examples and other means of establishing a communications link between the computers may be used.

[0174] When used in a LAN or WAN networking environment, in addition to or as an alternative to the external storage device 1516 described above, the computer 1502 may access a cloud storage system or other network-based storage system, such as, but not limited to, a network virtual machine that provides one or more aspects of information storage or processing. Generally speaking, the connection between the computer 1502 and the cloud storage system can be established through the LAN 1554 or WAN 1556, for example, by an adapter 1558 or a modem 1560, respectively. When the computer 1502 is connected to the associated cloud storage system, the external storage interface 1526 can manage the storage provided by the cloud storage system with the help of the adapter 1558 and / or the modem 1560, just like other types of external storage devices. For example, the external storage interface 1526 can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer 1502.

[0175] The computer 1502 can be operated to communicate with any wireless device or entity operatively arranged in wireless communication, such as a printer, scanner, desktop and / or portable computer, portable data assistant, communication satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., information kiosk, newsstand, store shelf, etc.), and telephone. This may include Wireless Fidelity (Wi-Fi) and Wireless technology. Therefore, the communication can be a predefined structure like a conventional network, or just an ad hoc communication between at least two devices.

[0176] Fig.1616 is a schematic block diagram of a sample computing environment 1600 with which the disclosed subject matter can interact. The sample computing environment 1600 includes one or more clients 1610. The client 1610 can be hardware and / or software (e.g., a thread, a process, a computing device). The sample computing environment 1600 also includes one or more servers 1630. The server 1630 can also be hardware and / or software (e.g., a thread, a process, a computing device). For example, the server 1630 can accommodate threads to perform transformations by adopting one or more embodiments described herein. One possible communication between the client 1610 and the server 1630 can be in the form of data packets suitable for transmission between two or more computer processes. The sample computing environment 1600 includes a communication framework 1650 that can be used to facilitate communication between the client 1610 and the server 1630. The client 1610 is operably connected to one or more client data repositories 1620, which can be used to store information local to the client 1610. Similarly, server 1630 may be operably connected to one or more server data repositories 1640 , which may be used to store information local to server 1630 .

[0177] The present invention may be a system, method, device and / or computer program product at any possible level of technical detail of integration. A computer program product may include a computer-readable storage medium (or multiple media) having a computer-readable program instruction thereon for causing a processor to perform aspects of the present invention. A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. An incomplete list of more specific examples of computer-readable storage media may also include the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the above items. As used herein, computer-readable storage media should not be understood as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0178] Computer-readable program instructions as described herein can be downloaded to corresponding computing / processing equipment from computer-readable storage media, or downloaded to external computers or external storage devices via networks (e.g., the Internet, local area networks, wide area networks and / or wireless networks). Networks can include copper transmission cables, optical transmission optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers and / or edge servers. Network adapter cards or network interfaces in each computing / processing equipment receive computer-readable program instructions from the network, and forward computer-readable program instructions for storage in computer-readable storage media in corresponding computing / processing equipment. Computer-readable program instructions for performing the operation of the present invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming languages ​​or similar programming languages). Computer readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can be executed by utilizing the state information of the computer readable program instructions to perform computer readable program instructions with personalized electronic circuits, so as to perform aspects of the present invention.

[0179] Various aspects of the present invention are described herein with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention.It should be understood that each frame of flowchart illustration and / or block diagram, and the combination of the frame in flowchart illustration and / or block diagram can be realized by computer-readable program instructions.These computer-readable program instructions can be provided to the processor of general-purpose computer, special-purpose computer or other programmable data processing device to produce machine, so that the instruction executed via the processor of computer or other programmable data processing device creates the device for implementing the function / action specified in one or more frames of flowchart and / or block diagram.These computer-readable program instructions can also be stored in computer-readable storage medium, which can guide computer, programmable data processing device and / or other equipment to work in a particular way, so that the computer-readable storage medium with instruction stored therein includes product, and the product includes the instruction of various aspects of function / action specified in one or more frames of flowchart and / or block diagram. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational actions to be performed on the computer, other programmable apparatus, or other devices to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other devices implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible specific implementation architecture, functionality and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, fragment or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the function indicated in the frame may not occur in the order indicated in the figure. For example, in fact, the two frames shown in succession can be executed substantially at the same time, or sometimes these frames may be executed in reverse order, depending on the functionality involved. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a system based on dedicated hardware that performs a specified function or action or implements a combination of dedicated hardware and computer instructions.

[0181] Although the subject matter is described above in the general context of computer executable instructions of a computer program product running on one and / or multiple computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that other computer system configurations may be used to practice the computer-implemented method of the present invention, including single-processor or multi-processor computer systems, small computing devices, large computers, and computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects may also be practiced in a distributed computing environment in which tasks are performed by a remote processing device linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on a stand-alone computer. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0182] As used in this application, the terms "component", "system", "platform", "interface", etc. may refer to and / or may include computer-related entities or entities related to an operating machine having one or more specific functionalities. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program, a processor, an object, an executable file, an execution thread, a program, and / or a computer running on a processor. By way of illustration, both an application program and a server running on a server may be a component. One or more components may reside in a program and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. For another example, a corresponding component may be executed according to various computer-readable media having various data structures stored thereon. Components may communicate via local and / or remote processes, such as according to signals having one or more data packets (e.g., data from a component, the component interacting with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems) via a signal). As another example, a component may be a device having a specific functionality provided by a mechanical part operated by an electrical or electronic circuit system operated by a software or firmware application executed by a processor. In such cases, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component may be a device that provides specific functionality through electronic components rather than mechanical parts, wherein the electronic components may include a processor or other device for executing software or firmware that at least partially imparts functionality to the electronic components. In one aspect, the component may simulate the electronic component, for example, via a virtual machine within a cloud computing system.

[0183] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any natural inclusive permutation. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. In addition, unless otherwise specified or clear from the context that it is for a singular form, the articles "a" and "an" used in this specification and the drawings should generally be understood to mean "one or more". As used herein, the terms "example" and / or "exemplary" are used to indicate use as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as being preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0184] As used in this specification, the term "processor" may refer to substantially any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, a discrete hardware component, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize a nanoscale architecture (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of user equipment. The processor may also be implemented as a combination of computing processing units. In the present disclosure, terms such as "repository", "storage device", "data repository", "data storage device", "database", and substantially any other information storage component related to the operation and functionality of the component are used to refer to a "memory component", an entity embodied in a "memory", or a component including a memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that may act as an external cache memory. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). In addition, the disclosed memory components of the system or computer-implemented method herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0185] What has been described above includes only examples of systems and computer-implemented methods. Of course, it is not possible to describe every conceivable combination of components or computer-implemented methods for the purposes of describing the present disclosure, but one of ordinary skill in the art will recognize that many other combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including," "having," "having," and the like are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "comprising," as interpreted when "including" is used as a transitional word in a claim.

[0186] Descriptions of various embodiments have been given for purposes of illustration, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best illustrate the principles of the embodiments, practical applications or technical improvements over technologies found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0187] Various non-limiting aspects of the embodiments described herein are provided in the following clauses.

[0188] Item 1: A system comprising: a processor that executes computer executable components stored in a computer readable memory, the computer executable components comprising: a receiver component that accesses a set of annotated data candidates defined by two or more feature categories; a training component that trains a machine learning classifier based on the set of annotated data candidates so that internal parameters of the machine learning classifier are iteratively updated; a feature component that sorts the two or more feature categories in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; and an execution component that performs one or more electronic actions based on the two or more feature categories sorted in order of classification importance.

[0189] Clause 2: The system of any preceding clause, wherein the machine learning classifier is an XGBoost model, and wherein the internal parameters include a decision tree threshold of the XGBoost model or a decision tree position of the XGBoost model.

[0190] Clause 3: The system of any preceding clause, wherein the one or more electronic actions include visually presenting the ranking of the two or more feature categories on an electronic display.

[0191] Clause 4: The system of any preceding clause, wherein the one or more electronic actions include visually presenting on an electronic display an indication of which of the two or more feature categories are ranked below a threshold.

[0192] Clause 5: A system according to any preceding clause, wherein the one or more electronic actions include identifying which of the two or more feature categories are ranked below a threshold and removing such feature categories from the set of annotated data candidates, thereby producing an updated set of annotated data candidates, and wherein the training component retrains the machine learning classifier based on the updated set of annotated data candidates.

[0193] Clause 6: A system according to any preceding clause, wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to not show up for the scheduled medical appointment.

[0194] Clause 7: A system according to any preceding clause, wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to be late for the scheduled medical appointment.

[0195] Clause 8: A system according to any preceding clause, wherein each annotated data candidate in the set of annotated data candidates belongs to a respectively corresponding medical patient having a respectively corresponding scheduled medical appointment, and wherein the two or more feature categories include a patient demographic feature category, a patient medical record feature category, a weather feature category, and a traffic feature category.

[0196] In various embodiments, any suitable combination of clauses 1 to 8 may be implemented.

[0197] Clause 9: A computer-implemented method, the computer-implemented method comprising: accessing a set of annotated data candidates defined by two or more feature categories by a device operatively connected to a processor; training a machine learning classifier based on the set of annotated data candidates by the device, thereby iteratively updating internal parameters of the machine learning classifier; sorting the two or more feature categories by the device in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; and performing one or more electronic actions by the device based on the two or more feature categories sorted in order of classification importance.

[0198] Clause 10: A computer-implemented method according to any preceding clause, wherein the machine learning classifier is an XGBoost model, and wherein the internal parameters include a decision tree threshold of the XGBoost model or a decision tree position of the XGBoost model.

[0199] Clause 11: The computer-implemented method of any preceding clause, wherein the one or more electronic actions include visually presenting, by the device and on an electronic display, a ranking of the two or more feature categories.

[0200] Clause 12: A computer-implemented method as described in any preceding clause, wherein the one or more electronic actions include visually presenting, by the device and on an electronic display, an indication of which of the two or more feature categories are ranked below a threshold.

[0201] Clause 13: A computer-implemented method according to any preceding clause, wherein the one or more electronic actions include: identifying, by the device, which of the two or more feature categories are ranked below a threshold; removing, by the device, such feature categories from the set of annotated data candidates, thereby generating an updated set of annotated data candidates; and retraining, by the device, the machine learning classifier based on the updated set of annotated data candidates.

[0202] Clause 14: A computer-implemented method according to any preceding clause, wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to not show up for the scheduled medical appointment.

[0203] Clause 15: A computer-implemented method according to any preceding clause, wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to be late for the scheduled medical appointment.

[0204] Clause 16: A computer-implemented method according to any preceding clause, wherein each annotated data candidate in the set of annotated data candidates belongs to a respectively corresponding medical patient having a respectively corresponding scheduled medical appointment, and wherein the two or more feature categories include a patient demographic feature category, a patient medical record feature category, a weather feature category, and a traffic feature category.

[0205] In various embodiments, any suitable combination of clauses 9 to 16 may be implemented.

[0206] Clause 17: A computer program product for facilitating automated training of a machine learning classifier for missed or delayed patient care opportunities, the computer program product comprising a computer readable memory having program instructions contained therein, the program instructions being executable by a processor to cause the processor to: access a set of annotated data candidates defined by two or more feature categories; train a machine learning classifier based on the set of annotated data candidates, thereby iteratively updating internal parameters of the machine learning classifier; sort the two or more feature categories in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; and perform one or more electronic actions based on the two or more feature categories sorted in order of classification importance.

[0207] Clause 18: A computer program product according to any preceding clause, wherein the machine learning classifier is an XGBoost model, and wherein the internal parameters include a decision tree threshold of the XGBoost model or a decision tree position of the XGBoost model.

[0208] Clause 19: The computer program product of any preceding clause, wherein the one or more electronic actions include visually presenting the ranking of the two or more feature categories on an electronic display.

[0209] Clause 20: The computer program product of any preceding clause, wherein the one or more electronic actions include visually presenting on an electronic display an indication of which of the two or more feature categories are ranked below a threshold value.

[0210] In various embodiments, any suitable combination of clauses 17 to 20 may be implemented.

Claims

1. A system, comprising: A processor that executes computer executable components stored in a computer readable memory, the computer executable components comprising: a receiver component having access to a set of annotated data candidates defined by two or more feature categories; A training component, wherein the training component trains a machine learning classifier according to the set of annotated data candidates, so that internal parameters of the machine learning classifier are iteratively updated; a feature component that sorts the two or more feature categories in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; and An execution component executes one or more electronic actions based on the two or more feature categories ranked in order of classification importance.

2. The system of claim 1, wherein the machine learning classifier is an XGBoost model, and wherein the internal parameters include a decision tree threshold of the XGBoost model or a decision tree position of the XGBoost model. 3 . The system of claim 1 , wherein the one or more electronic actions include visually presenting a ranking of the two or more feature categories on an electronic display. 4 . The system of claim 1 , wherein the one or more electronic actions include visually presenting on an electronic display an indication of which of the two or more feature categories are ranked below a threshold value.

5. The system of claim 1 , wherein the one or more electronic actions include identifying which of the two or more feature categories are ranked below a threshold and removing such feature categories from the set of annotated data candidates, thereby producing an updated set of annotated data candidates, and wherein the training component retrains the machine learning classifier based on the updated set of annotated data candidates.

6. The system of claim 1 , wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to not show up for the scheduled medical appointment.

7. The system of claim 1 , wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to be late for the scheduled medical appointment.

8. The system of claim 1 , wherein each annotated data candidate in the set of annotated data candidates belongs to a respectively corresponding medical patient having a respectively corresponding scheduled medical appointment, and wherein the two or more feature categories include a patient demographic feature category, a patient medical record feature category, a weather feature category, and a traffic feature category.

9. A computer-implemented method, the computer-implemented method comprising: accessing, by a device operatively coupled to the processor, a set of annotated data candidates defined by two or more feature categories; Training a machine learning classifier according to the set of annotated data candidates by the device, so that internal parameters of the machine learning classifier are iteratively updated; sorting, by the device, the two or more feature categories in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; as well as One or more electronic actions are performed, by the device, based on the two or more feature categories ranked in order of categorical importance.

10. The computer-implemented method of claim 9, wherein the machine learning classifier is an XGBoost model, and wherein the internal parameters include a decision tree threshold of the XGBoost model or a decision tree position of the XGBoost model.

11. The computer-implemented method of claim 9, wherein the one or more electronic actions include visually presenting, via the device and on an electronic display, a ranking of the two or more feature categories.

12. The computer-implemented method of claim 9, wherein the one or more electronic actions include visually presenting, by the device and on an electronic display, an indication of which of the two or more feature categories are ranked below a threshold.

13. The computer-implemented method of claim 9, wherein the one or more electronic actions include: identifying, by the device, which of the two or more feature categories are ranked below a threshold; removing, by the device, such feature categories from the set of annotated data candidates, thereby generating an updated set of annotated data candidates; as well as The machine learning classifier is retrained, by the device, based on the set of updated annotated data candidates.

14. The computer-implemented method of claim 9, wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to not show up for the scheduled medical appointment.

15. The computer-implemented method of claim 9, wherein the machine learning classifier is configured to receive as input a data candidate associated with a medical patient having a scheduled medical appointment, wherein the machine learning classifier is configured to generate as output a classification label based on the data candidate, and wherein the classification label indicates whether the medical patient is likely to be late for the scheduled medical appointment.

16. The computer-implemented method of claim 9, wherein each annotated data candidate in the set of annotated data candidates belongs to a respectively corresponding medical patient having a respectively corresponding scheduled medical appointment, and wherein the two or more feature categories include a patient demographic feature category, a patient medical record feature category, a weather feature category, and a traffic feature category.

17. A computer program product for facilitating automated training of a machine learning classification for missed or delayed patient care opportunities, the computer program product comprising a computer readable memory having program instructions embodied therein, the program instructions executable by a processor to cause the processor to: accessing a set of annotated data candidates defined by two or more feature categories; Training a machine learning classifier based on the set of annotated data candidates, thereby iteratively updating internal parameters of the machine learning classifier; sorting the two or more feature categories in order of classification importance based on the iteratively updated internal parameters of the machine learning classifier; as well as One or more electronic actions are performed based on the two or more feature categories ranked in order of categorical importance.

18. The computer program product of claim 17, wherein the machine learning classifier is an XGBoost model, and wherein the internal parameters include a decision tree threshold of the XGBoost model or a decision tree position of the XGBoost model.

19. The computer program product of claim 17, wherein the one or more electronic actions include visually presenting the ranking of the two or more feature categories on an electronic display.

20. The computer program product of claim 17, wherein the one or more electronic actions include visually presenting on an electronic display an indication of which of the two or more feature categories are ranked below a threshold value.

Citation Information

Cited By

  • Nuclear magnetic resonance reservation late prediction method and device

    CN119580980A