Authenticated customization of machine learning models

Abstract

Techniques for providing authenticated model customization for machine learning models are disclosed. A cloud service provider platform accesses a message including at least timestamp data and user identification data. A training group of data entities is identified based on data in the message. A training data set is determined based on the data entities of the training group. The machine learning model is modified based on the training data set. The modified machine learning model is provided during an authenticated network session associated with the user identification data. In some embodiments, modifications to the machine learning model are removed based on a determination that the authenticated network session has ended.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefits and priority of U.S. Provisional Application No. 63 / 583,214, filed September 15, 2023, and U.S. Provisional Application No. 63 / 583,234, filed September 15, 2023, the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0003] Clinical settings, such as healthcare facilities, typically involve different healthcare providers working collaboratively and communicating with each other to treat patients. Documenting patient contacts, capturing information communicated during and / or about events that occurred before and / or after these contacts, populating patient records such as electronic health records, and managing healthcare practices are integral parts of the practices of many healthcare providers and are crucial for ensuring high-quality healthcare. Traditional means of performing tasks associated with providing healthcare typically involve several different devices, such as listening devices, portable electronic devices, workstations, etc., and end-users with the training, knowledge, experience, and skills to properly utilize these devices and participate in healthcare processes. Relying on different devices and qualified end-users to perform clinical tasks is cumbersome, time- and resource-intensive, expensive, and inefficient, potentially leading to lower-quality healthcare.

[0004] In some approaches, artificial intelligence (AI)-based models are incorporated into digital assistance tools. In others, AI-based models have been used in healthcare settings to facilitate the care and management of patient populations. However, these models often do not perform as expected in the target environment, such as a healthcare setting. Building AI-based models for a target environment is a challenging task, as these models require domain-specific knowledge (e.g., healthcare information) and the application of certain technologies that may be relevant to the target environment. Therefore, building AI-based models for a target domain may be desirable. Summary of the Invention

[0005] This article discloses techniques for providing authenticated customizations of machine learning models.

[0006] In some embodiments, a computer-implemented method includes accessing a message. The message includes timestamp data and user identification data. The computer-implemented method includes identifying a training set of data entities including patient information. Each data entity in the training set is included in an appointment dataset associated with the user identification data. Each data entity in the training set includes appointment time data within a time window based on the timestamp data. The computer-implemented method includes determining at least one training dataset. The at least one training dataset is determined based on the data entities in the training set. The computer-implemented method includes modifying at least one pre-trained machine learning model based on the at least one training dataset. The computer-implemented method includes providing at least one modified pre-trained machine learning model during an authenticated network session associated with the user identification data.

[0007] In some embodiments, the computer-implemented method includes accessing an additional message. The additional message indicates the termination of an authenticated network session associated with user identification data. The computer-implemented method also includes removing modifications based on at least one training dataset. These modifications are removed from at least one pre-trained machine learning model.

[0008] In some embodiments, the data entities of the training group include secure data entities. An authenticated network session grants access to patient information included in the secure data entities. This access is granted to a client device authenticated via user identification data.

[0009] In some embodiments, the message is generated in response to a client application running on a client device associated with user identification data. The message indicates that the client application has been granted access to computing resources.

[0010] In some embodiments, the time window includes a forward time window, a backward time window, or an extended time window. A forward time window includes a first time period following the timestamp data. A backward time window includes a second time period preceding the timestamp data. An extended time window includes the most recent appointment time data item.

[0011] In some embodiments, the computer-implemented method includes accessing additional messages during an authenticated network session. The additional messages include user identification data and modified timestamp data. The computer-implemented method includes determining an additional training dataset based on at least one additional data entity. The additional data entity is included in a reservation dataset associated with the user identification data. The additional data entity includes additional reservation time data within an additional time window based on the additional timestamp data. The computer-implemented method includes further modifying at least one pre-trained machine learning model based on the additional training dataset. The computer-implemented method includes providing at least one further modified pre-trained machine learning model during an authenticated network session associated with the user identification data.

[0012] In some embodiments, further modification of at least one pre-trained machine learning model includes removing modifications based on at least one training dataset. These modifications based on at least one training dataset are removed from at least one pre-trained machine learning model.

[0013] In some embodiments, identifying the data entities in the training set includes determining a set of data entities associated with user identification data. Identifying the data entities in the training set includes determining a first subset and a second subset of the set of data entities. Each data entity included in the first subset is included in the appointment dataset associated with the user identification data. Each data entity included in the second subset includes appointment time data within a time window. Identifying the data entities in the training set includes selecting a third subset of the set of data entities associated with the user identification data. Each data entity in the third subset is included in the first and second subsets. The data entities in the training set include each data entity included in the third subset.

[0014] In some embodiments, at least one pre-trained machine learning model includes a speech recognition pre-trained machine learning model. At least one training dataset includes a speech recognition training dataset. Determining the speech recognition training dataset includes extracting text data from data entities within the training dataset. The text data is associated with patient information included in the data entities within the training dataset. Determining the speech recognition training dataset includes generating a customized recognition vocabulary that includes the extracted text data. The speech recognition training dataset includes a customized recognition vocabulary. Modifying at least one pre-trained machine learning model based on at least one training dataset includes modifying the speech recognition pre-trained machine learning model based on the speech recognition training dataset.

[0015] In some embodiments, the computer-implemented method includes accessing an additional message. The additional message indicates the termination of an authenticated network session associated with user identification data. The computer-implemented method also includes removing a speech recognition training dataset that includes a customized recognition vocabulary. This speech recognition training dataset is removed from a pre-trained machine learning model for speech recognition.

[0016] In some embodiments, at least one pre-trained machine learning model includes a language pre-trained machine learning model. At least one training dataset includes a language training dataset. Determining the language training dataset includes identifying a set of data objects among the data entities in the training set. This set of data objects is associated with patient information included in the data entities of the training set. Determining the language training dataset includes modifying a corresponding searchable data entity for each specific data object in the set of data objects. The corresponding searchable data entity is modified to include the specific data object. The language training dataset includes a corresponding searchable data entity for each specific data object in the set of data objects. Modifying at least one pre-trained machine learning model based on at least one training dataset includes modifying the language pre-trained machine learning model based on the language training dataset.

[0017] In some embodiments, the computer-implemented method includes accessing an additional message. The additional message indicates the termination of an authenticated network session associated with user identification data. The computer-implemented method also includes removing a language training dataset comprising a corresponding searchable data entity for each specific data object in the set of data objects. This language training dataset is removed from a language pre-trained machine learning model.

[0018] Some embodiments include a system comprising one or more processing systems and one or more computer-readable media storing instructions that, when executed by the one or more processing systems, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0019] Some embodiments include one or more non-transitory computer-readable media storing instructions that, when executed by one or more processing systems, cause the systems to perform some or all of the operations and / or methods disclosed herein.

[0020] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a few of the many implementations and contexts. Attached Figure Description

[0021] The features, embodiments, and advantages of this disclosure will be better understood when reading the following detailed description with reference to the accompanying drawings.

[0022] Figure 1 It is a high-level diagram depicting examples of computing environments, including the ability to provide customized digital aids, according to certain embodiments.

[0023] Figure 2 A simplified block diagram of a computing environment including a cloud service provider platform, configured to provide certified model customization for machine learning models, is depicted according to certain embodiments.

[0024] Figure 3 A simplified block diagram of a computing environment including a cloud service provider platform, configured to provide certified model customization for pre-trained machine learning models, is depicted according to certain embodiments.

[0025] Figure 4 An example processing flow for generating certified, customized models for machine learning, according to certain embodiments, is described.

[0026] Figure 5 Example processing flows for modifying certified custom-designed machine learning models, according to certain embodiments, are described.

[0027] Figure 6 An example processing flow, according to certain embodiments, is described for determining a set of data entities that can be identified from which a certified, customized training set for a machine learning model can be determined.

[0028] Figure 7 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0029] Figure 8 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0030] Figure 9 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0031] Figure 10 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0032] Figure 11 This is a block diagram illustrating an example computer system according to certain embodiments. Detailed Implementation

[0033] In the following description, specific details are set forth for illustrative purposes in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word “exemplary” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0034] introduce

[0035] Many healthcare providers use digital assistive tools to improve patient care, such as those for transcribing notes or verbal conversations, identifying potential diagnoses, accessing patients' electronic health records, scheduling appointments or procedures, and other types of digital assistive tools. Because digital assistive tools are used in various aspects of patient care, some can be improved based on a variety of types of information, such as by including different types of information in some cases (e.g., improving efficiency, improving accuracy, etc.). For example, a digital assistive tool for reducing scheduling conflicts might be improved based on information indicating appointment duration or procedure preparation time, while a digital assistive tool for automated speech recognition might be improved by indicating pronunciation or the spelling of the patient's name. In some cases, healthcare providers using multiple digital assistive tools expect each tool to provide a high-quality service (e.g., high accuracy, high efficiency, etc.) without wanting to spend time or effort determining the specific information available for each particular digital assistive tool. Furthermore, various digital assistive tools can utilize information in different ways, resulting in a variety of techniques for training, updating, or otherwise modifying digital assistive tools. In some cases, operators of healthcare computing environments may spend significant resources (e.g., time, money, etc.) to coordinate the training, updating, or other modifications of multiple digital assistive tools used within the healthcare computing environment. There is a need to provide high-quality digital assistive tools for healthcare providers and other professionals, tools that do not burden end users with the effort required to maintain high-quality service levels.

[0036] Some traditional techniques use machine learning training to modify digital assistive tools, such as the periodic training of machine learning models included in such tools. However, contemporary techniques for training machine learning can be excessively time-consuming or computationally resource-intensive, leading to reduced practical utility. For example, a healthcare computing environment comprising a large number of electronic health records (e.g., tens of thousands or more) may not be able to dedicate the time or computational resources (e.g., processing power) to updating the training of the machine learning model included in a digital assistive tool using all of these records. In some scenarios, such as healthcare computing environments serving emergency rooms or other 24-hour patient care facilities, traditional periodic training of machine learning models may be impractical because removing access to the digital assistive tool during training could negatively impact patient care. Furthermore, reducing the frequency of traditional periodic training of machine learning models can decrease the usefulness of digital assistive tools. For instance, a digital assistive tool that is a week behind in training (e.g., has not yet been updated to include updates to the patient's electronic health records from the previous week) could negatively impact patient care, such as failing to incorporate recent changes in the patient's condition.

[0037] Furthermore, conventional techniques used for machine learning training can compromise the security of protected information. For example, digital assistive tools may include machine learning models trained using contemporary technologies, such as data from large amounts of electronic health records. However, contemporary machine learning training techniques can negatively impact the security of protected patient information included in electronic health records, such as by exposing protected patient information to any user of a machine learning model trained with contemporary technologies. Security compromises can also occur in secure computing environments. For example, a secure healthcare computing environment may include digital assistive tools accessible to multiple authorized users within that environment. However, these authorized users may not be authorized to access all protected information within that secure healthcare computing environment; for example, various authorized users may be authorized to access different portions of protected information in different electronic health records. In this example, training a machine learning model in a digital assistive tool using contemporary machine learning training techniques may inadvertently expose specific portions of protected information to one or more authorized users who are not authorized to view that specific portion of the protected information.

[0038] Therefore, it may be desirable to improve traditional techniques for machine learning training by providing certified, customized techniques for machine learning models.

[0039] The methods described herein address these and other challenges by providing techniques for generating certified model customizations and / or applying certified model customizations to machine learning models. In various embodiments, a computer-implemented method includes accessing a message. The message includes timestamp data and user identification data. The computer-implemented method includes identifying a training set of data entities including patient information. Each data entity in the training set is included in an appointment dataset associated with user identification data. Each data entity in the training set includes appointment time data within a time window based on timestamp data. The computer-implemented method includes determining at least one training dataset. The at least one training dataset is determined based on the data entities in the training set. The computer-implemented method includes modifying at least one machine learning model based on at least one training dataset. The computer-implemented method includes providing at least one modified machine learning model during a certified network session associated with user identification data.

[0040] Certified Customization Technology

[0041] Figure 1 This is an example of a computing environment 100 that includes the capability to provide various services to users. End users (e.g., clinicians, such as doctors and nurses) can utilize the various services provided by the cloud service provider platform 110 to perform various functions. Figure 1 In this computing environment 100, a cloud service provider platform 110, at least one data repository (such as an electronic record database 105), and at least one client device (such as client device 190 and client device 180). The cloud service provider platform 110 includes the capability to provide various services, including cloud services, to its subscribers (e.g., end users). Within the computing environment 100, the cloud service provider platform 110 provides one or more services, including cloud services, to additional computing systems included in (or communicating with) the computing environment 100. For example, the cloud service provider platform 110 may provide one or more digital services to client devices 190 and 180.

[0042] The services provided by the cloud service provider platform 110 may include, but are not limited to, digital assistant services, authentication services, user management services, front-end services (e.g., a single entry point to all services), and other management services. Various services can be implemented on one or more servers of the cloud service provider platform 110 using one or more machine learning models, such as machine learning model 120. Furthermore, various services can be provided to end users who subscribe to services provided by platform 110. In one implementation, the services provided by the cloud service provider platform 110 can be implemented as machine learning-based or artificial intelligence (AI)-based digital assistance tools that can be provided to end users. For example, the cloud service provider platform 110 can provide one or more digital assistance tools that utilize one or more machine learning models, such as machine learning model 120. In some cases, machine learning model 120 can be used for voice services, where it implements automatic speech recognition (ASR) technology to convert audio-based content into text. Using ASR technology, humans can communicate with a computer interface using their voice in a manner similar to actual human conversation. In some cases, machine learning model 120 can be used in a dictation service where it implements Large Language Model (LLM) techniques to generate text (e.g., summaries, notes, etc.) from input data. Machine learning model 120 can be any kind of machine learning model that facilitates the provision of various services by the cloud service provider platform 110. Examples of machine learning models include, but are not limited to: pre-trained machine learning models, open-source machine learning models, licensed machine learning models, generative machine learning models, Transformer-based machine learning models, etc. In some cases, machine learning model 120 (or another service provided by the cloud service provider platform 110) can be configured as an AI-driven conversational interface for platform 110, such as a conversational interface service that can converse with end users (e.g., those using client devices 190 or 180) and perform functions and / or tasks based on information conveyed and / or determined from those conversations and other sources. Example conversational interface services can be configured with and / or be configured to access Natural Language Understanding (NLU) capabilities, such as Natural Language Processing, Named Entity Recognition, Intent Classification, etc. In some implementations, the example conversational interface service may be skill-driven, wherein the example service includes bots, each comprising one or more skills for engaging in conversation and performing functions and / or tasks. In some implementations, the example conversational interface service may be LLM-based and agent-driven, wherein one or more agents coordinate with one or more LLMs to engage in conversation and perform functions and / or tasks.Examples of skill-driven, LLM-based, and agent-driven digital assistants are described in U.S. Patent Application No. 17,648,376, filed January 19, 2022, and U.S. Patent Application No. 18 / 624,472, filed April 2, 2024, each of which is incorporated herein by reference as if fully set forth herein.

[0043] Various end users can interact with the cloud service provider platform 110 using one or more client devices (e.g., 190, 180), which can be communicatively coupled to one or more servers associated with the cloud service provider platform 110 via one or more communication channels. Examples of client devices may include mobile phones, tablets, desktop computers, or other suitable client devices capable of digital communication with the cloud service provider platform 110. Users can interact with various services through the user interface (UI) of applications installed on client devices 190 or 180, such as client application 195 installed on client device 190 or client application 185 installed on client device 180.

[0044] The computing environment 100 additionally includes an electronic record database 105. The electronic record database 105 may be a storage device, such as a physical or cloud-based storage device communicating with the cloud service provider platform 110. The electronic record database 105 may be configured to store electronic information. Each electronic record may be linked to other electronic records. In some cases, the cloud service provider platform 110 and the electronic record database 105 may be configured to securely store or transmit protected electronic information. Examples of protected information may include health information, financial information, personal information, or other types of protected information. Examples of secure storage or transmission of electronic information may include encryption, hashing, randomization, access authentication, or other technologies used to protect electronic information.

[0045] Figure 1 The computing environment 100 depicted herein is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the computing environment 100 may use a larger... Figure 1 The services shown can be implemented with more or different services, can be combined with two or more services, or can have different service configurations or arrangements.

[0046] In some cases, computing environment 100 is a healthcare computing environment, such as a healthcare computing environment configured for use by healthcare providers within healthcare settings (e.g., hospitals, medical practices, emergency care facilities, outpatient facilities, etc.). Figure 1In this context, computing environment 100 can be configured to assist multiple end users, who are healthcare providers, such as in assisting with the care and treatment of multiple patients. The term healthcare provider typically refers to healthcare practitioners and professionals, including but not limited to: physicians (e.g., general practitioners, specialists, surgeons, etc.); nursing professionals (e.g., nurse practitioners, physician assistants, nurses, registered nurses, licensed intern nurses, etc.); and other professionals (e.g., pharmacists, therapists, technicians, pathologists, dietitians, nutritionists, emergency medical technicians, psychiatrists, psychologists, counselors, dentists, orthodontists, hygienists, etc.). For example, within computing environment 100, end users can utilize the functions provided by the services of cloud service provider platform 110 to view, edit, or manage patients' electronic health records; examine or treat patients; facilitate patient treatment (e.g., recording doctor-patient interactions, generating medication or laboratory orders, etc.); perform administrative tasks such as scheduling appointments; manage patient groups; provide customer service to facilitate the operation of computing environment 100, and so on.

[0047] In computing environment 100, a patient may be associated with one or more electronic health records, such as those stored in electronic record database 105. Each electronic health record associated with a patient may be linked to other electronic health records associated with that patient. In some embodiments, electronic record database 105 stores electronic health records as one or more data entities, such as database records, digital files, or other types of data objects. For example, electronic record database 105 stores a particular electronic health record as one or more data entities describing the health information of a patient associated with that particular electronic health record, such as data entities describing personal information, appointment information, medical procedures (e.g., the process of scheduling, the process of completion, etc.), diagnoses, prescriptions, notes from a healthcare provider, or other types of digitally stored health information. In some embodiments, electronic record database 105 includes one or more electronic health records that are digitally secure data entities, such as electronic health records that include protected information about the associated patient. Examples of protected information may include a patient’s personal identifiers (e.g., name, address, etc.), a patient’s health information (e.g., diagnosis, scheduling process, provider notes, prescriptions, etc.), a patient’s financial information (e.g., insurance data, payment plan data, etc.), or other types of patient-related information identified as protected information in computing environment 100.

[0048] In a healthcare setting where computing environment 100 can be used, dialogue is ubiquitous among the participants in the healthcare setting. For example, at any given moment, there may be dialogue or conversation between a patient and a healthcare provider, between multiple healthcare providers, or between other groups of participants in the example healthcare setting. In some embodiments, digital entities may participate in dialogues such as between a healthcare provider and one or more digital assistive tools (e.g., diagnostic digital assistive tools, record-keeping digital assistive tools, etc.). In some cases, the dialogue in the example healthcare setting involves information related to one or more electronic health records, such as information about the care, treatment, observation, and diagnosis of one or more patients. In some other cases, the dialogue involves the logistics of patient care, such as when the physician's appointment is scheduled, what procedures the nurse will perform, what information about a particular patient is relevant to a particular appointment, and so on. In some embodiments, AI digital assistive tools may participate in the dialogue in the example healthcare setting (or provide other forms of healthcare). For example, a healthcare provider such as an internist may use an AI-based tool to automatically organize record notes from appointments with patients. In another example, a healthcare provider such as a laboratory technician may use an AI-based tool to automatically archive laboratory results (e.g., blood tests) in a patient's electronic health record. In yet another example, an AI-based tool can automatically generate a health summary report for each patient with an appointment with a healthcare professional on a specific date, allowing healthcare professionals to quickly review the medical information of each patient scheduled for that day.

[0049] exist Figure 1In this computing environment 100, the computing environment 100 includes the capability to protect patient information (such as secure electronic health records in an electronic records database 105) while providing one or more digital assistive tools to multiple healthcare providers. Examples of digital assistive tools may include authentication services, ASR transcription services (e.g., automatic speech-to-text transcription of conversations between patients or healthcare providers), LLM dictation services, or other types of digital assistive tools. In the computing environment 100, a cloud service provider platform 110 includes one or more machine learning models, such as machine learning model 120. In some cases, machine learning model 120 is a pre-trained machine learning model. Furthermore, the cloud service provider platform 110 provides digital assistive tools to healthcare providers via one or more additional computing systems (such as client devices 190 or 180). For example, each of client devices 190 and 180 is associated with a corresponding healthcare provider (such as a specific person using a particular client device). Additionally, each of client devices 190 and 180 (e.g., on behalf of a corresponding healthcare provider) accesses the digital assistive tools provided by the cloud service provider platform 110.

[0050] In some cases, one or more digital assistive tools are accessed via client applications operating on client devices within computing environment 100. For example, client device 190 includes client application 195, and client device 180 includes client application 185. Each of client applications 195 and 185 is configured to access one or more digital assistive tools, such as machine learning model 120, provided by cloud service provider platform 110.

[0051] Examples of healthcare providers may include doctors, nurses, technicians, pharmacists, or any other healthcare personnel, such as those interacting with the cloud service provider platform 110 via client devices 190 or 180. In an example scenario, a healthcare provider may use client devices 190 or 180 to view, update, or otherwise access the electronic health records of patients they care for. Examples of client devices, such as devices 190 or 180, may include electronic devices such as mobile phones, tablets, workstations, and other computing devices used to view, edit, and otherwise access patients' electronic health records.

[0052] In some embodiments, each of client devices 190 and 180 is utilized by a corresponding healthcare provider, such as a specific mobile computing device associated with a particular healthcare provider in computing environment 100. In this example scenario, the same healthcare provider utilizes the same client device, such as bringing the specific mobile computing device to each examination room (or other healthcare location) with appointments with multiple patients. In additional or alternative embodiments, each of client devices 190 and 180 is utilized by multiple healthcare providers, such as a specific desktop computing device associated with a particular healthcare resource (e.g., location, medical equipment, etc.). In this additional example scenario, multiple healthcare providers utilize the same client device, such as multiple healthcare providers utilizing a fixed (or largely fixed) computing workstation included in (or otherwise associated with) a particular X-ray machine.

[0053] In computing environment 100, cloud service provider platform 110 is configured to require authentication, such as user identification information, from each healthcare provider utilizing one or more of client devices 190 or 180. For example, if one or more of client devices 190 or 180 are utilized by a specific associated healthcare provider, then cloud service provider platform 110 may repeatedly (e.g., hourly, each time a healthcare provider moves to a new location in the example healthcare setup, etc.) require authentication from the associated healthcare provider. As an additional example, if one or more of client devices 190 or 180 are utilized by multiple healthcare providers, then cloud service provider platform 110 may require appropriate authentication from each specific healthcare provider utilizing that client device. Example technologies used for authentication may include username / password combinations, biometric data, multi-factor security technologies, or other types of technologies used to authenticate users or user devices attempting to access the computing system.

[0054] exist Figure 1In this computing environment 100, one or more digital security safeguards are included to prevent access to protected information, such as protected patient information in secure data entities stored in electronic records database 105, from unauthorized individuals, unauthorized computing systems, etc. For example, one or more of the cloud service provider platform 110 or electronic records database 105 are configured to require authentication from the user before allowing access to secure data entities in electronic records database 105. Furthermore, the cloud service provider platform 110 is configured to require authentication from the user before allowing access to machine learning model 120 or additional digital assistive tools provided by the cloud service provider platform 110. In some cases, the cloud service provider platform 110 (or additional computing systems in computing environment 100) enhances the security of protected patient information by requiring authentication from each user (or the client device used). In some cases, the cloud service provider platform 110 (or additional computing systems in computing environment 100) enhances the security of protected patient information by preventing specific authenticated users from accessing specific secure data entities that the authenticated user is not authorized to access. In some cases, authorization to access specific secure data entities is based on user roles or other characteristics. For example, an authorized user acting as a physician may be authorized (e.g., permitted) to access the electronic health records of patients receiving care from that physician, and may be prevented from accessing the electronic health records of additional patients not receiving care from that physician. In this example, an authorized physician may be authorized to access specific secure data entities in the electronic health records of patients she has assigned who describe medical health information (e.g., medical history), and may be prevented from accessing additional secure data entities describing financial health information (e.g., insurance availability). As an additional example, an authorized user acting as a scheduling specialist may be authorized to access specific secure data entities describing scheduling health information for any patient (e.g., appointments or procedures), and may be prevented from accessing additional secure data entities describing other types of health information (e.g., diagnoses, prescriptions, etc.). In some cases, cloud service provider platform 110 enhances the security of protected patient information within computing environment 100 by restricting authorized user access to electronic health records or secure data entities included in electronic health records. In computing environment 100, each specific authenticated user is allowed to access a specific electronic health record or a specific secure data entity that is authorized to that specific authenticated user, and each specific authenticated user is denied access to additional electronic health records or additional secure data entities that are not authorized to that specific authenticated user.

[0055] In an example healthcare setup, a healthcare provider utilizing one or more digital assistance tools provided by cloud service provider platform 110 may expect these tools to provide high-quality assistance in delivering patient care. Examples of high-quality assistance may include transcribing verbal conversations with high accuracy, identifying potential diagnoses with high accuracy, efficiently (e.g., quickly and accurately) accessing the correct electronic health records for a specific patient, or other types of digital assistance of sufficient quality to improve the care provided to the patient. For example, a healthcare provider utilizing machine learning model 120 may expect the machine learning model(s) used to be trained using recently available data, such as data including the most recent (e.g., same-day) updates to patient information. Training machine learning model 120 in computing environment 100 can improve the quality of patient care, such as increasing the accuracy of patient-specific details, such as the patient's name, diagnosis, prescriptions, or other patient information. Furthermore, digital security safeguards protecting patient information may prevent cloud service provider platform 110 from using patient data to train machine learning model 120, such as improving the protection of patient health information by preventing a specific user from accessing patient information that the specific user is not authorized to view via the trained model. In some cases, the existence of multiple objectives in training a machine learning model presents technical challenges within the computing environment 100. For example, a user of the computing environment 100 might wish to satisfy a first objective of improving the quality of assistance provided by the trained machine learning model, and also a second objective of preventing unauthorized access to patient information via the trained machine learning model. In some situations, these example objectives may conflict; for instance, improving patient information security may conflict with maximizing the quality of the machine learning model, which cannot be resolved using typical training techniques used for machine learning models.

[0056] In computing environment 100, cloud service provider platform 110 is configured to customize one or more trained machine learning models using custom data associated with a specific certified user. Additionally or alternatively, cloud service provider platform 110 is configured to remove the custom data associated with that specific certified user from one or more trained machine learning models, such as in response to determining that a specific certified user has deregistered from cloud service provider platform 110. In some embodiments, the described configuration of cloud service provider platform 110 provides a solution to an example conflict between the first and second objectives described above. For example, customizing the trained machine learning model using custom data associated with a specific certified user may satisfy the first objective, such as by improving the ability of the customized machine learning model to provide high-quality assistance in providing patient care. Furthermore, removing the custom data associated with a specific certified user may satisfy the second objective, such as by preventing additional certified users from accessing the custom data via the trained machine learning model.

[0057] exist Figure 1 In the computing environment 100, a first user can access one or more services provided by the cloud service provider platform 110 using client device 190 and client application 195. Similarly, a second user can access services provided by the cloud service provider platform 110 using client device 180 and client application 185. In the computing environment 100, the first and second users can each provide corresponding authentication data, such as corresponding user identifiers and password information, via client device 190 (e.g., for the first user) and client device 180 (e.g., for the second user), respectively. Based on the corresponding authentication data, the cloud service provider platform 110 determines that each of the first and second authenticated users is authorized to access the machine learning model 120. Furthermore, the cloud service provider platform 110 determines that each of the first and second users is authorized to access a corresponding portion of an electronic health record stored in the electronic record database 105. For example, the cloud service provider platform 110 determines that the first authenticated user is authorized to access the first electronic record group 104, and the second authenticated user is authorized to access the second electronic record group 106. In some cases, each of electronic record groups 104 and 106 includes a subset of secure data entities stored in the electronic record database 105. In computing environment 100, electronic record groups 104 and 106 are described as including different secure data entities, for example, a first authenticated user and a second authenticated user are authorized to access different electronic health records. However, other implementations are possible, such as multiple electronic health record groups (e.g., for multiple authenticated users) that include some, all, or no common electronic health records or secure data entities.

[0058] Based on the determination that the first and second authenticated users are authorized to access electronic record sets 104 and 106 respectively, the cloud service provider platform 110 identifies corresponding training data entities from electronic record sets 104 and 106. Furthermore, the cloud service provider platform 110 determines corresponding customized training datasets based on these training data entities. Figure 1 In this context, the cloud service provider platform 110 determines a specific custom training dataset associated with a particular authenticated user based on specific data entities that the authenticated user is authorized to access. For example, the cloud service provider platform 110 determines a first custom training dataset 114 associated with a first authenticated user by selecting one or more secure data entities that the first authenticated user is authorized to access from electronic record set 104. Furthermore, the cloud service provider platform 110 determines a second custom training dataset 116 associated with a second authenticated user by selecting one or more secure data entities that the second authenticated user is authorized to access from electronic record set 106.

[0059] In computing environment 100, cloud service provider platform 110 customizes machine learning model 120 based on customized training datasets 114 and 116. In some embodiments, the customization of machine learning model 120 is certified customization, such as being accessible to a specific certified user, for example, a specific customization accessible by a specific user via a specific client device or client application authenticated by the user to cloud service provider platform 110. Furthermore, cloud service provider platform 110 restricts access to the certified customization of machine learning model 120, for example, by preventing access by additional certified users not associated with a specific customization. For example, cloud service provider platform 110 implements a first certified customization of machine learning model 120 based on customized training dataset 114 associated with a first certified user. Additionally, cloud service provider platform 110 allows the first certified user to access the first certified customization of machine learning model 120 via client device 190 and client application 195, which provide authentication data. Furthermore, cloud service provider platform 110 denies access to the first certified customization of machine learning model 120 to one or more additional client devices or client applications (such as client device 180 and client application 185) not associated with the first certified user. As an additional example, based on a customized training dataset 116 associated with the second certified user, the cloud service provider platform 110 implements a second certified customization of the machine learning model 120. Furthermore, the cloud service provider platform 110 allows the second certified user to access the second certified customization of the machine learning model 120 via client device 180 and client application 185 that provide the authentication data. Additionally, the cloud service provider platform 110 denies access to the second certified customization of the machine learning model 120 to one or more additional client devices or client applications (such as client device 190 and client application 195) not associated with the second certified user.

[0060] In computing environment 100, cloud service provider platform 110 may remove first and second certified customizations of machine learning model 120 in response to receiving data indicating that a specific authenticated user is logging out of cloud service provider platform 110 (e.g., receiving user logout data, determining the timeout period of a user session, etc.). For example, in response to receiving first data indicating that client device 190 or client application 195 is terminating its session with cloud service provider platform 110 (e.g., a first user has logged out), cloud service provider platform 110 may remove the first certified customization from machine learning model 120. Furthermore, in response to receiving second data indicating that client device 180 or client application 185 is terminating its session with cloud service provider platform 110 (e.g., a second user has logged out), cloud service provider platform 110 may remove the second certified customization from machine learning model 120. In some embodiments, cloud service provider platform 110 may provide a non-customized implementation of machine learning model 120, such as a non-customized machine learning model accessible to unauthenticated users of cloud service provider platform 110 or authenticated users not associated with any secure data entity in electronic records database 105. In some embodiments, cloud service provider platform 110 may provide a non-customized implementation of machine learning model 120 during a portion of an authenticated network session, such as during a startup period when a user has started an authenticated network session with cloud service provider platform 110 and cloud service provider platform 110 has not yet determined an electronic health record or custom training dataset associated with the authenticated user.

[0061] In some cases, cloud service provider platform 110 provides improved digital assistive tools while enhancing the security of protected patient information. For example, based on custom training datasets 114 and 116, a custom machine learning model 120 provides high-quality assistance to first and second certified users, which incorporates patient information from specific secure data entities associated with the certified users, and protects (e.g., prevents access to) additional patient information from additional secure data entities not associated with the certified users.

[0062] Figure 2 An example of computing environment 200 is shown, in which cloud service provider platform 210 is configured to apply certified model customization to at least one machine learning model, such as machine learning model 220. In some embodiments, cloud service provider platform 210 is included in (or otherwise configured to communicate with) a healthcare computing environment such as computing environment 100. Figure 2In this system, computing environment 200 includes a cloud service provider platform 210, client devices 290, and an electronic health record database 205. Cloud service provider platform 210 includes a machine learning model 220. Client devices 290 include client applications 295 configured to access one or more digital assistance tools, such as machine learning models 220, provided by cloud service provider platform 210. Electronic health record database 205 includes one or more repositories of secure data entities, such as secure data entities describing protected patient information included in electronic health records. Figure 2 The electronic health record database 205 is described as including secure data entities describing protected patient information, but other implementations are possible. For example, a cloud service provider platform can be configured to communicate with a data repository (e.g., a database) that includes secure data entities describing additional protected information, such as protected personal information, protected financial information, protected employment information, or other types of protected information. Continuing this example, the cloud service provider platform can be configured to generate or apply certified model customizations based on secure data entities describing additional protected information.

[0063] In some embodiments, the machine learning model 220 is trained before use, such as pre-training performed before the cloud service provider platform 210 allows access to the machine learning model 220. In some embodiments, the machine learning model 220 includes base model data 225. Base model data 225 may include vocabulary data, entity data, parameter data, weight data, vector data, layer data, or other types of model data generated based on the pre-training of the machine learning model 220. In some embodiments, the machine learning model 220 is pre-trained based on unprotected information, such as publicly available training datasets, professional training datasets (e.g., training data related to a specific profession, such as medical terminology, local epidemiological data, educational diagnostic image sets, etc.), or other suitable training datasets excluding protected information and available to the cloud service provider platform 210 (or another computing system configured to train the machine learning model). Furthermore, base model data 225 is, for example, unprotected model data generated based on unprotected information during the pre-training of the machine learning model 220. Figure 2 In this context, machine learning model 220 may include one or more of a speech recognition machine learning model or a language machine learning model.

[0064] In some embodiments, cloud service provider platform 210 receives request data, such as message data 293, from client device 290. Message data 293 may include (or otherwise indicate) a request for access to one or more digital assistance tools provided by cloud service provider platform 210. In some cases, message data 293 includes authentication data, such as user identification data (e.g., username, password, biometric identifier, etc.) associated with a user of client device 290. Additionally, message data 293 includes timestamp data, such as timestamp data indicating the local time of client device 290. In some embodiments, message data 293 may include additional data associated with client device 290 or associated users, such as location data (e.g., identifying an examination room or other location of client device 290), data indicating additional users (e.g., multiple members of a surgical team or other care team), or other types of data associated with client device 290 or associated users. Figure 2 In this context, message data 293 is described as being received by cloud service provider platform 210 from client device 290, but other implementations are possible. For example, cloud service provider platform 210 may receive additional message data (e.g., describing a request from an additional client device) from at least one additional computing system or computing service (such as an authentication server, event handler service, or other types of computing systems or services that communicate with cloud service provider platform 210).

[0065] Based on message data 293, cloud service provider platform 210, such as in response to determining that the authentication data in message data 293 is valid (e.g., the user has successfully logged in), establishes an authenticated network session 250 (also referred to herein as "authenticated session 250") with client device 290. In computing environment 200, authenticated session 250 provides a secure communication channel between or among cloud service provider platform 210, client device 290, and client application 295. In some embodiments, cloud service provider platform 210 includes machine learning model 220 in authenticated session 250, such as in response to determining based on message data 293 that the user of client device 290 is authorized to access machine learning model 220.

[0066] In some embodiments, cloud service provider platform 210 generates (or otherwise determines) a custom training dataset 214 associated with client device 290 and accessible during authenticated session 250. For example, cloud service provider platform 210 determines, based on message data 293, that a user of client device 290 is authorized to access specific portions of secure data entities from electronic health record database 205, such as electronic health record group 204. Furthermore, cloud service provider platform 210 identifies a training set of secure data entities from electronic health record group 204, such as training group 224. Based on the secure data entities in training group 224, cloud service provider platform 210 determines the custom training dataset 214. For example, cloud service provider platform 210 evaluates the secure data entities in electronic health record group 204 based on one or more criteria, such as custom criterion 215. Based on this evaluation, cloud service provider platform 210 selects one or more secure data entities that meet custom criterion 215 and includes the selected secure data entities in training group 224. As an example in computing environment 200, cloud service provider platform 210 identifies training group 224 based on appointment criteria and multiple time criteria. For example, training group 224 may include data entities associated with patients who have scheduled appointments within a forward time window (e.g., during the next seven days) or a backward time window (e.g., during the previous seven days). Furthermore, cloud service provider platform 210 uses the secure data entities in training group 224 to generate a customized training dataset 214. In some cases, identifying one or more secure data entities for the training group based on customized criteria can improve the performance of certified model customization applied to machine learning models, such as by selecting data entities for patients most likely to be seen by a healthcare provider during the current time window. Examples of customized criteria may include appointment criteria (e.g., selecting data entities of patients with whom the user has scheduled appointments), time window criteria (e.g., selecting data entities of patients with appointments within a specific time window), patient criteria (e.g., selecting data entities of a specific patient or group of patients), location criteria (e.g., selecting data entities of patients with appointments at a specific office location), or other suitable criteria for selecting data entities in electronic health records or stored in electronic health records. In some cases, time window criteria may include forward time windows (e.g., a specific period of time in the future), backward time windows (e.g., a specific period of time in the past), extended time windows (e.g., extended to include the most recent appointment or a certain number of appointments) or other suitable time window criteria.

[0067] During the authenticated session 250, the cloud service provider platform 210 modifies the machine learning model 220 based on a custom training dataset 214. This modification may include authenticated model customization of the machine learning model 220. For example, the cloud service provider platform 210 or the machine learning model 220 may generate custom model data 227 using one or more secure data entities included in the custom training dataset 214. Custom model data 227 may include custom vocabulary data, custom entity data, custom parameter data, custom weight data, custom vector data, custom layer data, or other types of custom model data generated based on the custom training dataset 214. In some cases, the custom model data 227 includes a combination of the base model data 225 modified (or otherwise combined with) additional training data associated with the authenticated session 250. In the computing environment 200, the custom model data 227 is based on protected information associated with an authenticated user of the client device 290.

[0068] In some embodiments, customized model data 227 is available during the authenticated session 250. For example, a user associated with client device 290 can utilize a modified machine learning model 220 with customized model data 227. In some cases, the modified machine learning model 220 with customized model data 227 provides improved assistance to the user associated with client device 290 during the authenticated session 250. As an example, the modified machine learning model 220 can provide improved speech recognition of a user's patient's name, diagnosis, prescription, or other protected information. As an additional example, the modified machine learning model 220 can provide more accurate diagnostic assistance using current medical information (e.g., recent screening results, upcoming specialist appointments) for a specific patient with whom the user has an appointment.

[0069] In some embodiments, the customized model data 227 is unavailable outside of the certified session 250. For example, an additional client device excluded from the certified session 250 (e.g., client device 180) cannot access the customized training dataset 214 or the customized model data 227. As an additional example, after the certified session 250 terminates, client device 290 cannot access the customized training dataset 214 or the customized model data 227. For example, the cloud service provider platform 210 may receive additional message data indicating that the certified session 250 has ended (e.g., the user has logged out of client application 295, session 250 has reached its timeout limit, etc.). In response to determining that the certified session 250 has ended, the cloud service provider platform 210 removes the customized model data 227 from the machine learning model 220. In some embodiments, the cloud service provider platform 210 may delete or otherwise remove one or more of the training set 224 or the customized training dataset 214, such as when the certified session 250 terminates, when the certified customization of the machine learning model 220 is completed, or based on another suitable criterion.

[0070] In some cases, cloud service provider platform 210 removes customized model data 227 from machine learning model 220 in response to determining that additional authenticated customization is available to the associated user. For example, when authenticated session 250 terminates, cloud service provider platform 210 may, for instance, store customized model data 227 in secure storage that requires authentication by the associated user. Furthermore, in response to receiving additional message data initiating an additional authenticated session for the associated user (e.g., the user has logged in the next day), cloud service provider platform 210 may determine an additional customized training dataset, for instance, based on one or more modified customization criteria (e.g., a time window modified for the additional authenticated session). In some embodiments, cloud service provider platform 210 provides customized model data 227 during an initiation period, for example, providing the user with yesterday's authenticated customization before the additional authenticated customization becomes available.

[0071] exist Figure 2 In this context, the cloud service provider platform 210 can generate and / or apply specific certified model customizations for each specific certified client device accessing the machine learning model 220. In some cases, each specific client device accesses a corresponding (e.g., different) certified model customization of the machine learning model 220 within its respective certified session, such as a corresponding certified model customization generated based on corresponding customized training data associated with the specific certified client device.

[0072] In some embodiments, cloud service provider platform 210 may exclude machine learning model 220 from a specific authenticated network session, or include machine learning model 220 in a specific authenticated network session without applying authenticated model customization. For example, based on additional message data from an additional client device, cloud service provider platform 210 may determine that the additional message data does not include valid authentication data (e.g., the user of the additional client device has not successfully logged in). Continuing this example, cloud service provider platform 210 may prevent the additional client device from accessing machine learning model 220 (e.g., the user of the additional client device is not authorized to access machine learning model 220), or may allow the additional client device to access machine learning model 220 without authenticated model customization (e.g., the user of the additional client device is authorized to access machine learning model 220, but not authorized to access any secure data entities in the electronic health record database 205).

[0073] In some cases, a cloud service provider platform may include (or be otherwise configured to access) multiple machine learning models that can be modified through certified model customization. Furthermore, certified client devices may access one or more of these customized machine learning models during a certified network session. Figure 3 An example configuration 300 of computing environment 200 is shown, in which cloud service provider platform 210 is configured to apply certified model customization to machine learning model 220. Figure 3 In the example configuration shown, machine learning model 220 is or includes one or more of a pre-trained speech recognition machine learning model 330 or a pre-trained language machine learning model 340.

[0074] In example configuration 300, the pre-trained speech recognition machine learning model 330 includes basic recognition vocabulary model data 335, and the pre-trained language machine learning model 340 includes basic searchable entity model data 345. As generally described with respect to the basic model data 225, each of the basic recognition vocabulary model data 335 and the basic searchable entity model data 345 is, for example, unprotected model data generated based on unprotected information during the pre-training of each of the pre-trained speech recognition machine learning model 330 and the pre-trained language machine learning model 340. Figure 3 In this context, the basic recognition vocabulary model data 335 includes vocabulary data, such as vocabulary data that links sound data (e.g., indicating spoken words or phrases) with text data (e.g., indicating text words or phrases). Furthermore, the basic searchable entity model data 345 includes searchable entity data, such as a searchable entity space in which language-related data entities are embedded (e.g., vector representations of language data entities embedded in a vector space).

[0075] exist Figure 3 In response to receiving message data 293, cloud service provider platform 210 establishes an authenticated session 250 with client device 290, such as regarding Figure 2 As generally described. Furthermore, the cloud service provider platform 210 identifies a training group 224 from the electronic health record group 204 and determines a customized training dataset 214, as per [the description]. Figure 2 As generally described.

[0076] In example configuration 300, during an authenticated session 250, the cloud service provider platform 210 modifies one or more of a pre-trained speech recognition machine learning model 330 or a pre-trained language machine learning model 340 based on a custom training dataset 214. Modifications may include a first authenticated model customization for the pre-trained speech recognition machine learning model 330 and a second authenticated model customization for the pre-trained language machine learning model 340. For example, the cloud service provider platform 210 or the pre-trained speech recognition machine learning model 330 may generate a custom recognition vocabulary model data 337 using one or more secure data entities included in the custom training dataset 214. The custom recognition vocabulary model data 337 may include custom vocabulary data that links sound data with text data based on words or phrases included in the custom training dataset 214. In some cases, the custom recognition vocabulary model data 337 includes a combination of custom vocabulary data and base vocabulary data (e.g., from base recognition vocabulary model data 335). Furthermore, the cloud service provider platform 210 or the pre-trained language machine learning model 340 can generate custom searchable entity model data 347 using one or more secure data entities included in the custom training dataset 214. The custom searchable entity model data 347 may include custom searchable entity data associated with words or phrases included in the custom training dataset 214. In some cases, the custom searchable entity model data 347 includes a combination of custom searchable entity data and base searchable entity data, such as a custom searchable entity space embedded with language-related custom data entities from the custom training dataset 214 and language-related base data entities from pre-trained datasets.

[0077] In some embodiments, customized recognition vocabulary model data 337 and customized searchable entity model data 347 are available during the authenticated session 250. For example, a user associated with client device 290 may utilize one or more of a modified pre-trained speech recognition machine learning model 330 or a modified pre-trained language machine learning model 340. In some cases, the modified pre-trained speech recognition machine learning model 330 with customized recognition vocabulary model data 337 provides improved assistance to the user associated with client device 290 during the authenticated session 250, such as by performing patient-related speech recognition tasks with improved accuracy. For example, the modified pre-trained speech recognition machine learning model 330 may identify and transcribe (e.g., perform a speech-to-text task) a patient's name with improved accuracy based on the customized recognition vocabulary model data 337. Furthermore, the modified pre-trained language machine learning model 340 with customized searchable entity model data 347 provides improved assistance to the user associated with client device 290 during the authenticated session 250, such as by performing patient-related language analysis tasks with improved accuracy. For example, the modified pre-trained language machine learning model 340 can identify correlations between events in a patient's medical history with improved accuracy based on customized searchable entity model data 347.

[0078] In some embodiments, the customized recognition vocabulary model data 337 and the customized searchable entity model data 347 are unavailable outside of the authenticated session 250. For example, additional client devices excluded from the authenticated session 250 cannot access the customized recognition vocabulary model data 337 or the customized searchable entity model data 347. As an additional example, after the authenticated session 250 terminates, client device 290 cannot access the customized recognition vocabulary model data 337 or the customized searchable entity model data 347. For example, in response to determining that the authenticated session 250 has ended, the cloud service provider platform 210 removes the customized recognition vocabulary model data 337 from the pre-trained speech recognition machine learning model 330 and removes the customized searchable entity model data 347 from the pre-trained language machine learning model 340. In some cases, removing the customized recognition vocabulary model data 337 from the pre-trained speech recognition machine learning model 330 may include deleting the customized recognition vocabulary model data 337 or restoring one or more of the basic recognition vocabulary model data 335. Furthermore, removing customized searchable entity model data 347 from the pre-trained language machine learning model 340 may include deleting customized searchable entity model data 347 or restoring one or more of the base searchable entity model data 345.

[0079] (one or more) explanatory methods

[0080] Figure 4 A certified, customized processing flow 400 for generating machine learning models is described. Figure 4 The processing described herein can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 4 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 4 Various processing steps that occur in a specific sequence or order are described, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1-3 In the embodiments depicted, Figure 4 The processing described herein can be performed by a computing system (e.g., cloud service provider platform 110 or cloud service provider platform 210) to apply certified model customization to a machine learning model.

[0081] At box 405, the message is accessed by the computing system. In some cases, the computing system is as follows: Figure 1-3 The cloud service provider platform is described. Messages include at least timestamp data and user identification data. In some cases, the cloud service provider platform receives messages as data (such as request data or event data from a client device, authentication service, or another suitable computing system). In some cases, messages indicate a request for access to one or more digitally assisted tools provided by the cloud service provider platform, such as a request from a client device.

[0082] At box 410, a set of training data entities is identified based on data included in the message, such as timestamp data and user identification data. In this training set, each data entity satisfies a custom set of criteria, such as those associated with a cloud service provider platform, one or more digital assistance tools, or a user associated with the user identification data. In some cases, the cloud service provider platform generates the training set based on the selection of specific data entities that satisfy one or more custom criteria. The data entities in this training set may include secure data entities, such as data objects that include or otherwise represent protected information. In some cases, the cloud service provider platform generates the training set based on the selection of specific secure data entities authorized to access user identification data (or other data in the message).

[0083] At box 415, at least one training dataset is determined based on the data entities of the training group. The training dataset can be a custom training dataset associated with a machine learning model. The training dataset includes data entities extracted from or otherwise determined based on the data entities of the training group, such as secure data entities. For example, a cloud service provider platform generates a training dataset by determining one or more data entities or other types of data suitable for a custom machine learning model from the data entities of the training group. In some cases, the cloud service provider platform determines multiple training datasets, each associated with multiple machine learning models. For example, the cloud service provider platform can generate a speech recognition training dataset associated with a speech recognition machine learning model. Furthermore, the cloud service provider platform can generate a language training dataset associated with a language machine learning model.

[0084] At box 420, modify at least one machine learning model based on at least one training dataset. Modifying the machine learning model includes customization based on the training dataset. For example, a cloud service provider platform can modify a specific machine learning model by applying a specific customized training dataset, as per [reference to...]. Figure 2-3 As described. In some cases, cloud service provider platforms modify multiple machine learning models by applying multiple training datasets separately. For example, a cloud service provider platform can modify a speech recognition machine learning model by applying an associated speech recognition training dataset. Similarly, a cloud service provider platform can modify a language machine learning model by applying an associated language training dataset. In some embodiments, the cloud service provider platform can, for example, delete one or more data entities from the training dataset or that training set in response to determining that a particular customized training dataset has been applied to a particular machine learning model.

[0085] At box 425, at least one modified machine learning model is provided during an authenticated network session. Providing the modified machine learning model may include allowing access by at least one computing system included in the authenticated network session. For example, a cloud service provider platform may allow an authenticated client device associated with user identification data to access the modified machine learning model during the authenticated network session. In some cases, additional computing systems not included in the authenticated network session are denied access to the modified machine learning model, such as additional computing systems having additional authenticated network sessions not associated with user identification data.

[0086] At box 430, additional messages are accessed by the computing system, such as additional messages received by the cloud service provider platform. Additional messages include data that modifies the authenticated network session. In some cases, additional messages may include data indicating the end of the authenticated network session, such as data indicating that a user associated with user identification data has logged out of the authenticated network session.

[0087] At box 435, at least one modification is removed from at least one machine learning model. In some cases, removing a modification from a machine learning model is in response to determining that an authenticated network session has ended. For example, based on an additional message, a cloud service provider platform can determine that an authenticated network session associated with user identification data has ended. In response to determining that an authenticated network session has ended, the cloud service provider platform can remove a customized training dataset from a customized machine learning model. In some cases, the cloud service provider platform can remove a customized training dataset from a machine learning model in response to determining that a modified customized training dataset has been generated (e.g., a modified customized training dataset based on an additional authenticated network session update for the associated authenticated user). In some cases, the cloud service provider platform removes multiple modifications from multiple machine learning models. For example, the cloud service provider platform can remove a speech recognition training dataset from a speech recognition machine learning model. Furthermore, the cloud service provider platform can remove a language training dataset from a language machine learning model.

[0088] In some embodiments, training datasets removed from the machine learning model are deleted by the cloud service provider platform. In some embodiments, the training datasets removed from the machine learning model are stored by the cloud service provider platform, such as storing specific, customized training datasets for a particular user. For example, the cloud service provider platform may securely store the removed customized training datasets on a secure storage device that prevents access to the removed customized training datasets by additional computing systems (e.g., not associated with an authenticated network session). In some cases, the removed customized training datasets are retained for a specific period of time, such as until modifications to the machine learning model are completed, for a period of time after the user logs out (e.g., hours, minutes), until the user's shift ends (e.g., based on scheduling data associated with the user), or for another suitable period of time. In some cases, the cloud service provider platform deletes the removed customized training datasets when the specific period of time ends.

[0089] Figure 5 The document describes a certified custom processing flow 500 for modifying machine learning models. Figure 5The processing described herein can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 5 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 5 Various processing steps that occur in a specific sequence or order are described, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1-4 In the embodiments depicted, Figure 5 The processing described herein can be performed by a computing system (e.g., cloud service provider platform 110 or cloud service provider platform 210) to modify certified model customizations for machine learning models, such as modifications based on certified network sessions.

[0090] In some embodiments, regarding Figure 5 One or more of the operations described are related to... Figure 4 One or more operations in the described operations are executed in parallel, as alternatives to them, or as additional to them. For example, regarding Figure 4 The cloud service provider platform described can perform actions regarding access to digital assistive tools after receiving a first message requesting access (e.g., as described in box 405) and before receiving a second message indicating the end of an authenticated network session (e.g., as described in box 430). Figure 5 One or more of the operations described.

[0091] At box 505, the message is accessed by the computing system. In some embodiments, the computing system is as described above. Figure 1-4 The cloud service provider platform is described. In some cases, the access message is an additional message, such as in the context of... Figure 4 The additional messages received following the message described in the box. The cloud service provider platform determines that the additional messages include modifications to the authenticated network session (such as those related to...). Figure 4 The box describes user identification data associated with an authenticated network session. In some cases, the cloud service provider platform determines that additional messages include data that modifies and continues the authenticated network session (e.g., modifications that do not terminate the session).

[0092] In some cases, such as regarding Figure 4As described, additional messages may include data indicating the end of an authenticated network session, such as data indicating that a user associated with user identification data has logged out of the authenticated network session. As an additional example, additional messages may include data indicating modifications to the authenticated network session, such as modified timestamp data indicating a modified date (e.g., a user logged in during an overnight shift). Additional examples of data modifying an authenticated network session may include identification data indicating a modified computing system (e.g., a user has logged in on an additional client device), location data (e.g., a user has moved their logged-in client device to another inspection room), or other types of data indicating modifications to the authenticated network session.

[0093] At box 510, one or more additional data entities or additional training datasets are identified. For example, based on modified timestamp data included in the additional message, the cloud service provider platform identifies at least one additional data entity that meets one or more custom criteria, such as reservation criteria and time criteria based on modified time data. In some cases, the cloud service provider platform modifies data entities in a training set, such as those described with respect to box 410, to include additional data entities. Furthermore, the cloud service provider platform determines additional training datasets based on the additional data entities. For example, the cloud service provider platform may modify the training dataset described with respect to box 415 to include additional data entities. In some cases, the cloud service provider platform further modifies the data entities in the training set or the training dataset to omit specific data entities, such as removing data entities that no longer meet custom criteria based on modified timestamp data.

[0094] At box 515, at least one machine learning model is modified or further modified based on an additional training dataset. Modifying the machine learning model includes customization or further customization based on the additional training dataset. In some cases, the cloud service provider platform applies multiple training datasets to a specific machine learning model. For example, the cloud service provider platform may apply an additional training dataset to the modified machine learning model described with respect to box 420. In some cases, the cloud service provider platform removes previously applied training datasets from a specific machine learning model before applying additional training datasets. For example, the cloud service provider platform may remove training datasets from the modified machine learning model described with respect to box 420 before applying additional training datasets to the machine learning model.

[0095] At box 520, during an authenticated network session, such as the authenticated network session described with respect to box 425, a modified or further modified machine learning model is provided. For example, a cloud service provider platform may allow authenticated client devices associated with user identification data to access the further modified machine learning model during an authenticated network session. In some cases, additional computing systems not included in the authenticated network session are denied access to the further modified machine learning model.

[0096] Figure 6 A processing flow 600 is described for identifying data entities from which a certified, customized training set for a machine learning model can be derived. Figure 6 The processing described herein can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 6 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 6 Various processing steps that occur in a specific sequence or order are described, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1-5 In the embodiments depicted, Figure 6 The processing described herein can be performed by a computing system (e.g., cloud service provider platform 110 or cloud service provider platform 210) to determine a set of data entities based on which a certified model tailored for a machine learning model can be generated.

[0097] In some embodiments, regarding Figure 6 One or more of the operations described are related to... Figure 4-5 One or more operations in the described operations are executed in parallel, as alternatives to them, or as additional to them. For example, regarding Figure 4 The described cloud service provider platform can perform about Figure 6 One or more operations described in relation to identifying data entities in a training set (e.g., as described in box 410).

[0098] At box 605, a group of one or more data entities associated with the user identification data is identified. The identified data entities may be secure data entities, such as those extracted from (or otherwise determined from) one or more electronic health records. In some cases, the user identification data is included in messages received by the cloud service provider platform, such as those described with respect to box 405. For example, the cloud service provider platform may identify this group of data entities based on at least one electronic health record (such as the electronic health record of a patient of that healthcare provider) that is authorized to access by the healthcare provider associated with the user identification data.

[0099] At box 610, a first subset of the group of one or more data entities is identified. In some cases, the first subset is determined based on a first custom criterion. For example, a cloud service provider platform may determine the first subset by identifying which data entities in the group of data entities meet the first custom criterion. In some cases, the first custom criterion is one or more appointment criteria. For example, a cloud service provider platform may determine the first subset of data entities based on appointment criteria indicating that patients have scheduled appointments with healthcare providers and are associated with user identification data.

[0100] At box 615, a second subset of the group of one or more data entities is identified. In some cases, the second subset is determined based on a second custom criterion. For example, a cloud service provider platform can determine the second subset by identifying which data entities in the group meet the second custom criterion. In some cases, the second custom criterion is one or more time criteria. For example, a cloud service provider platform can determine the second subset of data entities based on a time criterion that indicates a specific time window associated with an appointment (such as a forward time window, a backward time window, an extended time window (e.g., determining the most recent n appointments), or includes another suitable time window for appointments with that healthcare provider).

[0101] In some embodiments, one or more additional subsets of the set of one or more data entities are determined based on one or more additional customization criteria. For example, a cloud service provider platform can determine additional subsets by identifying which data entities in the set of data entities meet the additional customization criteria.

[0102] At box 620, a third subset of the group of one or more data entities is determined. Furthermore, the third subset is selected from data entities included in the first and second subsets. In some cases, the third subset is selected from a combination of data entities included in the first subset, the second subset, and one or more additional subsets. For example, a cloud service provider platform selects the third subset based on one or more data entities included in both the first and second subsets (e.g., data entities that meet the first and second customization criteria). In some embodiments, the cloud service provider platform selects the third subset based on data entities included in all determined subsets (e.g., data entities that meet the first, second, and additional customization criteria).

[0103] At box 625, a training set of data entities is identified based on a third subset of one or more data entities in the group. In this training set, each data entity satisfies custom criteria evaluated by the cloud service provider platform, such as reservation criteria, time criteria, and additional custom criteria. For example, the cloud service provider platform identifies the data entities in the training set described in box 410 based on a third subset of the data entities.

[0104] Examples of cloud infrastructure

[0105] The term cloud service generally refers to services provided by a cloud service provider (CSP) to users (e.g., cloud service customers) on demand (e.g., via a subscription model) using systems and infrastructure (e.g., cloud infrastructure) provided by the CSP. Examples of CSPs may include cloud service provider platform 110 or cloud service provider platform 210. Typically, the servers and systems that make up the CSP's infrastructure are separate from the user's own on-premises servers and systems. Therefore, users can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for these services. Cloud services are designed to provide subscribers with simple, scalable access to applications and computing resources without requiring users to invest in the infrastructure used to deliver these services.

[0106] Several cloud service providers offer various types of cloud services. As discussed in this article, there are various types or models of cloud services, including Infrastructure as a Service (IaaS), Software as a Service (SaaS), Platform as a Service (PaaS), and more. Users can subscribe to one or more cloud services provided by a CSP. Users can be any entity, such as individuals, organizations, enterprises, etc. When a user subscribes to or registers for a service provided by a CSP, a lease or account is created for that user. The user can then access one or more subscribed cloud resources associated with that account.

[0107] As mentioned above, IaaS is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, since these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0108] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0109] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0110] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0111] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0112] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0113] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, there may also be one or more inbound / outbound traffic group rules, provisioned to define how inbound and / or outbound traffic will be configured for the network, and one or more Virtual Machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are desired.

[0114] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0115] Figure 7This is a block diagram 700 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 702 may be communicatively coupled to a secure host lease 704, which may include a virtual cloud network (VCN) 706 and a secure host subnet 708. In some examples, the service operator 702 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and enabled for the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device can be a workstation computer running various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as Google Chrome OS). Alternatively, or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without Kinect® gesture input), and / or a personal messaging device capable of communicating over a network that can access VCN 706 and / or the internet.

[0116] VCN 706 may include a local peering gateway (LPG) 710, which may be communicatively coupled to a secure shell (SSH) VCN 712 via the LPG 710 included in SSH VCN 712. SSH VCN 712 may include an SSH subnet 714, and SSH VCN 712 may be communicatively coupled to a control plane VCN 716 via the LPG 710 included in control plane VCN 716. Furthermore, SSH VCN 712 may be communicatively coupled to a data plane VCN 718 via the LPG 710. Control plane VCN 716 and data plane VCN 718 may be contained within a service lease 719 that may be owned and / or operated by an IaaS provider.

[0117] The control plane VCN 716 may include a control plane demilitarized zone (DMZ) layer 720 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 720 may include one or more load balancer (LB) subnets 722, a control plane application layer 724 that may include one or more application subnets 726, and a control plane data layer 728 that may include one or more database (DB) subnets 730 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 722 contained in the control plane DMZ layer 720 may be communicatively coupled to one or more application subnets 726 contained in the control plane application layer 724 and an Internet gateway 734 that may be contained in the control plane VCN 716. The application subnets 726 may be communicatively coupled to one or more DB subnets 730 contained in the control plane data layer 728, as well as a service gateway 736 and a Network Address Translation (NAT) gateway 738. The control plane VCN 716 may include the service gateway 736 and the NAT gateway 738.

[0118] The control plane VCN 716 may include a data plane mirror application layer 740, which may include one or more application subnets 726. The one or more application subnets 726 included in the data plane mirror application layer 740 may include a virtual network interface controller (VNIC) 742 capable of executing a compute instance 744. The compute instance 744 may communicatively couple the one or more application subnets 726 of the data plane mirror application layer 740 to the one or more application subnets 726 that may be included in the data plane application layer 746.

[0119] Data plane VCN 718 may include data plane application layer 746, data plane DMZ layer 748, and data plane data layer 750. Data plane DMZ layer 748 may include one or more LB subnets 722 communicatively coupled to one or more application subnets 726 of data plane application layer 746 and Internet gateway 734 of data plane VCN 718. One or more application subnets 726 may be communicatively coupled to service gateway 736 and NAT gateway 738 of data plane VCN 718. Data plane data layer 750 may also include one or more DB subnets 730 communicatively coupled to one or more application subnets 726 of data plane application layer 746.

[0120] The Internet gateway 734 of the control plane VCN 716 and data plane VCN 718 can be communicatively coupled to the metadata management service 752, which in turn can be communicatively coupled to the public Internet 754. The public Internet 754 can be communicatively coupled to the NAT gateway 738 of the control plane VCN 716 and data plane VCN 718. The service gateway 736 of the control plane VCN 716 and data plane VCN 718 can be communicatively coupled to the cloud service 756.

[0121] In some examples, the service gateway 736 of the control plane VCN 716 or data plane VCN 718 can make application programming interface (API) calls to the cloud service 756 without traversing the public internet 754. API calls from the service gateway 736 to the cloud service 756 can be unidirectional: the service gateway 736 can make API calls to the cloud service 756, and the cloud service 756 can send requested data to the service gateway 736. However, the cloud service 756 may not initiate API calls to the service gateway 736.

[0122] In some examples, secure host lease 704 can be directly connected to service lease 719, which would otherwise be isolated. Secure host subnet 708 can communicate with SSH subnet 714 via LPG 710, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 708 to SSH subnet 714 allows secure host subnet 708 to access other entities within service lease 719.

[0123] Control plane VCN 716 allows users of service lease 719 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 716 can be deployed or otherwise used in data plane VCN 718. In some examples, control plane VCN 716 can be isolated from data plane VCN 718, and the data plane mirror application layer 740 of control plane VCN 716 can communicate with the data plane application layer 746 of data plane VCN 718 via VNIC 742, which can be included in both the data plane mirror application layer 740 and the data plane application layer 746.

[0124] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 754, which can transmit requests to the metadata management service 752. The metadata management service 752 can transmit the request to the control plane VCN 716 via internet gateway 734. The request can be received by one or more LB subnets 722 contained in the control plane DMZ layer 720. The LB subnets 722 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 726 contained in the control plane application layer 724. If the request is validated and requires a call to the public internet 754, the call to the public internet 754 can be transmitted to a NAT gateway 738 that can make calls to the public internet 754. The request may expect the stored metadata to be stored in one or more DB subnets 730.

[0125] In some examples, the data plane mirroring application layer 740 can facilitate direct communication between the control plane VCN 716 and the data plane VCN 718. For example, it might be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 718. Through VNIC 742, the control plane VCN 716 can communicate directly with the resources contained in the data plane VCN 718, and thus can perform configuration changes, updates, or other appropriate modifications to them.

[0126] In some embodiments, the control plane VCN 716 and data plane VCN 718 may be contained within a service lease 719. In this case, the system's users or customers may not own or operate the control plane VCN 716 or data plane VCN 718. Alternatively, the IaaS provider may own or operate both the control plane VCN 716 and data plane VCN 718, and both planes may be contained within the service lease 719. This embodiment enables the isolation of networks that might prevent users or customers from interacting with resources from other users or customers. Moreover, this embodiment allows the system's users or customers to privately store databases without relying on the public internet 754, which may not have the desired level of threat prevention for storage.

[0127] In other embodiments, one or more LB subnets 722 included in the control plane VCN 716 may be configured to receive signals from the service gateway 736. In this embodiment, the control plane VCN 716 and the data plane VCN 718 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 754. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 719, which may be isolated from the public internet 754.

[0128] Figure 8 This is a block diagram 700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 802 (e.g., Figure 7 The service provider (702) can communicatively couple to the secure host lease (804) (e.g., Figure 7 Secure hosting lease 704), the secure hosting lease 804 may include a Virtual Cloud Network (VCN) 806 (e.g., Figure 7 VCN706) and Secure Host Subnet 808 (e.g., Figure 7 The secure host subnet 708). VCN 706 may include a local peering gateway (LPG) 810 (e.g., Figure 7 The LPG 710), which can be communicatively coupled to the Secure Shell (SSH) VCN 812 (e.g., via the LPG 810 contained in the SSH VCN 812) Figure 7 SSH VCN 712). SSH VCN 812 can include SSH subnet 814 (e.g., Figure 7 SSH subnet 714), and SSH VCN 812 can be communicatively coupled to control plane VCN 816 via LPG 810 included in control plane VCN 816 (e.g., Figure 7 Control plane VCN 716). Control plane VCN 816 may be included in service lease 819 (e.g., Figure 7 In the service lease 719), and the data plane VCN818 (e.g., Figure 7 The data plane VCN 718 may be included in a customer lease 821 that may be owned or operated by a user or customer of the system.

[0129] Control plane VCN 816 may include control plane DMZ layer 820 (e.g., Figure 7 The control plane DMZ layer 720), which may include one or more LB subnets 822 (e.g., Figure 7(one or more) LB subnets 722), may include (one or more) application subnets 826 (e.g., Figure 7 The control plane application layer 824 of (one or more) application subnets 726 (e.g., Figure 7 The control plane application layer 724) may include one or more database (DB) subnets 830 (e.g., similar to...). Figure 7 The control plane data layer 828 of (one or more) DB subnets 730 (e.g., Figure 7 The control plane data layer 728). One or more LB subnets 822 contained in the control plane DMZ layer 820 can be communicatively coupled to one or more application subnets 826 contained in the control plane application layer 824 and an Internet gateway 834 that can be contained in the control plane VCN 816 (e.g., Figure 7 Internet gateway 734), and application subnet(s) 826 can communicatively couple to DB subnet(s) 830 contained in control plane data layer 828 and service gateway 836 (e.g., Figure 7 Service gateway 736) and Network Address Translation (NAT) gateway 838 (e.g., Figure 7 (NAT gateway 738). The control plane VCN 816 may include the service gateway 836 and the NAT gateway 838.

[0130] The control plane VCN 816 may include a data plane mirror of the application layer 840, which may include one or more application subnets 826 (e.g., Figure 7 The data plane mirror application layer 740). One or more application subnets 826 contained in the data plane mirror application layer 840 may include compute instances 844 capable of performing computations (e.g., similar to...). Figure 7 The virtual network interface controller (VNIC) 842 (e.g., the VNIC of 742) of the computing instance 744. The computing instance 844 may facilitate the mirroring of the application subnet(s) 826 of the application layer 840 in the data plane and may be included in the application layer 846 in the data plane (e.g., Figure 7 Communication between one or more application subnets 826 in the data plane application layer 746 via VNIC 842 contained in the data plane mirror application layer 840 and VNIC 842 contained in the data plane application layer 846.

[0131] The Internet gateway 834 included in the control plane VCN 816 can be communicatively coupled to the metadata management service 852 (e.g., Figure 7 Metadata management service 752), which can communicatively couple to the public Internet 854 (e.g., Figure 7 The public internet 854 can communicatively couple to a NAT gateway 838 included in the control plane VCN 816. The service gateway 836 included in the control plane VCN 716 can communicatively couple to a cloud service 856 (e.g., ...). Figure 7 Cloud services 756).

[0132] In some examples, data plane VCN 818 may be included in customer lease 821. In this case, the IaaS provider may provide control plane VCN 816 for each customer, and the IaaS provider may set up a unique compute instance 844 for each customer, included in service lease 819. Each compute instance 844 may allow communication between control plane VCN 816 included in service lease 819 and data plane VCN 818 included in customer lease 821. Compute instance 844 may allow resources provisioned in control plane VCN 816 included in service lease 819 to be deployed or otherwise used in data plane VCN 818 included in customer lease 821.

[0133] In other examples, an IaaS provider's customer may have a database residing in customer lease 821. In this example, control plane VCN 816 may include data plane mirror application layer 840, which may include one or more application subnets 826. Data plane mirror application layer 840 may reside in data plane VCN 818, but may not reside in data plane VCN 818. That is, data plane mirror application layer 840 may have access to customer lease 821, but may not reside in data plane VCN 818 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 840 may be configured to invoke data plane VCN 818, but may not be configured to invoke any entity contained in control plane VCN 816. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 816 in the data plane VCN 818, and the data plane mirroring application layer 840 can facilitate the customer's desired deployment or other use of resources.

[0134] In some embodiments, an IaaS provider's customer can apply filters to data plane VCN 818. In this embodiment, the customer can determine what data plane VCN 818 can access, and the customer can restrict access from data plane VCN 818 to the public Internet 854. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 818's access to any external networks or databases. Applying filters and controls to data plane VCN 818 contained in customer lease 821 can help isolate data plane VCN 818 from other customers and the public Internet 854.

[0135] In some embodiments, cloud service 856 may be invoked by service gateway 836 to access services that may not exist on public internet 854, control plane VCN 816, or data plane VCN 818. The connection between cloud service 856 and control plane VCN 816 or data plane VCN 818 may not be real-time or continuous. Cloud service 856 may reside on different networks owned or operated by an IaaS provider. Cloud service 856 may be configured to receive calls from service gateway 836 and may be configured not to receive calls from public internet 854. Some cloud services 856 may be isolated from other cloud services 856, and control plane VCN 816 may be isolated from cloud services 856 that may not be in the same region as control plane VCN 816. For example, control plane VCN 816 may be located in "Region 1," and cloud service "Deployment 7" may be located in both "Region 1" and "Region 2." If the service gateway 836, contained in the control plane VCN 816 located in region 1, makes a call to deployment 7, then that call can be transmitted to deployment 7 in region 1. In this example, the control plane VCN 816 or deployment 7 in region 1 may be uncoupled from or communicate with deployment 7 in region 2.

[0136] Figure 9 This is a block diagram 900 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 902 (e.g., Figure 7 The service provider (702) can communicatively couple to the secure host lease (904) (e.g., Figure 7 Secure hosting lease 704), the secure hosting lease 904 may include a Virtual Cloud Network (VCN) 906 (e.g., Figure 7 VCN706) and Secure Host Subnet 908 (e.g., Figure 7 The secure host subnet 708). VCN 906 can include LPG 910 (e.g., Figure 7The LPG 710), which can be communicatively coupled to the SSH VCN 912 via the LPG 910 included in the SSH VCN 912 (e.g., Figure 7 SSH VCN 712). SSH VCN 912 can include SSH subnet 914 (e.g., Figure 7 SSH subnet 714), and SSH VCN 912 can be communicatively coupled to control plane VCN 916 via LPG 910 contained in control plane VCN 916 (e.g., Figure 7 The control plane VCN 716) and coupled to the data plane VCN 918 via the LPG 910 contained in the data plane VCN 918 (e.g., Figure 7 Data plane 718). Control plane VCN 916 and data plane VCN 918 may be included in service lease 919 (e.g., Figure 7 (Service rental 719).

[0137] The control plane VCN 916 may include one or more load balancer (LB) subnets 922 (e.g., Figure 7 The control plane DMZ layer 920 of (one or more) LB subnets 722) (e.g., Figure 7 The control plane DMZ layer 726 may include one or more application subnets 926 (e.g., similar to...). Figure 7 The control plane application layer 924 of (one or more) application subnets 726 (e.g., Figure 7 The control plane application layer 724), may include (one or more) DB subnets 930, and the control plane data layer 928 (e.g., Figure 7 The control plane data layer 728). One or more LB subnets 922 contained in the control plane DMZ layer 920 can be communicatively coupled to one or more application subnets 926 contained in the control plane application layer 924 and an Internet gateway 934 that can be contained in the control plane VCN 916 (e.g., Figure 7 Internet gateway 734), and application subnet 926(e.g.) can communicatively couple to DB subnet 930(e.g., contained in control plane data layer 928) and service gateway 936(e.g., Figure 7 The service gateway) and Network Address Translation (NAT) gateway 938 (e.g., Figure 7 (NAT gateway 738). The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0138] The data plane VCN 918 may include the data plane application layer 946 (e.g., Figure 7Data plane application layer 746), data plane DMZ layer 948 (e.g., Figure 7 Data plane DMZ layer 748), and data plane data layer 950 (e.g., Figure 7 The data plane data layer 750. The data plane DMZ layer 948 may include one or more trusted application subnets 960 and one or more untrusted application subnets 962 communicatively coupled to the data plane application layer 946, and one or more LB subnets 922 of the Internet gateway 934 contained in the data plane VCN 918. One or more trusted application subnets 960 may be communicatively coupled to the service gateway 936 contained in the data plane VCN 918, the NAT gateway 938 contained in the data plane VCN 918, and one or more DB subnets 930 contained in the data plane data layer 950. One or more untrusted application subnets 962 may be communicatively coupled to the service gateway 936 contained in the data plane VCN 918 and one or more DB subnets 930 contained in the data plane data layer 950. The data plane data layer 950 may include one or more DB subnets 930 communicatively coupled to the service gateway 936 contained in the data plane VCN 918.

[0139] One or more untrusted application subnets 962 may include one or more primary VNICs 964(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N). Each tenant VM 966(1)-(N) may be communicatively coupled to a corresponding application subnet 967(1)-(N) that may be contained in a corresponding container egress VCN 968(1)-(N), which may be contained in a corresponding customer lease 970(1)-(N). A corresponding secondary VNIC 972(1)-(N) may facilitate communication between one or more untrusted application subnets 962 contained in a data plane VCN 918 and the application subnets contained in the container egress VCN 968(1)-(N). Each container egress VCN 968(1)-(N) may include a NAT gateway 938 that can be communicatively coupled to the public Internet 954 (e.g., Figure 7 The public internet (754).

[0140] An Internet gateway 934, contained in the control plane VCN 916 and the data plane VCN 918, can be communicatively coupled to a metadata management service 952 (e.g., Figure 7Metadata management service 952 can be communicatively coupled to the public internet 954. The public internet 954 can be communicatively coupled to a NAT gateway 938 contained in a control plane VCN 916 and a data plane VCN 918. Service gateway 936 contained in the control plane VCN 916 and the data plane VCN 918 can be communicatively coupled to a cloud service 956.

[0141] In some embodiments, the data plane VCN 918 can be integrated with the customer lease 970. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider can determine whether to run the code provided by the customer to the IaaS provider.

[0142] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 946. The code running this functionality may execute in VMs 966(1)-(N) and may not be configured to run anywhere else on the data plane VCN 918. Each VM 966(1)-(N) may be connected to a customer lease 970. The corresponding container 971(1)-(N) contained in VMs 966(1)-(N) may be configured to run the code. In this case, dual isolation may exist (e.g., container 971(1)-(N) runs the code, where container 971(1)-(N) may be contained in at least one or more untrusted application subnets 962 containing VMs 966(1)-(N)), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Container 971(1)-(N) may be communicatively coupled to Customer Lease 970 and may be configured to transmit or receive data from Customer Lease 970. Container 971(1)-(N) may not be configured to transmit or receive data from any other entity in Data Plane VCN 918. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of Container 971(1)-(N).

[0143] In some embodiments, one or more trusted application subnets 960 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 960 may be communicatively coupled to one or more database subnets 930 and configured to perform CRUD operations in one or more database subnets 930. One or more untrusted application subnets 962 may be communicatively coupled to one or more database subnets 930, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 930. Containers 971(1)-(N) that may be contained in each customer's VM 966(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 930.

[0144] In other embodiments, the control plane VCN 916 and the data plane VCN 918 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 916 and the data plane VCN 918. However, communication can occur indirectly through at least one method. The LPG 910 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 916 and the data plane VCN 918. In another example, either the control plane VCN 916 or the data plane VCN 918 may invoke the cloud service 956 via the service gateway 936. For example, an invocation of the cloud service 956 from the control plane VCN 916 may include a request for a service that can communicate with the data plane VCN 918.

[0145] Figure 10 This is a block diagram 1000 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1002 (e.g., Figure 7 The service provider 702 can communicatively couple to the secure host lease 1004 (e.g., Figure 7 Secure hosting lease 704), the secure hosting lease 1004 may include a virtual cloud network (VCN) 1006 (e.g., Figure 7 VCN 706) and Secure Host Subnet 1008 (e.g., Figure 7 The secure host subnet 708). VCN 1006 may include LPG1010 (e.g., Figure 7 The LPG 710), the LPG 1010 can be accessed via SSH VCN 1012 (e.g., LPG 710), Figure 7 The LPG 1010 in SSH VCN 1012 is communicatively coupled to SSH VCN 1012. SSH VCN 1012 may include SSH subnet 1014 (e.g., Figure 7 SSH subnet 714), and SSH VCN 1012 can be communicatively coupled to control plane VCN 1016 via LPG1010 included in control plane VCN 1016 (e.g., Figure 7 The control plane VCN 716) and coupled to the data plane VCN 1018 via the LPG 1010 contained in the data plane VCN 1018 (e.g., Figure 7 Data plane 718). Control plane VCN 1016 and data plane VCN 1018 may be included in service lease 1019 (e.g., Figure 7 (Service rental 719).

[0146] The control plane VCN 1016 may include one or more LB subnets 1022 (e.g., Figure 7 The control plane DMZ layer 1020 of (one or more) LB subnets 722) (e.g., Figure 7 The control plane DMZ layer 720 may include one or more application subnets 1026 (e.g., Figure 7 The control plane application layer 1024 of (one or more) application subnets 726 (e.g., Figure 7 The control plane application layer 724) may include (one or more) DB subnets 1030 (e.g., Figure 9 The control plane data layer 1028 of (one or more) DB subnets 930 (e.g., Figure 7 The control plane data layer 728). One or more LB subnets 1022 contained in the control plane DMZ layer 1020 can be communicatively coupled to one or more application subnets 1026 contained in the control plane application layer 1012 and an Internet gateway 1034 that can be contained in the control plane VCN 1016 (e.g., Figure 7 Internet gateway 734), and application subnet(s) 1026 can communicatively couple to DB subnet(s) 1030 contained in control plane data layer 1028 and service gateway 1036 (e.g., Figure 7 The service gateway) and Network Address Translation (NAT) gateway 1038 (e.g., Figure 7 (NAT gateway 738). The control plane VCN 1016 may include the service gateway 1036 and the NAT gateway 1038.

[0147] Data plane VCN 1018 may include data plane application layer 1046 (e.g., Figure 7 Data plane application layer 746), data plane DMZ layer 1048 (e.g., Figure 7Data plane DMZ layer 748), and data plane data layer 1050 (e.g., Figure 7 The data plane data layer 750). The data plane DMZ layer 1048 may include one or more trusted application subnets 1060 that can be communicatively coupled to the data plane application layer 1046 (e.g., Figure 9 (one or more) trusted application subnets 960 and (one or more) untrusted application subnets 1062 (e.g., Figure 9 The data plane VCN 1018 may include one or more untrusted application subnets 962 and one or more LB subnets 1022 of Internet gateway 1034. One or more trusted application subnets 1060 may communicatively couple to service gateway 1036, NAT gateway 1038, and DB subnets 1030 in data plane VCN 1018. One or more untrusted application subnets 1062 may communicatively couple to service gateway 1036 and DB subnets 1030 in data plane VCN 1018. Data plane VCN 1050 may include one or more DB subnets 1030 that may communicatively couple to service gateway 1036 in data plane VCN 1018.

[0148] One or more untrusted application subnets 1062 may include a primary VNIC 1064(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1066(1)-(N) residing within one or more untrusted application subnets 1062. Each tenant VM 1066(1)-(N) may run code in a corresponding container 1067(1)-(N) and is communicatively coupled to an application subnet 1026 that may be contained in a data plane application layer 1046 that may be contained in a container egress VCN 1068. A corresponding secondary VNIC 1072(1)-(N) may facilitate communication between one or more untrusted application subnets 1062 contained in a data plane VCN 1018 and the application subnet contained in a container egress VCN 1068. The container egress VCN may include a public internet 1054 (e.g., Figure 7 The public internet (754) uses NAT gateway 1038.

[0149] Internet gateway 1034, contained in control plane VCN 1016 and data plane VCN 1018, can be communicatively coupled to metadata management service 1052 (e.g., Figure 7Metadata management service 1052 can be communicatively coupled to the public internet 1054. The public internet 1054 can be communicatively coupled to a NAT gateway 1038 contained in a control plane VCN 1016 and a data plane VCN 1018. A service gateway 1036 contained in a control plane VCN 1016 and a data plane VCN 1018 can be communicatively coupled to a cloud service 1056.

[0150] In some examples, Figure 10 The architecture shown in block diagram 1000 can be considered as... Figure 9 This is an exception to the pattern shown in the architecture of block diagram 900, and this pattern may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). Customers can access in real time the corresponding containers 1067(1)-(N) contained in each customer's tenant VM 1066(1)-(N). Containers 1067(1)-(N) can be configured to invoke corresponding auxiliary VNICs 1072(1)-(N) contained in one or more application subnets 1026 of the data plane application layer 1046, which may be contained in the container egress VCN 1068. The auxiliary VNICs 1072(1)-(N) can transmit the calls to a NAT gateway 1038, which can then transmit the calls to the public internet 1054. In this example, containers 1067(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1016 and from other entities contained in the data plane VCN 1018. Containers 1067(1)-(N) can also be isolated from resources from other clients.

[0151] In other examples, a client may use container 1067(1)-(N) to invoke cloud service 1056. In this example, the client may run code within container 1067(1)-(N) requesting a service from cloud service 1056. Container 1067(1)-(N) may transmit the request to auxiliary VNIC 1072(1)-(N), which may transmit the request to a NAT gateway, which may then transmit the request to public internet 1054. Public internet 1054 may then transmit the request via internet gateway 1034 to one or more LB subnets 1022 contained in control plane VCN 1016. In response to determining that the request is valid, one or more LB subnets may transmit the request to one or more application subnets 1026, which may then transmit the request to cloud service 1056 via service gateway 1036.

[0152] It should be recognized that the IaaS architectures 700, 800, 900, and 1000 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0153] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0154] Figure 11 An example computer system 1100 in which various embodiments can be implemented is illustrated. System 1100 can be used to implement any of the computer systems described above. As shown, computer system 1100 includes a processing unit 1104 that communicates with a plurality of peripheral subsystems via a bus subsystem 1102. These peripheral subsystems may include a processing acceleration unit 1106, an I / O subsystem 1108, a storage subsystem 1118, and a communication subsystem 1124. Storage subsystem 1118 includes a tangible computer-readable storage medium 1122 and system memory 1110.

[0155] Bus subsystem 1102 provides a mechanism for allowing various components and subsystems of computer system 1100 to communicate with each other as intended. While bus subsystem 1102 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1102 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0156] A processing unit 1104, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1100. One or more processors may be included in the processing unit 1104. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1104 may be implemented as one or more independent processing units 1132 and / or 1134, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1104 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0157] In various embodiments, processing unit 1104 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processing unit(s) 1104 and / or storage subsystem 1118. With appropriate programming, processing unit(s) 1104 can provide the various functions described above. Computer system 1100 may additionally include processing acceleration unit 1106, which may include a digital signal processor (DSP), a dedicated processor, etc.

[0158] I / O subsystem 1108 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input in an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0159] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0160] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1100 to a user or other computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0161] Computer system 1100 may include a storage subsystem 1118 that provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which provide the aforementioned functionality when executed by one or more cores or processors of processing unit 1104. Storage subsystem 1118 may also provide a repository for storing data used according to this disclosure.

[0162] like Figure 11 As depicted in the example, storage subsystem 1118 may include various components, including system memory 1110, computer-readable storage medium 1122, and computer-readable storage medium reader 1120. System memory 1110 may store program instructions that can be loaded and executed by processing unit 1104. System memory 1110 may also store data used during the execution of instructions and / or data generated during the execution of program instructions. Various kinds of programs may be loaded into system memory 1110, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0163] System memory 1110 may also store operating system 1116. Examples of operating system 1116 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems (such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS). In some implementations where computer system 1100 executes one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 1110 and executed by one or more processors or cores of processing unit 1104.

[0164] System memory 1110 may be configured differently depending on the type of computer system 1100. For example, system memory 1110 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, system memory 1110 may include a basic input / output system (BIOS) containing basic routines that facilitate the transfer of information between components within computer system 1100, such as during startup.

[0165] Computer-readable storage medium 1122 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1104 of the computer system 1100) for use by the computer system 1100.

[0166] Computer-readable storage medium 1122 may include any suitable medium known or used in the art, including storage and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0167] For example, computer-readable storage medium 1122 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a disk drive that reads or writes to a removable non-volatile magnetic disk, and an optical disc drive that reads or writes to a removable non-volatile optical disc (such as a CD-ROM, DVD, and Blu-ray® disc or other optical media). Computer-readable storage medium 1122 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and so on. Computer-readable storage medium 1122 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for computer system 1100.

[0168] Machine-readable instructions executable by one or more processors or cores of processing unit 1104 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0169] The communication subsystem 1124 provides an interface to other computer systems and networks. The communication subsystem 1124 serves as an interface for receiving data from other systems and sending data from computer system 1100 to other systems. For example, the communication subsystem 1124 enables computer system 1100 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1124 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.12 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1124 may provide a wired network connection (e.g., Ethernet).

[0170] In some embodiments, the communication subsystem 1124 may also represent one or more users who can use the computer system 1100 to receive input communications in the form of structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc.

[0171] For example, the communication subsystem 1124 can be configured to receive data feeds 1126 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0172] Furthermore, the communication subsystem 1124 can also be configured to receive data in the form of a continuous data stream, which may include an event stream 1128 and / or event updates 1130 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quotation machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0173] The communication subsystem 1124 can also be configured to output structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1100.

[0174] The computer system 1100 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing system.

[0175] Due to the constantly evolving nature of computers and networks, the description of the computer system 1100 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0176] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0177] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0178] Accordingly, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0179] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as encompassing both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0180] As used herein, when an action is “based on” something, it means that the action is based at least partially on at least a portion of that thing. As used herein, the terms “substantially,” “about,” and “approximately” are defined as largely (but not necessarily completely) conforming to the specified content (and including fully conforming to the specified content), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “substantially,” “about,” and “approximately” may be replaced with “[percentage]” of the specified content, where percentages include 0.1%, 1%, 5%, and 10%.

[0181] Disjunctive language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, each individually.

[0182] This document describes preferred embodiments of the present disclosure, including known best modes for carrying out the present disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to appropriately employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Moreover, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0183] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0184] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A computer-implemented method, comprising: Access messages that include timestamp data and user identification data; Identify a training set of data entities that include patient information, wherein each data entity in the training set is: (i) is included in the appointment dataset associated with user identification data, and (ii) Includes appointment time data within a time window based on timestamp data; At least one training dataset is determined based on the data entities in the training group; Modify at least one pre-trained machine learning model based on the at least one training dataset; as well as During an authenticated network session associated with user identification data, a modified version of the at least one pre-trained machine learning model is provided.

2. The computer-implemented method as described in claim 1, further comprising: An additional message indicating the end of an authenticated network session associated with user identification data; as well as Remove modifications based on the at least one training dataset from the at least one pre-trained machine learning model.

3. The computer-implemented method of claim 1, wherein the data entity of the training group includes a secure data entity, and the authenticated network session grants access to patient information included in the secure data entity to a client device authenticated via user identification data.

4. The computer-implemented method of claim 1, wherein the message is generated in response to the execution of a client application on a client device associated with user identification data, the message indicating that the client application is granted access to computing resources.

5. The computer-implemented method of claim 1, wherein the time window includes one or more of the following: (a) A forward time window, including the first time period following the timestamp data. (b) A backward time window, including a second time period preceding the timestamp data, or (c) An extended time window, which includes the most recent appointment time data item.

6. The computer-implemented method as described in claim 1, further comprising: During an authenticated network session, access additional messages including user identification data and modified timestamp data; Determine an additional training dataset based on at least one additional data entity, the additional data entity being (i) included in the appointment dataset associated with user identification data, and (ii) included in additional appointment time data within an additional time window based on additional timestamp data; The at least one pre-trained machine learning model is further modified based on the additional training dataset; as well as During an authenticated network session associated with user identification data, the at least one pre-trained machine learning model is provided with further modifications.

7. The computer-implemented method of claim 6, wherein further modification of the at least one pre-trained machine learning model further comprises removing modifications based on the at least one training dataset from the at least one pre-trained machine learning model.

8. The computer-implemented method of claim 1, wherein identifying the data entities of the training group further includes: Identify a set of data entities associated with user identification data; Determine a first subset of the set of data entities, including that each data entity in the first subset is included in the appointment dataset associated with user identification data; Determine a second subset of the set of data entities, including that each data entity in the second subset includes appointment time data within the time window; as well as A third subset of the set of data entities associated with user identification data is selected, wherein each data entity in the third subset is included in the first subset and the second subset. The data entities in the training group include each data entity included in the third subset.

9. The computer-implemented method as described in claim 1, wherein: The at least one pre-trained machine learning model includes a machine learning model pre-trained for speech recognition. The at least one training dataset includes a speech recognition training dataset; The speech recognition training dataset is determined by: Extract text data associated with patient information included in the data entities of the training group from the data entities of the training group, and Generate a customized recognition vocabulary that includes the extracted text data, wherein the speech recognition training dataset includes the customized recognition vocabulary; as well as Modifying the at least one pre-trained machine learning model based on the at least one training dataset includes modifying the pre-trained machine learning model for speech recognition based on the speech recognition training dataset.

10. The computer-implemented method of claim 9, further comprising: An additional message indicating the end of an authenticated network session associated with user identification data; as well as Remove the speech recognition training dataset, which includes the customized recognition vocabulary, from the pre-trained machine learning model for speech recognition.

11. The computer-implemented method as described in claim 1, wherein: The at least one pre-trained machine learning model includes a language pre-trained machine learning model. The at least one training dataset includes a language training dataset; Determining the language training dataset includes: Identify a set of data objects in the data entities of the training group that are associated with patient information included in the data entities of the training group; as well as For each specific data object in the set of data objects, modify the corresponding searchable data entity to include the specific data object, wherein the language training dataset includes the corresponding searchable data entity for each specific data object in the set of data objects; and Modifying the at least one pre-trained machine learning model based on the at least one training dataset includes modifying the language pre-trained machine learning model based on the language training dataset.

12. The computer-implemented method of claim 11, further comprising: An additional message indicating the end of an authenticated network session associated with user identification data; as well as Remove the language training dataset, which includes the corresponding searchable data entities for each specific data object in the set of data objects, from the language pre-trained machine learning model.

13. A system comprising: One or more processing systems; as well as One or more computer-readable media storing instructions that, when executed by the one or more processing systems, cause the systems to perform operations, including: Access messages that include timestamp data and user identification data; Identify a training set of secure data entities that include patient information, wherein each secure data entity in the training set is: (i) is included in the appointment dataset associated with user identification data, and (ii) Includes appointment time data within a time window based on timestamp data; At least one training dataset is determined based on the secure data entities of the training group; Modify at least one pre-trained machine learning model based on the at least one training dataset; and During an authenticated network session associated with user identification data, the modified at least one pre-trained machine learning model is provided.

14. The system of claim 13, wherein the operation further comprises: An additional message indicating the end of an authenticated network session associated with user identification data; as well as Remove modifications based on the at least one training dataset from the at least one pre-trained machine learning model.

15. The system of claim 13, wherein the data entity of the training group includes a secure data entity, and the authenticated network session grants access to patient information included in the secure data entity of the training group to a client device authenticated via user identification data.

16. The system of claim 13, wherein the time window comprises one or more of the following: (a) A forward time window, including the first time period following the timestamp data. (b) A backward time window, including a second time period preceding the timestamp data, or (c) Expand the time window, which includes the most recent appointment time data item.

17. A non-transitory computer-readable medium storing one or more instructions, said instructions, when executed by one or more processors, causing a system to perform operations, including: Access messages that include timestamp data and user identification data; Identify a training set of data entities that include patient information, wherein each data entity in the training set is: (i) is included in the appointment dataset associated with user identification data, and (ii) Includes appointment time data within a time window based on the timestamp data; At least one training dataset is determined based on the data entities in the training group; Modify at least one pre-trained machine learning model based on the at least one training dataset; as well as During an authenticated network session associated with user identification data, a modified version of the at least one pre-trained machine learning model is provided.

18. The one or more non-transitory computer-readable media of claim 17, wherein the operation further comprises: An additional message indicating the end of an authenticated network session associated with user identification data; as well as Remove modifications based on the at least one training dataset from the at least one pre-trained machine learning model.

19. One or more non-transitory computer-readable media as claimed in claim 17, wherein the data entity of the training group includes a secure data entity, and the authenticated network session grants access to patient information included in the secure data entity of the training group to a client device authenticated via user identification data.

20. One or more non-transitory computer-readable media as claimed in claim 17, wherein the time window includes one or more of the following: (a) A forward time window, including the first time period following the timestamp data. (b) A backward time window, including a second time period preceding the timestamp data, or (c) Expand the time window, which includes the most recent appointment time data item.

Images