Method and system for generating automation commands, and computer readable medium

Abstract

Systems and methods for generating automated commands for a treatment planning system for use with radiotherapy treatment, including: receiving a textual input from a user, the textual input including information related to a radiotherapy treatment plan of a patient; determining a command of the treatment planning system and a command parameter to be used by the treatment planning system based on the parsed text input; generating, using the generative chat robot, a script based on the text input, the script including a command and a command parameter; and outputting a script for automation of the treatment planning system in combination with establishing a radiotherapy treatment plan for radiotherapy treatment of the patient.

Description

TECHNICAL FIELD

[0001] Embodiments herein relate to methods and systems for processing medical data. In particular, the methods and systems are directed to processing medical data associated with radiotherapy. BACKGROUND

[0002] Radiotherapy, or radiation therapy, can be described as the use of ionizing radiation to damage or destroy unhealthy cells in humans and animals. For example, the unhealthy cells can include cancerous cells. The ionizing radiation can be directed to a tumor on the skin surface or deep inside the body. Common forms of ionizing radiation include X-rays and charged particles. An example of radiotherapy technology is the use of multiple lower intensity gamma rays that are conformed at a target region (e.g., a tumor) with higher intensity and high precision to irradiate a patient or Leksell Another example of radiotherapy includes the use of a linear accelerator (“linac”), whereby a target region is irradiated by high-energy particles (e.g., electrons, high-energy photons, etc.). In another example, a heavy charged particle accelerator (e.g., protons, carbon ions, etc.) is used to provide radiotherapy.

[0003] A computer system is used to create a treatment plan that controls and personalizes the radiotherapy output for a particular patient. Treatment plan data can include details of the treatment delivery, such as the number of beams, machines, modalities, beam energies and machine output of monitor units (MUs), among other aspects. Software applications that create and modify treatment plans are referred to as “treatment planning systems,” and these systems typically include complex functionality that computes, customizes, validates, and optimizes details of a particular radiotherapy treatment and treatment plan data.

[0004] A treatment plan can be created by a treatment planning system based on data including, but not limited to: patient data (e.g., Electronic Medical Record (EMR) or Electronic Health Record (EHR)), medical imaging data, treatment plan information, Dose-Volume Histogram (DVH), dose information, dosimetric metrics, clinical outcomes, Instructions For Use (IFU), etc. For example, the medical imaging data can include information about certain anatomical structures and a treatment (e.g., planned target volume, target, organ(s) at risk, etc.) to be delivered to the anatomical structures.

[0005] Converting clinical requirements from a planned protocol into goals and controls for a specific treatment plan for a patient is not simple. For a routine medical procedure, this involves discussions between a radiation oncologist, medical physicist, and planner to assess feasibility and find the right goals and constraints to input into a treatment planning system. Thus, there is a need for improved methods and systems for analyzing medical data and inputs associated with a treatment plan to enable users and computing systems to properly process medical data, improve accuracy of treatment, and reduce the number of computational operations in implementing a system process. BRIEF DESCRIPTION OF DRAWINGS

[0006] Systems and methods according to non-limiting examples will now be described with reference to the accompanying drawings, wherein:

[0007] Figure 1 A radiotherapy system adapted to perform radiotherapy planning processing operations using the methods discussed herein is depicted.

[0008] Figure 2 A use case for invoking a treatment plan automation workflow is depicted.

[0009] Figure 3 A sequence of operations for treatment plan automation based on a treatment planning system script generated by natural language is depicted.

[0010] Figure 4 A detailed processing workflow for a treatment plan automation workflow for converting human commands into executable commands is depicted.

[0011] Figure 5 A schematic illustration of an example language model that can be used by a chatbot is depicted.

[0012] Figure 6 A flowchart of an example method for generating automated commands for a treatment planning system based on automation provided from the methods discussed herein is depicted.

[0013] Figure 7 A flowchart of an example method for providing a radiotherapy treatment based on automation provided from the methods discussed herein is depicted.

[0014] Figure 8 is a block diagram of a device, apparatus, system, or machine according to various embodiments. DETAILED DESCRIPTION

[0015] Methods and systems are described below that improve the generation and operation of radiotherapy treatment planning by processing natural language to generate executable scripts or other data outputs that control a radiotherapy treatment planning system. This can help limit human interaction with radiotherapy treatment planning software while accurately translating clinical needs into inputs for treatment planning applications to produce improved outputs for radiotherapy machines. In addition to clear technical benefits, the automatic generation of executable scripts or software control instructions from natural language also provides a beneficial process that can help clinical users save time while improving accuracy and efficiency.

[0016] The following methods and systems can be invoked by a variety of users including dosimetrists or treatment planning personnel. Such personnel can invoke automated and improved computer functionality in a treatment planning system without needing to know how to write source code or how to implement complex medical requirements specified by radiation oncologists. Thus, the methods and systems can be used to automatically control and operate a treatment planning system to produce treatment plans and data that control a radiation therapy process and plan workflows with improved patient outcomes.

[0017] In particular, the methods and systems include generating scripts based on the entry of user commands (also referred to as queries, requests, or prompts) provided in natural language using a chatbot. A chatbot is a software agent configured to converse with a user via text or speech. Chatbots can also be referred to as conversational agents or systems, intelligent assistants, or Artificial Intelligence (AI) agents. Natural language refers to language that occurs naturally in human communication that is provided in spoken or written form. In this context, natural language can occur in the context of user commands that are entered to ask a treatment planning system to perform certain complex types of actions (or series of actions) with simple words. Natural language is different from structured languages such as computer programming languages. Due to the nuances of human language and the complexity of programming languages, the translation of natural language commands to useful computer programming commands has not been solved by existing methods in the context of radiotherapy planning settings.

[0018] Accordingly, the methods and systems below introduce text / speech chatbot functionality that can be used to generate scripts (e.g., scripts with computer programming commands) to implement automation in a treatment planning system. Additionally, these text / speech chatbot functionalities can be used to translate clinical needs into automated commands and other inputs to a treatment planning system as part of individual commands, conversations, or contexts or sessions associated with a patient or treatment use case. As an example, a user can provide natural language by writing or speaking to an AI agent in an ongoing conversation about what a patient needs to do to proceed with a treatment plan. Based on these inputs, the AI agent can generate scripts and commands to implement tasks in a treatment planning software automatically (or in the form of simpler human interaction).

[0019] With the present methods and systems, any user (e.g., a treatment planner, dosimetrist, clinician, health care worker, or analyst) can invoke automation from natural language to produce intelligent automation and adaptation of treatment planning operations. A dosimetrist or planner can use automation without needing to know how to write code or perform complex treatment planning, thereby achieving the same results and refinement as a skilled radiation oncologist without training or expertise. In the same way that a user uses natural language to converse with human colleagues, the natural language input can be interpreted by the system and used to control various subsystems with accurate script commands.

[0020] By using natural language (rather than, for example, a computer programming language), ease of use is increased, and more efficient computational operations can be achieved. Accordingly, the methods and systems herein address the technical problem that arises in the field of processing medical data (particularly, radiation therapy medical data), namely, how to facilitate accurate processing of medical data based on commands received and interpreted from a user in natural language, while providing precise control of the system via programmatic instructions. This benefit can be leveraged especially when a user presents a complex list of intentions or sequence of commands or attempts to invoke a unique or customized action in a treatment planning software.

[0021] Further explanation of natural language processing is provided after an overview of the present systems and methods, including an overview of radiation therapy treatment and treatment planning systems.

[0022] Figure 1A radiation therapy system 100 adapted to perform radiation therapy plan processing operations using one or more methods discussed herein is illustrated. These radiation therapy plan processing operations are performed to enable the radiation therapy system 100 to provide radiation treatment to a patient based on captured medical imaging data and particular aspects of treatment dose calculations or radiation machine configuration parameters. In particular, the following processing operations can be implemented as part of a treatment planning system 120 (also referred to as a “TPS”) for developing radiation therapy treatment plans based on natural language commands. However, it should be appreciated that many variations and use cases of the treatment planning system 120 and automated operations can be provided, such as in response to data optimization, visualization, and other medical assessment and diagnostic operations.

[0023] The radiation therapy system 100 includes a radiation data processing computing system 110 hosting a treatment planning system 120. The radiation data processing computing system 110 can be connected to a network (not illustrated) and such a network can be connected to the Internet. For example, the network can connect the radiation data processing computing system 110 with one or more private and / or public medical information sources (e.g., a radiology information system (RIS), a medical records system (e.g., an electronic medical record (EMR) / electronic health record (EHR) system), an oncology information system (OIS)), one or more image data sources, image acquisition devices (e.g., imaging modalities). In the described example, the radiation data processing computing system 110 is operatively coupled to a treatment planning data source 150 (e.g., a database storing treatment plans) and a treatment device 160 (e.g., a radiation therapy device implementing treatment plans).

[0024] As an example, the radiation data processing computing system 110 can be configured to receive a treatment goal for a subject (e.g., an anatomical region to be delivered treatment) and generate a radiation therapy treatment plan by executing instructions or data within the treatment planning system 120 as part of an operation to generate a treatment plan to be used by the treatment device 160 and / or output on an output device 142. In one embodiment, the treatment planning system 120 is a software application or software platform that includes programmed functionality to generate, validate, and optimize radiation therapy treatment plans for individual patients.

[0025] The radiotherapy data processing computing system 110 may include processing circuitry 112, memory 114, storage device 116, and other hardware and software operable features, such as user interface 140, communication interface (not shown), etc. Storage device 116 may store transient or non-transient computer-executable instructions, such as operating systems, radiotherapy treatment plans, training data, software programs (e.g., image processing software, image or anatomical visualization software, artificial intelligence (AI) or ML implementations and algorithms, such as those provided by deep learning models, ML models, and neural networks (NN), etc.), and any other computer-executable instructions to be executed by processing circuitry 112.

[0026] In one example, processing circuitry 112 may include at least one processing device, such as one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc. More specifically, processing circuitry 112 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing circuitry 112 may also be implemented by one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), systems on a chip (SoCs), etc.

[0027] As those skilled in the art will understand, in some examples, the processing circuitry 112 may be a dedicated processor rather than a general-purpose processor. The processing circuitry 112 may include one or more known processing devices, such as those from Intel. Manufactured Pentium TM ), Core TM XeonTM ) or Itanium TM ) family, AMD TM Turion manufactured TM Athlon TM ), Sempron TM Opteron TM ) FX TM Phenom TM The processing circuitry 112 may include a microprocessor from the Nvidia family, or any processor from various processors manufactured by Sun Microsystems. The processing circuitry 112 may also include a graphics processing unit, such as one from Nvidia. TM Precision manufacturing Litai Tesla Family, by Intel The manufactured GMA and Arc TM ) family or by Radeon manufactured TM GPU devices provided by the Intel family. The processing circuitry 112 may also include accelerated processing units, such as those integrated into GPUs provided by Intel. The manufactured Xeon TM The family of accelerated processing units.

[0028] In some examples, the processing circuitry 112 may include or be arranged in a parallel processing configuration. For example, a group of graphics processing units (e.g., from NVIDIA) may be arranged. Manufacturing precision Litai Tesla Family, by Intel The manufactured GMA and Arc TM ) family or by Radeon manufactured TM) family of GPU cores, units, devices, or cards) to perform highly parallel or repetitive computing tasks simultaneously. The disclosed embodiments are not limited to any type of processor(s) otherwise configured to meet the computational demands of identifying, analyzing, maintaining, generating, and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. Additionally, the term “processor” can include more than one physical (circuit-based) or software-based processor (e.g., multi-core designs or multiple processors, each with multi-core designs). The processing circuitry 112 can execute sequences of computer program instructions stored in memory 114 and accessed from storage device 116 to perform various operations, processes, and methods that will be explained in greater detail below. It should be understood that any of the components in the radiotherapy system 100 can be implemented separately and operate as standalone devices, and can be coupled to any of the other components in the radiotherapy system 100 to perform the techniques described in the present disclosure.

[0029] The memory 114 can include read-only memory (ROM), phase-change random access memory (PRAM), static random access memory (SRAM), flash, random access memory (RAM), dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM)), electrically erasable programmable read-only memory (EEPROM), static memory (e.g., flash memory, flash disk, static random access memory), and other types of random access memory, cache, registers, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs), or other optical storage, magnetic cassettes, other magnetic storage devices, or any other non-transitory medium that can be used to store information including images, training data, ML model(s), or technical parameter(s), data, or transitory or non-transitory computer executable instructions (e.g., stored in any format) that are accessible by the processing circuitry 112 or any other type of computer device. For example, the computer program instructions can be accessed by the processing circuitry 112, read from the ROM or any other suitable memory location, and loaded into the RAM for execution by the processing circuitry 112.

[0030] The storage device 116 can constitute a drive unit that includes a transitory or non-transitory machine-readable medium having stored thereon one or more sets of instructions and data structures (e.g., software) embodying any one or more of the methodologies or functions described herein, including the treatment planning system 120 and the user interface 140 in various examples. The instructions can also reside, completely or at least partially, within the memory 114 and / or the processing circuit 112 during execution thereof by the radiation therapy data processing computing system 110, with the memory 114 and the processing circuit 112 also constituting transitory or non-transitory machine-readable media.

[0031] The memory 114 and the storage device 116 can constitute non-transitory computer- readable media. For example, the memory 114 and the storage device 116 can store or load transitory or non-transitory instructions for one or more software applications on the computer-readable media. The software applications stored or loaded in the memory 114 and the storage device 116 can include, for example, an operating system for common computer systems and software-controlled devices. The radiation therapy data processing computing system 110 can also operate various software programs, including software code for implementing the treatment planning system 120, the treatment planning automation workflow 130, and the user interface 140. Further, the memory 114 and the storage device 116 can store or load entire software applications, portions of software applications, or code or data associated with software applications, executable by the processing circuit 112.

[0032] By way of non-limiting example, the memory 114 and the storage device 116 can store, load, and manipulate one or more radiation therapy treatment plans, script commands, imaging data, segmentation data, treatment visualizations, histograms or measurements, one or more AI model data (e.g., weights and parameters of the ML model(s)), training data, labels, and mapping data, etc. It is contemplated that software programs can be stored not only on the storage device 116 and the memory 114, but also on removable computer media such as a hard disk drive, a computer diskette, a CD-ROM, a DVD, a Blu-ray DVD, a USB flash drive, an SD card, a memory stick, or any other appropriate media; such software programs can also be transmitted or received over a network.

[0033] Although not depicted, the radiotherapy data processing computing system 110 can include communication interfaces, network interface cards, and communication circuitry. Example communication interfaces can include, for example, network adapters, cable connectors, serial connectors, USB connectors, parallel connectors, high-speed data transfer adapters (e.g., such as fiber optic, USB 3.0, Thunderbolt, etc.), wireless network adapters (e.g., such as IEEE 802.11 / Wi-Fi adapters), telecommunication adapters (e.g., to communicate with 3G, 4G / LTE, and 5G networks, etc.), and the like. Such communication interfaces can include one or more digital and / or analog communication devices that allow the machine to communicate with other machines and devices (e.g., remotely located components) via a network. The network can provide the functionality of a Local Area Network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a Wide Area Network (WAN), and the like. For example, the network can be a LAN or WAN that can include other systems (including additional image processing computing systems or image-based components associated with medical imaging or radiotherapy operations).

[0034] The processing circuitry 112 can be communicatively coupled to the memory 114 and the storage device 116, as the processing circuitry 112 is configured to execute computer-executable instructions from the memory 114 or the storage device 116 stored thereon. In particular, the treatment plan automation workflow 130 adapts the treatment planning system 120 via scripts and computer software commands to implement operations, procedures, parameters, and other workflow actions for radiotherapy treatment. The treatment planning system 120 can be controlled with the results of the treatment plan automation workflow 130 to generate new or updated treatment planning parameters (for deployment to the treatment plan data source 150 and / or presentation on the output device 142). The processing circuitry 112 can subsequently transmit the new or updated treatment planning parameters to the treatment device 160 via the communication interface and network, where, according to the results of the treatment planning system 120 and the treatment plan automation workflow 130 (e.g., according to the processes discussed below), a radiation treatment plan will be used to treat the patient with radiation via the treatment device 160.

[0035] The radiotherapy data processing computing system 110 can communicate with external databases over a network to send / receive a plurality of various types of data related to image processing and radiotherapy operations. For example, the external databases can include machine data (including device constraints) that provide information associated with medical treatment devices 160, image acquisition devices, or other machines related to radiotherapy or medical processes. Machine data information (e.g., control points) can include radiation beam size, arc placement, beam on and off durations, machine parameters, segments, Multi-Leaf Collimator (MLC) configurations, gantry speed, MRI pulse sequences, etc. The external databases can be storage devices and can be equipped with appropriate database management software programs. Further, such databases or data sources can include multiple devices or systems located in a centralized or distributed manner.

[0036] The radiotherapy data processing computing system 110 can collect and obtain data and communicate with other systems via a network using one or more communication interfaces communicatively coupled to the processing circuitry 112 and the memory 114. For example, the communication interfaces can provide a communication connection between the radiotherapy data processing computing system 110 and radiotherapy system components (e.g., allowing exchange of data with external devices). For example, in some examples, the communication interfaces can have appropriate interface circuitry to connect from the output device 142 or input device 144 to the user interface 140, which can be a hardware keyboard, keypad, or touchscreen through which a user can input information into the radiotherapy system.

[0037] As an example, the output device 142 can include a display device that outputs a representation of the user interface 140 and one or more aspects, visualizations, or representations of medical images, treatment plans, and a status of training, generation, validation, or implementation of such plans. The output device 142 can include one or more display screens that display medical images, interface information, treatment plan parameters (e.g., contours, dose, beam angles, labels, maps, etc.), treatment plans, targets, locating targets, and / or tracking targets, or any related information to a user. The input device 144 connected to the user interface 140 can be a keyboard, keypad, touchscreen, or any type of device that a user can use for the radiotherapy system 100. Alternatively, features of the output device 142, input device 144, and user interface 140 can be integrated into a single device, such as a smartphone or tablet computer (e.g., Apple iPhone®, iPad®, or Samsung Galaxy®). Associates Samsung etc.

[0038] Further, any and all components of the radiotherapy system can be implemented as virtual machines (e.g., via a VMWare, Hyper-V, or the like virtualization platform) or standalone devices. For example, a virtual machine can be software that acts as hardware. Thus, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that together act as hardware. For example, the radiotherapy data processing computing system 110, the image data source, or the like component can be implemented as a virtual machine or within a cloud-based virtualization environment.

[0039] The treatment planning system 120 in the radiotherapy data processing computing system 110 implements a treatment planning automation workflow 130 according to the following examples. The treatment planning automation workflow 130 can implement operations for identifying and developing a radiotherapy plan based on capturing natural language input and converting the natural language input into relevant operational computerized commands in the treatment planning system 120. In particular examples, the treatment planning automation workflow 130 includes natural language processing 132 that converts and segments individual commands from language speech and text input; chatbot functionality 134 that interacts with a user to identify relevant user commands and provides output to the user in a human conversation format to elicit user commands; scripting functionality 136 that converts individual commands into a scripting language that can be used with the treatment planning system 120 or another automation tool associated with the treatment planning system 120; and optimization functionality 138 that provides improvements to a treatment plan, including invoking built-in treatment plan optimization and processes or implementing custom automation associated with optimization.

[0040] Further details of implementing the computing system are provided in Figure 8 The following references Figure 2 and Figure 3 provide further details of the treatment planning automation workflow 130. Likewise, the following references Figure 4 and Figure 5 provide further details of the natural language processing 132, chatbot functionality 134, and scripting functionality 136 used in the treatment planning automation workflow 130. Figure 6 and Figure 7 Further details are described of how a treatment plan can be further evaluated, selected, optimized, and deployed in a radiotherapy planning system as a result of the treatment planning automation workflow 130 and then brought to a radiotherapy machine.

[0041] Radiotherapy treatment plans can arise in a variety of contexts. Different users or clinical centers can use different protocols or prescriptions to arrive at a suitable treatment plan, and the outcome of a designed treatment plan can depend on the experience of the health care professional. Automation in treatment planning systems generally improves the efficiency of the user (dosimetrist) while maintaining treatment quality. However, even for partially automated systems, the outcome and optimization of a particular radiotherapy plan is generally based on the skill of the user to implement scripts and programming.

[0042] Improvements in treatment planning systems described herein can distinguish from existing automated solutions that involve scripting. Automation of treatment planning systems via scripting has standardized the process and enabled staff to master more complex techniques, for example in conjunction with increased utilization of volumetric modulated arc therapy (VMAT) procedures and related VMAT planning software. However, such scripting generally requires the use of a programming language such as C sharp (C#) or Python, or requires knowledge of how to use visual scripting to call specific programming libraries and functions.

[0043] Most clinicians (e.g., physicists, dosimetrists, or therapists) do not have a background in computer science, so the learning curve is very steep to effectively call scripts. Additionally, the use of a programming language in scripts adds an enhanced barrier to debugging and trial running scripts. These two drawbacks have posed a barrier to widespread adoption of scripting and automated processes. The following techniques introduce methods for converting natural language commands into scripts and automation to address these drawbacks and related technical problems in automation.

[0044] Figure 2 A use case for invoking a treatment planning automation workflow 130 is depicted. First, user commands 205 are received by a text chatbot or voice agent 210. The text chatbot or voice agent 210 applies logic that converts the user commands 205 into one or more script commands 215. Based on the type of user command and the type of action to be invoked, individual user commands can result in the creation of a single script command or multiple script commands. Additionally, in some settings, multiple user commands (or a series of user commands) can result in the creation or modification of multiple script commands.

[0045] One or more script commands 215 can affect various aspects of the operation of the treatment planning system 120. These aspects include, but are not limited to, automation of user interface controls 222, use of radiotherapy planning templates 224, and access, modification, creation, or updating of treatment planning data 226. In the following example, the treatment planning system can automate the creation or modification of a treatment plan for a particular patient in the treatment planning data 226 based on a set of one or more script commands 215 that invoke user interface controls 222 and one or more templates 224. The treatment planning system 120 can also provide an Application Programming Interface (API) and other programming features (not shown) that can be invoked and automated from one or more script commands 215.

[0046] The automation of the treatment planning system 120 is thus used to produce one or more treatment plans as treatment planning output 230 (e.g., in the form of data for one or more treatment plans and related treatment control data). The treatment planning output 230 can then be used to control a radiotherapy treatment operation 240.

[0047] The presently disclosed methods for scripting and automation provide benefits over existing uses of automated treatment planning tools for radiotherapy. Such planning automation often relies on some form of intelligence to translate clinical requirements into treatment plans, such as by directly suggesting or inferring some treatment output. Some examples include the use of Knowledge-Based Planning (KBP) and Multicriterial Optimization (MCO). For example, KBP involves the use of a library of high-quality plans that are clinically accepted. KBP can be used to suggest how well a plan can be by comparing the anatomy of a new patient to the plan library, allowing a planner to learn from the suggestions of experienced colleagues. On the other hand, some approaches to MCO include sequentially trying to best protect organs at risk (OARs) from treatment, without compromising target coverage, by replacing the planner in a typical trial-and-error process. MCO often seeks optimal solutions that belong to a so-called Pareto surface, which means that a plan cannot be further improved for any objective without degrading the outcome of at least one of the other objectives. Such surface navigation can be performed by some treatment planning systems with a priori MCO, often proposing only one plan solution for a listed request. In a posteriori approaches, a user can navigate between multiple plans generated to select the one that best meets the clinical request.

[0048] The present method for translating human language user commands into treatment planning system commands can provide an enhanced approach that cannot be implemented by KBP and MCO automation alone. However, the scripting and automation methods discussed herein can be used in conjunction with KBP and MCO automation methods. For example, user commands can be used to guide the use and results from KBP or MCO, or to apply aspects of verification or optimization in some automated plan results that are assisted by KBP or MCO.

[0049] Figure 3 An operational sequence for treatment automation is shown (e.g., with use cases involving treatment planning automation workflow 130, which indicates the creation of scripted commands 215). The operational sequence includes an example of an automation workflow for chatbot / voice control driven automation in a treatment planning system. For example, the treatment planning system 120 can be adapted to produce an automated VMAT plan and optimize the VMAT plan based on the use of some templates for a patient.

[0050] First, a text or voice input 310 is collected, which includes user commands related to a radiotherapy planning operation. The user commands can be provided by a user in the form of a voice recording or audio capture, or in the form of a text string. When the commands are in the form of a voice recording or audio capture, an automatic speech recognition (ASR) tool (e.g., implemented in a speech-to-text or voice-to-text engine) can be used to transcribe the speech into a text string. This allows the user to speak / type commands, such as “Please help me create a VMAT plan for patient 002, using a plan template named ‘PKUNPC6160 cGy’, and then compute and optimize it.”

[0051] Next, the text or voice input is provided to natural language processing by a chatbot or other language engine. In one example, the natural language processing 320 can be implemented by an AI language model chatbot (e.g., a generative large language model (LLM) neural network) that is specifically trained to translate clinical needs from natural language interactions into inputs for software. For example, the chatbot’s algorithm can receive and parse the text to intelligently identify different commands, identify and generate different actions corresponding to different scripted commands of a workflow (e.g., scripted commands that must be used in a valid order, step-by-step).

[0052] As shown, a radiotherapy plan template and parameters for the radiotherapy plan template can be automatically selected and generated by the chatbot, including through the use of system calls or API that are combined together into a script. Code compilation is then created at 330, which in this example is configured to produce a script in a programming language that is understandable or usable by the treatment planning system (here, a C# scripting language that is executed by a code interpreter of the treatment planning system). The described script provides user interface automation 340 in the treatment planning system, and is shown to invoke actions in the user interface to select a patient, load images, instantiate a new plan, establish characteristics of a new plan, select a plan template for the plan, optimize the plan based on the plan template, and save the plan, among other actions.

[0053] The script and other script commands can be automatically executed to implement automation in the user interface or API of the treatment planning system 120. For example, a particular script can cause the treatment planning system 120 to customize a radiotherapy plan (e.g., a VMAT plan) for a particular patient, and invoke associated processing actions to optimize the generated plan. Automation 350 can include other interpretation of commands in conjunction with user interface automation 340 in the treatment planning system.

[0054] Figure 4 A detailed processing workflow is depicted that illustrates how the human commands provided in the voice input 401 or text input 403 can be converted into executable commands by the treatment planning automation workflow 130. In particular, the described processing workflow demonstrates how text values from human input are split into particular vectors and used to invoke particular script commands.

[0055] As shown, the voice input 401 or text input 403 is received or provided to the chatbot via the chatbot. The voice input 401 can be converted to text with a speech-to-text functionality 402, and the text input 403 can be converted or modified based on a text formatting functionality 404. The output of the functionality 402 or 404 produces a text formatted human provided command 405 for further processing.

[0056] The human provided command 405 is then provided to a text splitting functionality 410 to segment portions of the text into individual text vectors 411, 412, 413. In one example, the text splitting functionality 410 is implemented via Python coding. A text vector (also referred to as an embedding) is a representation of some portion of text, typically having some numerical value. The vectors are used to capture the semantic meaning of the text, such that words and phrases with similar meanings will have vectors that are close to each other in a high-dimensional space.

[0057] Each of these text vectors 411, 412, 413 can be used to produce different script actions or commands. For example, a first vector can correspond to one or more commands utilizing a first API, a second vector can correspond to one or more commands utilizing a second API, and so on. The text vectors 411, 412, 413 as shown are provided into respective script API templates (template 1 421, template 2 422, template 3 433) for generating different code functions. In some examples, separate vectors can be used to call multiple APIs or provide multiple parameters to separate APIs. Other forms of natural language processing can also be used.

[0058] One or more methods can be used to convert the respective text vectors into code based on the code templates designed to utilize the script API to call specific functions or actions. As a first example of text processing 461, the text vectors can be translated into using the corresponding script API based on a similarity metric that evaluates which API is most relevant to the function. This can be performed utilizing a fuzzy matching method, where the API parameters can be identified from the text vector and converted into script commands.

[0059] As a second example of text processing 462, the text vectors can be translated into using the corresponding script API based on a generation or prediction from an AI engine or co-pilot engine (a specialized engine that implies code completion). Here, the individual text vectors can be provided as input to the engine, where the engine model provides code that calls the script API as output. This co-pilot model can be similar to other types of programming engines that generate code based on text command input.

[0060] The code results from all of the text vectors can be combined or merged. As shown, a code compilation function 430 can be invoked to combine the results of the templates 421, 422, 423. This produces an executable template of script commands 440 in a programming language format. These commands can be executed to directly control one or more workflows of the treatment planning system 120 with the resulting automation 450.

[0061] The automated systems and methods discussed above provide a number of technical advantages. One significant advantage is that the generation of scripts from generative chatbots can greatly reduce the complexity of implementing automated workflows in a treatment planning system for users without any programming background. Even for skilled users with a programming background, the generation of scripts from generative chatbots can provide advanced capabilities that reduce the time to deploy and debug scripts. Further, both skilled and unskilled users can use natural language to directly control a treatment planning system and implement automation of various functions and capabilities. This can also assist users who can easily articulate in natural language what type of modification or change to a patient’s radiation therapy treatment plan is needed (including real-time or immediate changes) and the associated treatment constraints.

[0062] In further examples, users can provide human commands to not only create treatment plans, but to invoke various other treatment planning actions. Such commands can be invoked with real-world scenarios, such as sending a message to the system to ask it to run an automation, such as “create a treatment plan,” “export plan DICOM data to {proprietary format},” “print a treatment plan report,” etc. Such actions can be automated without the need for a customized training model for a particular patient, facility, or particular type of treatment plan or output. This enables customization of treatment plans by other users, whether skilled or unskilled, including those plans maintained by a medical organization supervisor. This use of generative AI can be used to translate needs into a format that software will understand to produce optimal outputs.

[0063] While the above examples are discussed with reference to treatment planning software, other types of automation and control of client applications can be provided. Client applications in this context can refer to additional types of computer programs configured to process medical data via one or more features, such as functions, methods, services, or components provided or controlled by the client application. Thus, client applications can include any software that can access and analyze medical data related to radiation therapy, such as any one or more of patient data, medical imaging data, treatment plan information, Dose-Volume Histograms (DVHs), dose information, dosimetric metrics, and clinical outcomes, etc.

[0064] Figure 5An illustrative diagram of an example language model 500 that can be used by a chatbot is shown. The language model is configured to predict and / or generate plausible words for a given input. In one example, the language model 500 comprises a neural network, and the language model 500 is based on a transformer network 502. A transformer network is a type of neural network that uses a self-attention mechanism. Transformer models are described, for example, in Vaswani et al., “Attention is all you need.” Advances in neural information processing systems 30 (2017).

[0065] In one example, the language model 500 comprises a Generative Pre-trained Transformer (GPT) network (configured as the transformer network 502). Language models that include a GPT network can be referred to as Large Language Models (LLMs). Transformer models are capable of handling long-term dependencies in text and / or natural language processing, and can be used in chatbots. A chatbot can include other components (not shown), such as components for pre-processing text before it is fed to the language model or components for post-processing the output of the language model before returning a response to a user. Pre-processing operations include, for example, operations for tokenization (i.e., splitting a text string into smaller sequences of characters, referred to as tokens). Post-processing operations depend on whether the language model is used for natural language inference, question answering tasks, similarity assessment tasks, classification tasks, and so on.

[0066] The text and position embeddings 501 represent an input to the transformer network 502 (or a similar GPT network). The text and position embeddings 501 correspond to an encoded form of the user query described herein. To obtain the text and position embeddings, byte pair encoding (BPE) can be used to convert a text string corresponding to the user query into a sequence of tokens. The tokens are selected from a predetermined vocabulary of tokens. The size of the vocabulary is denoted by V. For example, the vocabulary includes 32,000 to 64,000 tokens (i.e., V can be between 32,000 and 64,000). Each token is represented by a token embedding. For example, the size of the token embedding D can be any one of 768, 1024, 1280, or 1600. The vocabulary can be represented by a token embedding matrix of size V x D. The token embedding matrix can be denoted as W eFurthermore, for each token in the sequence, a positional encoding vector (which indicates the order of the tokens in the token sequence provided to the transformer) is added to the token embedding. This positional encoding vector has the same size (D) as the token embedding. For example, the positional encoding vector can represent any one of the 1024 positions in the input sequence. The positional encoding vector can be learned (i.e., it includes parameters determined during training). The positional encoding vector is represented as W. p The length of the input sequence (e.g., 1024) can be referred to as the context size. The text and location embeddings 501 are obtained based on the token embedding matrix and the location-encoded vectors. The text and location embeddings 501 consist of a sequence of vectors representing the user query, where each vector has a length D.

[0067] Figure 5 The language model 500 shown includes a transformer network 502. The transformer network 502 comprises a stack of twelve (12x) decoder blocks 503. Each decoder block 503 includes a masked self-attention layer 505 and a feedforward neural network 507. The first decoder block receives a sequence of vectors from text and positional embeddings 501, where each vector represents a token. Layer normalization (not shown) can be provided at the inputs of each masked self-attention layer 505 and each feedforward neural network 507. Additional layer normalization can be included after the last (in this case, the twelfth) self-attention layer. The purpose of layer normalization is to provide smoother gradients, faster training, and improved accuracy.

[0068] For ease of explanation, consider a sequence of four tokens, represented by four vectors x1, x2, x3, and x4. Each of x1, x2, x3, and x4 has a length D. The tokens are fed into the mask of the first decoder block 503 from the attention layer 505. For each token, a query vector q, a key vector k, and a value vector v are obtained. This is achieved by using the weight matrix W... Q W K and W V Multiply by the token vector to obtain the query, key, and value vector for each token. Weight matrix W Q W K and W V It has a size D. The weight matrix includes the weights determined during network training. For each token i, the query vector q is... i Multiply by the (dot product) key vector (for all tokens) to obtain a score indicating how well other tokens match the current token. In masked self-attention, the score for future tokens (those that appear after the current token in the sequence) is set to 0. The value vector v for each token... iThey are multiplied by their respective scores and summed to obtain a vector denoted z. For x1, x2, x3, and x4, the corresponding vectors z1, z2, z3, and z4 are obtained. z1, z2, z3, and z4 are the outputs of the mask self-attention layer 505. These outputs are then presented to a feed-forward neural network 507 layer of the decoder block. The feed-forward NN is a fully connected NN in which the vector z i is projected (by multiplication by additional matrices - these matrices include weights determined during network training) to a result vector R j .

[0069] The result vector of each decoder block 503 is passed as input to the next decoder block. Each block 503 includes its own weight matrix (determined during model training). For the final decoder block, the result vector is used to derive the predictor 509 of the transformer model. In more detail, each result vector is multiplied by a token embedding matrix. The result of this multiplication corresponds to a score for each of the V tokens in the vocabulary. This result can be considered as the prediction 509. In some examples, the token with the highest score is selected and used to form the prediction. In another example, the k tokens with the highest scores are considered. For example, k = 40. The model iterates through all the tokens until the end of the sequence is reached.

[0070] The language model 500 can be trained as follows. The training process can include two phases: (i) unsupervised pre-training and (ii) supervised fine-tuning. In the unsupervised pre-training phase, given an unsupervised corpus of tokens u = {u1,..., u n}, a standard language modeling objective is used to maximize a likelihood function. For example, the likelihood function is:

[0071]

[0072] where k is the context size and the conditional probability P is modeled using a neural network with parameters Θ. The corpus of tokens can be obtained from a dataset including approximately 8 million documents (corresponding to 40 GB of text). An example is the WebText corpus of OpenAI. Alternatively, the corpus of tokens can be obtained from a dataset such as the BooksCorpus. The parameters can be determined using stochastic gradient descent. For example, the Adam optimization scheme can be used (with a learning rate of 2.5e-4).

[0073] The language model 500 applies the following operation on the input tokens to produce an output distribution over the target tokens:

[0074] R o = U W e + W p

[0075]

[0076] P(u) = softmax(R n W e )

[0077] Equation 2

[0078] Here, U is a vector of tokens, n denotes the number of decoder blocks (n = 12 in the Figure 5 transformer network 502, R l = transformer(R l-1 ) denotes the operation applied by the / -th decoder block of the transformer to the resulting vector R l-1 from the (l-1)-th block, and P(u) = softmax(R n W e ) denotes the application of the softmax function to the resulting vector R n from the last decoder block multiplied by the token embedding matrix W e to provide an output distribution.

[0079] In the supervised fine-tuning phase, a labeled dataset is used to further train the pre-trained model from the first phase. The dataset includes a sequence of input tokens c 1 ,...,c m and a label y. In one example, the label y is a gold standard for the sequence of input tokens and can include known commands and known command parameters of a medical treatment planning system corresponding to the sequence of input tokens. The label y can be obtained by labeling with human artificially and / or with machine assistance. The input is fed into the pre-trained model to obtain a resulting vector R m n from the last block of the transformer. The resulting vector R m n is fed into an additional linear output layer with parameters W y to obtain a prediction for y as follows: P(y|c 1 ,…,c m ) = softmax(R m n W y ).

[0080] The objective to maximize in the supervised fine-tuning phase is:

[0081]

[0082] As in the first stage, the parameters can be determined using stochastic gradient descent. For example, the Adam optimization scheme can be used (with a maximum learning rate of 2.5e-4).

[0083] Figure 5 The depicted model includes a single attention head. Alternatively, the model can include multiple attention heads.

[0084] In an alternative example, the language model 500 includes an architecture as described in Radford, A, et al. "Improving Language Understanding by Generative tPre-Training." OpenAI.

[0085] In an alternative example, the language model 500 includes a GPT-2 architecture as described in Radford, et al. "Language models are unsupervised multitask learners." OpenAI blog 1.8 (2019): 9.

[0086] In an alternative example, the language model 500 includes an architecture as described in Brown, T, et al. "Language models are few-shot learners." Advances in neural information processing systems, 33, 1877-1901 (2020).

[0087] In yet another alternative example, the language model 500 includes a transformer-based architecture as described in Touvron, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[0088] In yet another alternative example, the language model 500 includes a GPT-4 or PaLM2-based architecture.

[0089] In yet another alternative example, the language model 500 includes a Recurrent Neural Network (RNN) or a Long Short-Term Memory (LSTM) network.

[0090] Figure 6 is a flowchart 600 of an example method for generating automated commands for a treatment planning system used with radiotherapy treatment. The method can include determining a treatment plan for a patient based on the above references Figures 2 to 4Variations of the text processing and generation examples discussed.

[0091] At 601 : receiving text input from a user, wherein the text input includes information related to use of a treatment plan for a patient. In some examples, the text input from the user is derived from audio, and the method further includes: obtaining audio from the user; and converting the audio to the text input using a speech-to-text engine.

[0092] In some examples, receiving the text input includes using at least one chat session with a generative chatbot such that the at least one chat session includes a plurality of questions and responses related to a radiation treatment of a patient provided between the generative chatbot and the user. For example, the generative chatbot can include and use a trained language model, such as a Generative Pre-trained Transformer (GPT) network.

[0093] At 602: parsing the text input to identify a command and command parameters for a treatment planning system. In one example, parsing the text input includes: splitting the text input into a vector, and identifying at least one command of a script based on a relevance match of the vector. In another example, parsing the text input includes: splitting the text input into a vector, parsing the vector using an Artificial Intelligence (AI) coding engine, and identifying at least one command of a script using the coding engine.

[0094] In some examples, where the text is obtained using a generative chatbot, the method can include: receiving medical data associated with the patient, and determining the command and command parameters by the generative chatbot based on the medical data associated with the patient. Such medical data can include at least one of: an electronic medical record, an electronic health record, patient data, a treatment plan, a treatment plan template, and medical imaging data.

[0095] At 603: generate a script based on the text input (e.g., with a generative chatbot), the script including commands and command parameters. In one example, the commands of the treatment planning system are provided for programmatic use of a plurality of application programming interfaces (APIs) of the treatment planning system, and the command parameters are provided as input to the programmatic use of the plurality of APIs. In one example, the script is generated with a generative chatbot, and the generative chatbot operates based on a trained language model. For example, the trained language model can be trained based on known commands and known command parameters of the treatment planning system associated with creating and modifying radiation therapy treatment plans. The commands and command parameters can correspond to known (e.g., previously used) values of specific commands that invoke the plurality of APIs from the script based on previous treatment planning use cases and patient data examples. The trained language model can also be trained based on a plurality of constraints and instructions associated with creating and modifying radiation therapy treatment plans, the plurality of constraints and instructions including specific actions required or prohibited to generate a radiation therapy treatment plan in previous treatment planning use cases and patient data examples. In other examples, the trained language model is further trained to invoke one or more cost functions based on the plurality of constraints and instructions, the one or more cost functions used to establish a radiation therapy treatment plan for a radiation therapy treatment of a patient.

[0096] At 604: output the script for automation of the treatment planning system in conjunction with establishing a radiation therapy treatment plan for a radiation therapy treatment of a patient. The commands provided by the script can include use of a treatment planning template, where the use of the treatment planning template includes customizing a radiation therapy treatment plan based on the treatment planning template and optimizing the radiation therapy treatment plan.

[0097] In some examples, the output of the script includes presenting the script and receiving modifications to the script from a user. The script can include source code provided in a programmatic language format, and outputting the script can include providing the output of the source code for automation of a graphical user interface. In other examples, the script can include output as a visual script, and outputting the script can include providing the output of the visual script in a graphical user interface.

[0098] At 605: execute the script for automation of the treatment planning system to create or modify a treatment plan. As described above, execution of the script can result in creation, modification, and optimization of the treatment plan.

[0099] At 606: deploy the treatment plan(s) for the radiation therapy treatment(s) of the patient, including controlling a respective radiation machine based on the treatment plan. The deployment of the treatment plan can occur in conjunction with the aspects discussed above with reference to Figure 1 FIG. 6.

[0100] Figure 7 is a flowchart 700 of an example method for implementing automation in a radiotherapy treatment planning system. The method can be coordinated in conjunction with the operations of flowchart 600, and can include variations on the text processing and generation examples discussed above with reference to Figures 2 to 4

[0101] At 701 : receiving a natural language command in text form from a user, where the natural language command includes information related to an intended or planned use of a radiotherapy treatment for a patient. In some examples, the natural language command is received from the user in an audio format, and the method includes converting the audio of the natural language command to text.

[0102] At 702: parsing the text of the natural language command to identify a treatment planning system action and a parameter. In one example, parsing the text includes splitting the text into vectors, and identifying an automation command based on a relevance match of the vectors. In one example, parsing the text includes splitting the text into vectors, parsing the vectors with an artificial intelligence (AI) coding engine, and identifying an automation command with the coding engine.

[0103] At 703: generating an automation command for the treatment planning system based on the identified action and parameter. In some examples, prior to generating the automation command, the method can determine that the natural language command cannot be directly executed to control the treatment planning system, and must therefore be converted into another format as an automation command. The automation command can be provided in a script, and the automation command can be provided as source code in a programming language format within the script. The automation command can invoke a plurality of application programming interfaces (APIs) of the treatment planning system within the script, where the plurality of APIs are provided with programmatic use to provide the parameter. In some examples, the automation command includes use of a radiotherapy planning template, and where the use of the radiotherapy planning template includes customizing a radiotherapy treatment plan for the patient based on the radiotherapy planning template, and optimizing the radiotherapy treatment plan. Also in some examples, the automation command invokes at least one command for optimizing a radiotherapy treatment plan in the treatment planning system based on the natural language command.

[0104] ​The generation of the automation commands can be performed using a generative chatbot (e.g., a chatbot that includes a language model provided from a generative pre-trained transformer (GPT) network). The natural language commands can also be received from the user using a chat session with the generative chatbot, e.g., where the chat session includes a plurality of questions and responses provided between the generative chatbot and the user. In further examples, the generative chatbot can operate with a trained language model that is trained based on a plurality of constraints and instructions related to the generation of a radiation therapy treatment plan. Additionally, the trained language model can be trained to invoke one or more cost functions for optimizing the radiation therapy treatment plan based on the plurality of constraints and instructions. Additionally, in some examples, the method can include receiving medical data associated with the patient and determining the automation commands based on the medical data associated with the patient by the generative chatbot. Such medical data can include at least one of the following: electronic medical records, electronic health records, patient data, treatment plans, treatment plan templates, and medical imaging data.

[0105] At 704: implementing the automation commands in the treatment planning system (e.g., to create or modify a patient-specific treatment plan). In the context of generating a script, implementing the automation commands for programmatically controlling the treatment planning system includes executing the script. In one example, the result of implementing the automation commands can include outputting a radiation therapy treatment plan for a radiation therapy treatment, where the automation commands for programmatically controlling the treatment planning system are used to create the radiation therapy treatment plan based on the natural language commands. In another example, modifying a radiation therapy treatment plan associated with a patient involves using the automation commands for programmatically controlling the treatment planning system to update the radiation therapy treatment plan based on the natural language commands.

[0106] At 705: directing radiation therapy to the patient using a treatment device in accordance with the patient-specific treatment plan. The deployment of the treatment plan can occur in conjunction with the aspects discussed above with reference to Figure 1 FIG. 1.

[0107] Figure 8is a block diagram of an example of an apparatus, device, system, or machine 800 that can perform any one or more of the techniques (e.g., methods) discussed herein. In alternative embodiments, the machine 800 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine 800 can operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 800 can act or operate as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 800 can be a personal computer (PC), a tablet PC, a server computer, a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0108] The machine 800 can receive user commands through any of the input devices 812, UI navigation devices 814, display devices 810, or microphones (optional) as described herein. The machine 800 can output an indication that an action has been performed through any of the display devices 810 and signal generation devices 818. The machine 800 can output processed medical data in the machine-readable medium 822 as described herein. The machine 800 can be used to implement operations performed by the client application and / or chatbot and related data processing systems involved in translating natural language commands into automated commands.

[0109] Examples, as described herein, can include, or can operate by, logic or a number of components, or mechanisms. A circuit set is a collection of circuits implemented in a tangible entity containing hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership can be flexible over time and underlying hardware variability. A circuit set includes components that can carry out, alone or in combination, specific operations. In one example, hardware of the circuit set can be immutably designed to carry out or perform a specific operation (e.g., well-formed fixed logic combined with a predetermined number and arrangement of transistors). In an example, hardware of the circuit set can include variably-connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer-readable medium physically modified (e.g., magnetically, electrically, by movement of a mechanically-compliant part, etc.) to encode instructions of a specific operation. In

[0110] The machine (e.g., computer system) 800 can include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, a field-programmable gate array (FPGA), or any combination thereof), a main memory 804 and a static memory 806, some or all of which can communicate with one another via an interlink (e.g., bus) 830. The machine 800 can further include a display device 810, an input device 812 (e.g., a keyboard or other alphanumeric input device), and a user interface (UI) navigation device 814 (e.g., a mouse). In an example, the display device 810, input device 812 and UI navigation device 814 can be a touch screen display. The machine 800 can optionally include a microphone (e.g., for receiving spoken input for conversion into text input). The machine 800 can additionally include a storage device 808, such as a drive unit or other similar mass storage device or unit, a signal generation device 818, such as a speaker, a network interface device 820 to communicate with one or more networks 826, and one or more sensors 821, such as a Global Positioning System (GPS) sensor, compass, accelerometer, or other sensor. The machine 800 can include an output controller 828, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0111] The storage device 808 can include a machine readable medium 822 on which is stored one or more sets of data structures or instructions 824 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 824 can also reside, completely or at least partially, within the main memory 804, static memory 806, or hardware processor 802 during execution thereof by the machine 800. In an example, one or any combination of the hardware processor 802, the main memory 804, the static memory 806, or the storage device 808 can constitute machine readable media.

[0112] Although the machine-readable medium 822 is illustrated as a single medium, the term“machine-readable medium” can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 824. The term“machine-readable medium” can include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 800 and that cause the machine 800 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples can include solid-state memories, and optical and magnetic media. In one example, a massed machine-readable medium includes a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, the massed machine-readable medium is not a transitory propagating signal. Specific examples of massed machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0113] Unless specifically stated otherwise, discussions herein using terms such as“receiving,”“determining,”“comparing,”“implementing,”“maintaining,”“identifying,”“obtaining,”“accessing,” or the like, refer to the actions and processes of a computer system or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0114] Additional aspects and features of the present application are set forth in the following numbered examples, which illustrate the present application.

[0115] Example 1 is a method for generating automated commands for a treatment planning system used with a radiation therapy treatment, the method comprising: receiving a textual input from a user, the textual input including information related to a radiation therapy treatment plan for a patient; determining, based on parsing the textual input, a command for the treatment planning system and a command parameter to be used by the treatment planning system; generating, based on the textual input, a script using a generative chatbot, wherein the script includes the command and the command parameter; and outputting the script for automation of the treatment planning system in conjunction with establishing a radiation therapy treatment plan for the radiation therapy treatment of the patient.

[0116] In Example 2, the subject matter of Example 1 optionally includes, wherein the method further comprises executing the script.

[0117] In Example 3, the subject matter of Example 2 optionally includes, wherein the command to provide the treatment planning system is provided for programmatic use of a plurality of application programming interfaces (APIs) of the treatment planning system, and wherein the command parameters are provided as input to the programmatic use of the plurality of APIs.

[0118] In Example 4, the subject matter of any one or more of Examples 1-3 optionally includes, wherein the text input from the user is derived from audio, and wherein the method further comprises obtaining the audio from the user; and converting the audio to the text input with a speech-to-text engine.

[0119] In Example 5, the subject matter of any one or more of Examples 1-4 optionally includes, wherein the generative chatbot operates based on a trained language model, and wherein the trained language model is trained based on known commands and known command parameters of a treatment planning system associated with creating and modifying radiation therapy treatment plans.

[0120] In Example 6, the subject matter of Example 5 optionally includes, wherein the trained language model is also trained based on a plurality of constraints and instructions associated with creating and modifying radiation therapy treatment plans.

[0121] In Example 7, the subject matter of Example 6 optionally includes, wherein the trained language model is further trained to invoke one or more cost functions for establishing a radiation therapy treatment plan for a radiation therapy treatment for a patient based on the plurality of constraints and instructions.

[0122] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes, wherein the command comprises use of a treatment planning template, and wherein the use of the treatment planning template comprises customizing a radiation therapy treatment plan based on the treatment planning template, and optimizing the radiation therapy treatment plan.

[0123] In Example 9, the subject matter of any one or more of Examples 1-8 optionally includes, wherein parsing the text input comprises splitting the text input into a vector, and identifying the at least one command of the script based on a relevance match of the vector.

[0124] In Example 10, the subject matter of any one or more of Examples 1-9 optionally includes, wherein parsing the text input comprises splitting the text input into a vector, parsing the vector with an artificial intelligence (AI) coding engine, and identifying the at least one command of the script with the coding engine.

[0125] In Example 11, the subject matter of any one or more of Examples 1-10 can optionally include the subject matter wherein the script includes source code provided in a programming language format, and wherein outputting the script includes providing an output of the source code for automation of the graphical user interface.

[0126] In Example 12, the subject matter of any one or more of Examples 1-11 can optionally include the subject matter wherein the script includes a visualization script, and wherein outputting the script includes providing an output of the visualization script in the graphical user interface.

[0127] In Example 13, the subject matter of any one or more of Examples 1-12 can optionally include the subject matter wherein outputting the script includes rendering the script; and receiving modifications to the script from a user.

[0128] In Example 14, the subject matter of any one or more of Examples 1-13 can optionally include the subject matter wherein receiving the textual input includes using at least one chat session with a generative chatbot, and wherein the at least one chat session includes a plurality of questions and responses provided between the generative chatbot and the user related to the radiation therapy treatment of the patient.

[0129] In Example 15, the subject matter of Example 14 can optionally include the subject matter wherein the generative chatbot includes a language model.

[0130] In Example 16, the subject matter of Example 15 can optionally include the subject matter wherein the language model includes a generative pre-trained transformer (GPT) network.

[0131] In Example 17, the subject matter of any one or more of Examples 1-16 can optionally include the subject matter wherein the method further includes receiving medical data associated with the patient; wherein the command and the command parameter are determined by the generative chatbot based on the medical data associated with the patient.

[0132] In Example 18, the subject matter of Example 17 can optionally include the subject matter wherein the medical data includes at least one of the following: an electronic medical record, an electronic health record, patient data, a treatment plan, a treatment plan template, and medical imaging data.

[0133] Example 19 is a system for generating automated commands for a radiation therapy treatment planning system, the system comprising: processing circuitry; and a memory including instructions stored thereon that, when executed by the processing circuitry, cause the processing circuitry to perform the method of any of Examples 1-18.

[0134] Example 20 is a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor of a computing device, cause the processor to perform the method of any one of Examples 1-18.

[0135] Example 21 is a method for implementing automation in a treatment planning system, the method comprising: receiving a natural language command in text form from a user, the natural language command including information related to an intended use of a radiation treatment for a patient; parsing the text of the natural language command to identify an action to be performed in the treatment planning system and a parameter to be used by the action to be performed in the treatment planning system; generating an automation command using a generative chatbot, the automation command controlling the treatment planning system based on the action and the parameter; and implementing the automation command for programmatically controlling the treatment planning system in conjunction with a computerized plan for the radiation treatment of the patient.

[0136] In Example 22, the subject matter of Example 21 optionally includes outputting a radiation treatment plan for the radiation treatment of the patient, wherein the automation command for programmatically controlling the treatment planning system is used to create the radiation treatment plan based on the natural language command.

[0137] In Example 23, the subject matter of any one or more of Examples 21-22 optionally includes modifying a radiation treatment plan associated with the patient, wherein the automation command for programmatically controlling the treatment planning system is used to update the radiation treatment plan based on the natural language command.

[0138] In Example 24, the subject matter of any one or more of Examples 21-23 optionally includes the subject matter wherein the automation command invokes at least one command to optimize a radiation treatment plan in the treatment planning system based on the natural language command.

[0139] In Example 25, the subject matter of any one or more of Examples 21-24 optionally includes the subject matter wherein the automation command is provided in a script, and wherein implementing the automation command for programmatically controlling the treatment planning system includes executing the script.

[0140] In Example 26, the subject matter of any one or more of Examples 21-25 optionally includes the subject matter wherein the automation command is provided as source code in a programming language format.

[0141] In Example 27, the subject matter of any one or more of Examples 21-26 optionally includes the subject matter wherein the automation command invokes a plurality of application programming interfaces (APIs) of the treatment planning system, and wherein the parameter is provided using programmatic use of the plurality of APIs.

[0142] In Example 28, the subject matter of any one or more of Examples 21-27 optionally include the subject matter of: wherein the automation command comprises use of a radiotherapy plan template, and wherein the use of the radiotherapy plan template comprises customizing a radiotherapy treatment plan for the patient based on the radiotherapy plan template and optimizing the radiotherapy treatment plan.

[0143] In Example 29, the subject matter of any one or more of Examples 21-28 optionally include the subject matter of: wherein the natural language command is received from the user in an audio format, and wherein the method further comprises converting the audio of the natural language command to text.

[0144] In Example 30, the subject matter of any one or more of Examples 21-29 optionally include the subject matter of: wherein parsing the text comprises splitting the text into vectors and identifying the automation command based on a relevance match of the vectors.

[0145] In Example 31, the subject matter of any one or more of Examples 21-30 optionally include the subject matter of: wherein parsing the text comprises splitting the text into vectors, parsing the vectors with an artificial intelligence (AI) coding engine, and identifying the automation command with the coding engine.

[0146] In Example 32, the subject matter of any one or more of Examples 21-31 optionally include the subject matter of: wherein the generative chatbot operates with a trained language model that is trained based on a plurality of constraints and instructions related to generating a radiotherapy treatment plan.

[0147] In Example 33, the subject matter of Example 32 optionally include the subject matter of: wherein the trained language model is trained to invoke one or more cost functions for optimizing the radiotherapy treatment plan based on the plurality of constraints and instructions.

[0148] In Example 34, the subject matter of any one or more of Examples 21-33 optionally include the subject matter of: wherein the natural language command is received from the user using a chat session with the generative chatbot, and wherein the chat session comprises a plurality of questions and responses provided between the generative chatbot and the user.

[0149] In Example 35, the subject matter of Example 34 optionally include the subject matter of: wherein the generative chatbot comprises a language model provided from a generative pre-trained transformer (GPT) network.

[0150] In Example 36, the subject matter of any one or more of Examples 21-35 optionally include determining that the natural language command cannot be directly executed to control the treatment planning system prior to generating the automation command.

[0151] In Example 37, the subject matter of any one or more of Examples 21-36 optionally include receiving medical data associated with the patient; wherein the automation command is determined by the generative chatbot based on the medical data associated with the patient.

[0152] In Example 38, the subject matter of Example 37 optionally includes the subject matter, wherein the medical data comprises at least one of: an electronic medical record, an electronic health record, patient data, a treatment plan, a treatment plan template, and medical imaging data.

[0153] Example 39 is a system for implementing automation in a radiotherapy treatment planning system, the system comprising: processing circuitry; and a memory comprising instructions stored thereon that, when executed by the processing circuitry, cause the processing circuitry to perform any of the methods of Examples 21-38.

[0154] Example 40 is a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor of a computing device, cause the processor to perform the method of any of Examples 21-38.

[0155] According to another example, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above examples. The computer program and / or code for performing such methods can be provided to a system (e.g. a system according to the second aspect) on one or more computer-readable media or, more generally, on a computer program product.

[0156] Examples described herein are computer-implemented methods. As some of the methods according to the examples can be implemented by software, some examples encompass computer code provided on any suitable carrier medium to a general purpose computer. The carrier medium can comprise any storage medium (e.g. a floppy disk, a CD ROM, a magnetic device or a programmable memory device) or any transient medium (e.g. any signal suitable for downloading the computer code from a server to a computer). The carrier medium can comprise a non-transitory computer-readable storage medium.

[0157] The present disclosure also relates to a system for performing the operations herein. The system can be specially configured for the desired purposes, or it can comprise a general purpose computer that is selectively activated or reconfigured by a computer program stored in the computer. The order of execution or performance of the operations in embodiments of the disclosure illustrated and described herein is not essential, unless otherwise specified. That is, the operations can be performed in any order unless expressly specified otherwise, and embodiments of the disclosure can include more or fewer operations than those disclosed herein. For example, it is contemplated that executing or performing a particular operation prior to, contemporaneously with, or subsequent to another operation is within the scope of aspects of the disclosure.

[0158] In view of the foregoing, it will be seen that the several objects of the disclosure are achieved and other advantageous results attained. As various changes could be made in the above-described constructions, products, and methods without departing from the scope of aspects of the disclosure as defined by the appended claims, it is intended that all matter contained in the above description be interpreted as illustrative only and not in a limiting sense.

[0159] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the disclosure without departing from its scope. While the dimensions, types of materials and coatings described herein are intended to define the parameters of the disclosure, they are by no means limiting, but are exemplary embodiments. Many other embodiments will be apparent to those of ordinary skill in the art upon reviewing the above description. The scope of the disclosure should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

[0160] Moreover, in the detailed description above, various features can be grouped together to streamline the disclosure. This should not be interpreted as intending that the disclosed features are essential to any claim. Rather, inventive subject matter can lie in fewer than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the detailed description, where each claim by itself is stood alone as a separate embodiment. The scope of the disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A method for generating automated commands for a treatment planning system used in conjunction with radiotherapy, the method comprising: Receive text input from a user, the text input including information related to the patient's radiotherapy treatment plan; Based on parsing the text input, determine the commands of the treatment planning system and the command parameters to be used by the treatment planning system; Based on the text input, a generative chatbot is used to generate a script, wherein the script includes the command and the command parameters; as well as Combined with the radiotherapy treatment plan established for the patient, the script for automating the treatment plan system is output.

2. The method according to claim 1, wherein, The method further includes: executing the script.

3. The method according to claim 2, wherein, The commands of the treatment planning system are provided for programmatic use of a plurality of application programming interfaces (APIs) of the treatment planning system, wherein the command parameters are provided as input to the programmatic use of the plurality of APIs.

4. The method according to claim 1, wherein, The text input from the user is derived from audio, and the method further includes: Obtain the audio from the user; and The audio is converted into the text input using a speech-to-text engine.

5. The method according to claim 1, wherein, The generative chatbot operates based on a trained language model, wherein the trained language model is trained based on known commands and known command parameters of the treatment plan system associated with creating and modifying radiotherapy treatment plans.

6. The method according to claim 5, wherein, The trained language model is also trained based on multiple constraints and instructions associated with creating and modifying the radiotherapy treatment plan.

7. The method according to claim 6, wherein, The trained language model is also trained to invoke one or more cost functions, based on the multiple constraints and instructions, to establish a radiotherapy treatment plan for the patient.

8. The method according to claim 1, wherein, The command includes the use of a treatment plan template, wherein the use of the treatment plan template includes: customizing the radiotherapy treatment plan based on the treatment plan template, and optimizing the radiotherapy treatment plan.

9. The method according to claim 1, wherein, Parsing the text input includes: splitting the text input into vectors, and identifying at least one command of the script based on relevance matching of the vectors.

10. The method according to claim 1, wherein, Parsing the text input includes: splitting the text input into vectors, parsing the vectors using an artificial intelligence (AI) encoding engine, and using the encoding engine to identify at least one command of the script.

11. The method according to claim 1, wherein, The script includes source code provided in a programming language format, and outputting the script includes providing the output of the source code for automation of a graphical user interface.

12. The method according to claim 1, wherein, The script includes a visualization script, and outputting the script includes providing the output of the visualization script in a graphical user interface.

13. The method according to claim 1, wherein, The output of the script includes: Present the script; and Receive modifications to the script from the user.

14. The method according to claim 1, wherein, Receiving the text input includes using at least one chat session with the generative chatbot, wherein the at least one chat session includes multiple questions and responses related to the patient's radiotherapy treatment provided between the generative chatbot and the user.

15. The method according to claim 14, wherein, The generative chatbot includes a language model.

16. The method according to claim 15, wherein, The language model includes a generative pre-trained transformer (GPT) network.

17. The method according to claim 1, wherein, The method further includes: Receive medical data associated with the patient; The command and the command parameters are determined by the generative chatbot based on the medical data associated with the patient.

18. The method according to claim 17, wherein, The medical data includes at least one of the following: electronic medical records, electronic health records, patient data, treatment plans, treatment plan templates, and medical imaging data.

19. A system for generating automated commands for a radiotherapy treatment planning system, the system comprising: Processing circuitry; as well as A memory including instructions stored thereon, which, when executed by the processing circuitry, cause the processing circuitry to perform the method according to any one of claims 1 to 18.

20. A non-transient computer-readable medium having instructions stored thereon, which, when executed by a processor of a computing device, cause the processor to perform the method according to any one of claims 1 to 18.

21. A method for automating a treatment planning system, the method comprising: Receive natural language commands in text form from the user, the natural language commands including information related to the intended use of radiotherapy for the patient; The text of the natural language command is parsed to identify the actions to be performed in the treatment planning system and the parameters to be used by the actions to be performed in the treatment planning system. Generative chatbots are used to generate automated commands that control the treatment planning system based on the actions and parameters. as well as In conjunction with the computerized plan for the patient's radiotherapy treatment, automated commands are implemented to programmatically control the treatment planning system.

22. The method according to claim 21, wherein, Also includes: Output a radiotherapy treatment plan for the radiotherapy treatment, wherein the automated commands for programmatically controlling the treatment plan system are used to create the radiotherapy treatment plan based on the natural language commands.

23. The method according to claim 21, wherein, Also includes: Modify the radiotherapy treatment plan associated with the patient, wherein the automated commands for programmatically controlling the treatment plan system are used to update the radiotherapy treatment plan based on the natural language commands.

24. The method according to claim 21, wherein, The automated commands, based on the natural language commands, invoke at least one command to optimize the radiotherapy treatment plan in the treatment planning system.

25. The method according to claim 21, wherein, The automation commands are provided in a script, and implementing the automation commands for programmatically controlling the treatment planning system includes executing the script.

26. The method according to claim 21, wherein, The automation commands are provided as source code in a programming language format.

27. The method according to claim 21, wherein, The automated command invokes multiple application programming interfaces (APIs) of the treatment planning system, and the parameters are provided using the programmatic use of the multiple APIs.

28. The method according to claim 21, wherein, The automated commands include the use of a radiotherapy plan template, wherein the use of the radiotherapy plan template includes: customizing the patient's radiotherapy treatment plan based on the radiotherapy plan template, and optimizing the radiotherapy treatment plan.

29. The method according to claim 21, wherein, The method receives the natural language command from the user in an audio format, and the method further includes converting the audio of the natural language command into the text.

30. The method according to claim 21, wherein, Parsing the text includes splitting the text into vectors and identifying the automated command based on relevance matching of the vectors.

31. The method according to claim 21, wherein, Parsing the text includes: splitting the text into vectors, parsing the vectors using an artificial intelligence (AI) encoding engine, and using the encoding engine to identify the automated commands.

32. The method according to claim 21, wherein, The generative chatbot operates using a trained language model, which is trained based on multiple constraints and instructions related to the generation of radiotherapy treatment plans.

33. The method according to claim 32, wherein, The trained language model is trained to invoke one or more cost functions for optimizing the radiotherapy treatment plan, based on the multiple constraints and instructions.

34. The method according to claim 21, wherein, The natural language commands are received from the user using a chat session with the generative chatbot, wherein the chat session includes multiple questions and responses provided between the generative chatbot and the user.

35. The method according to claim 34, wherein, The generative chatbot includes a language model provided from a generative pre-trained transformer (GPT) network.

36. The method according to claim 21, wherein, Also includes: Before generating the automated commands, it is determined that the natural language commands cannot be directly executed to control the treatment planning system.

37. The method according to claim 21, wherein, Also includes: Receive medical data associated with the patient; The automated commands are determined by the generative chatbot based on the medical data associated with the patient.

38. The method according to claim 37, wherein, The medical data includes at least one of the following: electronic medical records, electronic health records, patient data, treatment plans, treatment plan templates, and medical imaging data.

39. A system for automating a radiotherapy treatment planning system, the system comprising: Processing circuitry; as well as A memory including instructions stored thereon, which, when executed by the processing circuitry, cause the processing circuitry to perform the method according to any one of claims 21 to 38.

40. A non-transient computer-readable medium having instructions stored thereon, which, when executed by a processor of a computing device, cause the processor to perform the method according to any one of claims 21 to 38.

Images