Modifying data using large language models

Abstract

In an example embodiment, instead of sending the state of the UI picture as a whole to the LLM to generate or modify one or more values on the UI picture, an identifier and a data type of data on the UI picture are collected and sent to the LLM. The LLM is indicated to return a list of expressions defining what needs to do to meet the user cues. The list of process evaluation expressions is then invoked to actually manipulate the data content.

Description

TECHNICAL FIELD

[0001] This document relates generally to computer systems. More specifically, this document relates to using large language models (LLMs) to modify data. BACKGROUND

[0002] Enterprise resource planning (ERP) software is integrated into a single system that has various processes for running an organization, such as finance, manufacturing, human resources, supply chain, service, procurement, etc. These processes often provide intelligence, visibility, and efficiency across most, if not all, aspects of an organization. One example of ERP software is SAP® S / 4 HANA from SAP SE of Walldorf, Germany.

[0003] LLMs refer to artificial intelligence (AI) systems that have been trained on a broad set of data to understand and generate human language. These models are designed to process and understand natural language in ways that allow them to answer questions, engage in conversations, generate text, and perform various language-related tasks. BRIEF DESCRIPTION OF DRAWINGS

[0004] The disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. FIG. 1 is a block diagram showing a system for using an LLM to manipulate data in a user interface screen, according to an example embodiment.

[0005] Figure 1 is a block diagram showing a system for using an LLM to manipulate data in a user interface screen, according to an example embodiment.

[0006] Figure 2 is a flow diagram showing a method of using an LLM to modify data in a user interface screen, according to an example embodiment.

[0007] Figure 3 is a block diagram showing an architecture of software that can be installed on any one or more of the devices described above.

[0008] Figure 4 FIG. 1 is a block diagram showing a system for using an LLM to manipulate data in a user interface screen, according to an example embodiment. DETAILED DESCRIPTION

[0009] The following description discusses illustrative systems, methods, techniques, instruction sequences, and computer machine program products. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various example embodiments of the present subject matter. It will be apparent, however, to one skilled in the art that

[0010] In many software applications, users can enter data for storage or later manipulation. An example is that users of an ERP system can enter data about products or transactions into an ERP client application for ultimate storage in an associated database.

[0011] Various tools have been developed over the years to make data entry easier for users. However, all of these tools still rely on the user knowing or generating the data to be entered. For example, a transaction screen may allow the user to enter the number of products sold in a transaction. Tools such as simple user interfaces and voice recognition can make it easier for users to convey, for example, that 100 products were sold in a transaction, but these tools still require the user to actually know or infer that 100 products were sold during the transaction.

[0012] One solution to this problem is to use artificial intelligence, and specifically LLM, to generate the data content to be entered. In this case, for example, the user interface could be provided with a button that the user can select, causing the LLM to generate a value for a specific data field. The user interface (UI) could then send the LLM the state of the UI screen where the user is entering data (also known as the context), along with instructions to generate the value for the specific field. The LLM could then generate the value and return it to the UI, which could then automatically enter it into the field. A similar process could be used to modify data already entered into a field, or to perform mathematical operations (such as averaging) on ​​the data on the screen.

[0013] However, there are several technical issues with this approach. First, potentially sensitive data is sent to the LLM. This can create security risks because some data on the UI screen may be private and should not, or even legally cannot, be shared with third parties (such as the LLM). In addition, data communication restrictions and limitations on the size of LLM inputs (usually based on the number of input data or tokens entered in the LLM prompt) make this solution unfeasible for scenarios where there is a large amount of data to be sent to the LLM, such as when the field to be entered or modified is part of a very large table with a large amount of data. Third, with respect to mathematical operations, while the LLM can perform mathematical operations, the LLM's effectiveness in doing so is subpar and their accuracy cannot be relied upon.

[0014] In this example embodiment, rather than sending the entire state of the UI screen to the LLM to generate or modify one or more values ​​on the UI screen, the identifiers and data types of the data on the UI screen are collected and sent to the LLM. The LLM is instructed to return a list of expressions that define what needs to be done to satisfy the user prompt. The calling process then evaluates the expression list to actually manipulate the data content.

[0015] LLMs used to generate information are commonly referred to as generative artificial intelligence (GAI) models. GAI models can be implemented as generative pre-trained transformer (GPT) models or bidirectional encoders. GPT models are a type of machine learning model that uses a transformer architecture, which is a deep neural network that excels at processing sequential data, such as natural language.

[0016] Bidirectional encoders are a type of neural network architecture in which an input sequence is processed in two directions: forward and backward. The forward direction starts at the beginning of the sequence and processes the input one token at a time, while the backward direction starts at the end of the sequence and processes the input in reverse order.

[0017] By processing the input sequence in both directions, bidirectional encoders can capture more contextual information and dependencies between words, resulting in better performance.

[0018] Bidirectional encoders can be implemented as bidirectional long short-term memory (BiLSTM) or BERT (Bidirectional Encoder Representations from Transformers) models.

[0019] Each direction has its own hidden state, and the final output is a combination of the two hidden states.

[0020] Long short-term memory (LSTM) is a type of recurrent neural network (RNN) that is designed to overcome the vanishing gradient problem in traditional RNNs, which can make it difficult to learn long-term dependencies in sequential data.

[0021] LSTMs include a cell state, which serves as a memory that stores information over time. The cell state is controlled by three gates: an input gate, a forget gate, and an output gate. The input gate determines how much new information is added to the cell state, while the forget gate decides how much old information is discarded. The output gate determines how much of the cell state is used to compute the output. Each gate is controlled by a sigmoid activation function, which outputs a value between 0 and 1 that determines the amount of information that passes through the gate.

[0022] In BiLSTMs, there is a separate LSTM for the forward direction and the backward direction. At each time step, the forward and backward LSTM cells receive the current input token and the hidden state from the previous time step. The forward LSTM processes the input tokens from left to right, while the backward LSTM processes them from right to left.

[0023] The output of each LSTM cell at each time step is a combination of the input token and the previous hidden state, which allows the model to capture both short-term and long-term dependencies between input tokens.

[0024] BERT applies bidirectional training of the model, which will be called a Transformer, to language modeling. This contrasts with prior art solutions that look at a sequence of text from left to right or a combination of left to right and right to left. Bidirectionally trained language models have a deeper perception of linguistic context and flow than unidirectional language models.

[0025] More specifically, the Transformer encoder reads the entire sequence of information at once and is therefore considered bidirectional (although one can argue that it is actually non-directional). This property allows the model to learn the context of a piece of information based on all its surroundings.

[0026] In other example embodiments, a generative adversarial network (GAN) embodiment can be used. GANs are supervised machine learning models that have two sub-models: a generator model trained to generate new examples and a discriminator model that attempts to classify examples as real or generated. The two models are trained together in an adversarial manner (using a zero-sum game according to game theory) until the discriminator model is fooled about half the time, which means that the generator model is generating plausible examples.

[0027] The generator model takes a random vector of fixed length as input and generates a sample in the domain in question. The vector is randomly drawn according to a Gaussian distribution and is used to seed the generative process. After training, points in this multidimensional vector space will correspond to points in the problem domain, forming a compressed representation of the data distribution. This vector space is called the latent space, or the vector space composed of latent variables. Latent or hidden variables are those variables that are important to the domain but are not directly observable.

[0028] The discriminator model takes examples from the domain as input (real or generated) and predicts a binary class label of real or fake (generated).

[0029] Generative modeling is an unsupervised learning problem, although the clever property of the GAN architecture is that the training of the generative model is framed as a supervised learning problem.

[0030] The two models, the generator and the discriminator, are trained together. The generator generates a batch of samples, and these samples, along with real examples from the domain, are provided to the discriminator and classified as real or fake.

[0031] The discriminator is then updated to better discriminate between real and fake samples in the next round, and importantly, the generator is updated based on how well it fooled the discriminator based on the generated samples.

[0032] In another example embodiment, the GAI model is a variational autoencoder (VAE) model. A VAE includes an encoder network that compresses input data into a lower-dimensional representation called a latent code, and a decoder network that generates new data from the latent code. In either case, the GAI model includes a generative classifier, which can be implemented as, for example, a Naive Bayes classifier.

[0033] The present solution works with any type of GAI model, although implementations will be described that are specifically used with LLMs.

[0034] Figure 1 is a block diagram illustrating a system 100 for manipulating data in a user interface screen using an LLM 102, according to an example embodiment. A user interface 104 operating on a client device 106 is used to display a screen in which data can be entered or manipulated. It should be noted that the term “screen” in this disclosure should be interpreted broadly to mean any data structure that is displayed at least partially on a display of some computing device. It is not necessary that the entire screen be actually displayed at one time. For example, a UI screen can include a large 100x100 table, only 10% of which can be visible at any one time.

[0035] An artificial intelligence (AI) handler 108 on the client device 106 then obtains an identifier for the UI screen and a list of data types from the user interface 104. In some example embodiments, the identifier and the list of data types can be extracted from a JavaScript Object Notation (JSON) file associated with the UI screen.

[0036] The AI handler 108 then adds the identifier and the list of data types to a system prompt and sends the combined prompt to the LLM 102. The system prompt instructs the LLM to generate a list of expressions in an abstract syntax tree (AST) to perform generation or modification of the data in the UI screen. Notably, the LLM is not instructed to generate or modify the data itself, but rather to generate a list of expressions to cause such generation or modification to occur.

[0037] The list of expressions is then returned to the AI handler 108, which parses them and evaluates the expressions to modify the data in the UI screen. The modified UI screen is then passed back to the user interface 104 for display.

[0038] An AST is a tree representation of the abstract syntactic structure of source code written in a programming language. Each node of the tree represents a construct that occurs in the source code. The syntax is "abstract" because it does not represent every detail that occurs in a real grammar, but only details that are relevant to the structure or content. An AST consists of nodes, edges, and leaves. Nodes are the basic elements of an AST. Each node represents a part of a program, such as a statement, expression, or declaration. Nodes are connected in a way that reflects the syntactic structure of the code. Edges represent parent-child relationships between nodes. For example, a node representing a control structure may have child nodes that represent the conditional and body of the structure. Nodes without any children are called leaves. They represent the most basic elements, such as identifiers, literals, or operators.

[0039] Essentially, LLM 102 converts a "user prompt" into a list of expressions using a specific list of identifiers and their data types, along with specific instructions included in the system prompt. A user prompt is information about what the user of user interface 104 is trying to accomplish. This information can take a variety of forms. In some example embodiments, the user may explicitly specify what they want to do ("Set an order to match row 1 of the overview and the column labeled 'Sold'"). In other example embodiments, AI processor 108 may infer what the user wants to do based on context, such as by pressing a button that says "AutoFill" next to a blank field that the user wants to fill in.

[0040] The following are examples of system prompts that may be used in example embodiments:

[0041] ——————

[0042] You need to convert expressions contained in the user's natural language input into an abstract syntax tree in JSON.

[0043] As operators in the abstract syntax tree, only the following operators are allowed:

[0044] - JavaScript operators ensure the same meaning as in JavaScript, and

[0045] -Math provides mathematical functions similar to those in Math.js

[0046] The following is an example of a user prompt and a corresponding expression listing in AST form generated by LLM 102 and the result after parsing and processing the expression.

[0047] 1. User Tip: Set Order to row 1, Sold column of the overview

[0048] result:

[0049] {

[0050] "type": "Program",

[0051] "body": [

[0052] {

[0053] "type": "AssignmentExpression",

[0054] "operator": "=",

[0055] "left": {

[0056] "type": "Identifier",

[0057] "name": "Order"

[0058] },

[0059] "right": {

[0060] "type": "MemberExpression",

[0061] "object": {

[0062] "type": "MemberExpression",

[0063] "object": {

[0064] "type": "Identifier",

[0065] "name": "Overview"

[0066] },

[0067] "property": {

[0068] "type": "Literal",

[0069] "value": "Row:1"

[0070] },

[0071] },

[0072] "property": {

[0073] "type": "Identifier",

[0074] "name": "Column:Sold"

[0075] },

[0076] }

[0077] } ]

[0079] }

[0080] Which is: Order = Overview[1][Sold]

[0081] 2. User Tip: Set 10% of the inventory (Store) as the order, round up

[0082] result:

[0083] {

[0084] "type": "Program",

[0085] "body": [

[0086] {

[0087] "type": "AssignmentExpression",

[0088] "operator": "=",

[0089] "left": {

[0090] "type": "Identifier",

[0091] "name": "Order"

[0092] },

[0093] "right": {

[0094] "type": "CallExpression",

[0095] "callee": {

[0096] "type": "MemberExpression",

[0097] "object": {

[0098] "type": "Identifier",

[0099] "name": "Math"

[0100] },

[0101] "property": {

[0102] "type": "Identifier",

[0103] "name": "round"

[0104] },

[0105] },

[0106] "arguments": [

[0107] {

[0108] "type": "BinaryExpression",

[0109] "operator": " ",

[0110] "left": {

[0111] "type": "Identifier",

[0112] "name": "Stock"

[0113] },

[0114] "right": {

[0115] "type": "Literal",

[0116] "value": 0.1

[0117] }

[0118] } ]

[0120] }

[0121] } ]

[0123] }

[0124] Which is: Order = Math.round ($Stock 0.1)

[0125] 3. User Tip: Set the average of the Sold column to the order and round the result

[0126] result:

[0127] {

[0128] "type": "Program",

[0129] "body": [

[0130] {

[0131] "type": "AssignmentExpression",

[0132] "operator": "=",

[0133] "left": {

[0134] "type": "Identifier",

[0135] "name": "Order"

[0136] },

[0137] "right": {

[0138] "type": "CallExpression",

[0139] "callee": {

[0140] "type": "MemberExpression",

[0141] "object": {

[0142] "type": "Identifier",

[0143] "name": "Math"

[0144] },

[0145] "property": {

[0146] "type": "Identifier",

[0147] "name": "round"

[0148] },

[0149] },

[0150] "arguments": [

[0151] {

[0152] "type": "CallExpression",

[0153] "callee": {

[0154] "type": "MemberExpression",

[0155] "object": {

[0156] "type": "Identifier",

[0157] "name": "Math"

[0158] },

[0159] "property": {

[0160] "type": "Identifier",

[0161] "name": "avg"

[0162] },

[0163] },

[0164] "arguments": [

[0165] {

[0166] "type": "MemberExpression",

[0167] "object": {

[0168] "type": "Identifier",

[0169] "name": "Overview"

[0170] },

[0171] "property": {

[0172] "type": "Identifier",

[0173] "name": "Column:Sold"

[0174] },

[0175] } ]

[0177] } ]

[0179] }

[0180] } ]

[0182] }

[0183] Which is: Order = Math.round (Math.avg (Overview.Sold))

[0184] As another example, a user interface screen may contain a table of vehicle sales, where each sale has a sales ID, the name of the product sold, and the quantity. The user may wish to enter a field ("Order Estimate") for an estimated number of vehicles to be deducted from inventory for upcoming orders and may therefore indicate that this field should have a value representing the average number of vehicles sold per order. In this case, the user prompt may be "Set order estimate to the average number of orders sold." The identifiers and data types for the user interface screen may be as follows:

[0185] Stock.type = Integer;

[0186] Order.type=Integer;

[0187] Delivery.type=Date;

[0188] Product.type=String;

[0189] Save.type=Boolean;

[0190] Cancel.type=Boolean;

[0191] Overview.type=Array[[],[]];

[0192] Overview.columns=

[0193] (["Id", "Name", "Sold"]);

[0194] Overview.columns.type=

[0195] ([Integer,String,Integer]);

[0196] The LLM hint can then generate the following expression list in AST format:

[0197] {

[0198] "type": "Program",

[0199] "body": [

[0200] {

[0201] "arguments": [

[0202] {

[0203] "type": "CallExpression",

[0204] "callee": {

[0205] "type": "MemberExpression",

[0206] "object": {

[0207] "type": "Identifier",

[0208] "name": "Math"

[0209] },

[0210] "property": {

[0211] "type": "Identifier",

[0212] "name": "avg"

[0213] },

[0214] },

[0215] "arguments": [

[0216] {

[0217] "type": "MemberExpression",

[0218] "object": {

[0219] "type": "Identifier",

[0220] "name": "Overview"

[0221] },

[0222] "property": {

[0223] "type": "Identifier",

[0224] "name": "Column:Sold"

[0225] }

[0226] }

[0227] As previously mentioned, the above solution enhances security by not sending any actual values ​​of the data fields in the user interface screen to the LLM (or otherwise externally to any third party). In some example embodiments, it may be desirable to further enhance security by obscuring or otherwise hiding the actual identifiers of the data fields. This can be achieved by, for example, replacing some or all identifiers with virtual identifiers. Thus, for example, a UI element "Stock.type=String" indicating that the field identifier "Stock" has a type "String" may be replaced with "A.type=String" before it is included in the LLM prompt. When the expression list is received from the LLM and ultimately parsed and processed, the virtual identifier "A" in the expression list may be replaced with the actual identifier "Stock", thereby preserving the privacy of the data field identifiers.

[0228] Figure 2 2 is a flow chart illustrating a method 200 for modifying data in a user interface screen using an LLM according to an example embodiment. Step 210 includes receiving a request to modify data in a user interface screen of a user interface. Step 220 includes accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen. Step 230 includes forming a large language model (LLM) hint based on the request, the identifier, and the data type, the LLM hint instructing the LLM to generate an expression list to perform the data modification.

[0229] Step 240 includes sending an LLM prompt to the LLM. Step 250 includes receiving an expression list from the LLM. Step 260 includes parsing and evaluating the expression to perform the modification of the data in the user interface screen. Step 270 includes causing the user interface screen with the modified data to be displayed to the user.

[0230] In view of the above disclosure, various examples are set forth below. It should be noted that one or more features of the examples, alone or in combination, should be considered within the disclosure of this application.

[0231] Example 1 is a system comprising: at least one hardware processor; and a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context comprising an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large language model (LLM) prompt from the request, the identifier, and the data type, the LLM prompt instructing an LLM to generate a list of expressions to perform the modification to the data; sending the LLM prompt to the LLM; receiving a list of expressions from the LLM; parsing and evaluating the expressions to perform the modification to the data in the user interface screen; and causing display to a user of the user interface screen with the modified data.

[0232] In Example 2, the subject matter of Example 1 includes, wherein values contained in the plurality of data fields are not sent to the LLM.

[0233] In Example 3, the subject matter of Examples 1-2 includes, wherein the list of expressions is in an abstract syntax tree (AST) format.

[0234] In Example 4, the subject matter of Examples 1-3 includes, wherein the forming comprises adding a system prompt to the request, the identifier, and the data type.

[0235] In Example 5, the subject matter of Examples 1-4 includes, wherein the request is an explicit request from a user received via input by the user in the user interface.

[0236] In Example 6, the subject matter of Examples 1-5 includes, wherein the request is inferred based on a user action within the user interface and based on the data context.

[0237] In Example 7, the subject matter of Examples 1-6 includes, wherein the operations are performed on an artificial intelligence processor on a client device that also contains the user interface.

[0238] Example 8 is a method comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context of the user interface screen, the data context comprising an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large language model (LLM) prompt from the request, the identifier, and the data type, the LLM prompt instructing an LLM to generate a list of expressions to perform the modification to the data; sending the LLM prompt to the LLM; receiving a list of expressions from the LLM; parsing and evaluating the expressions to perform the modification to the data in the user interface screen; and causing display to a user of the user interface screen with the modified data.

[0239] In Example 9, the subject matter of Example 8 includes, wherein values contained in the plurality of data fields are not sent to the LLM.

[0240] In Example 10, the subject matter of Examples 8-9 includes, wherein the list of expressions is in an abstract syntax tree (AST) format.

[0241] In Example 11, the subject matter of Examples 8-10 includes, wherein the forming comprises adding a system prompt to the request, the identifier, and the data type.

[0242] In Example 12, the subject matter of Examples 8-11 includes, wherein the request is an explicit request from a user received via input by the user in the user interface.

[0243] In Example 13, the subject matter of Examples 8-12 includes, wherein the request is inferred based on a user action within the user interface and based on the data context.

[0244] In Example 14, the subject matter of Examples 8-13 includes, wherein the method is performed on an artificial intelligence processor on a client device that also contains the user interface.

[0245] Example 15 is a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large language model (LLM) prompt from the request, the identifier, and the data type, the LLM prompt instructing an LLM to generate a list of expressions to perform the modification to the data; sending the LLM prompt to the LLM; receiving a list of expressions from the LLM; parsing and evaluating the expressions to perform the modification to the data in the user interface screen; and causing display of the user interface screen with the modified data to a user.

[0246] In Example 16, the subject matter of Example 15 includes, wherein values contained in the plurality of data fields are not sent to the LLM.

[0247] In Example 17, the subject matter of Examples 15-16 includes, wherein the list of expressions is in an abstract syntax tree (AST) format.

[0248] In Example 18, the subject matter of Examples 15-17 includes, wherein the forming includes adding a system prompt to the request, the identifier, and the data type.

[0249] In Example 19, the subject matter of Examples 15-18 includes, wherein the request is an explicit request from a user received via input by the user in the user interface.

[0250] In Example 20, the subject matter of Examples 15-19 includes, wherein the request is inferred based on a user action within the user interface and based on the data context.

[0251] Example 21 is at least one machine readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-20.

[0252] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0253] Example 23 is a system for implementing any of Examples 1-20.

[0254] Example 24 is a method for implementing any of Examples 1-20.

[0255] Figure 3 is a block diagram 300 illustrating a software architecture 302, which can be installed on any one or more of the above devices.Figure 3 are merely non-limiting examples of a software architecture, and it will be appreciated that many other architectures can be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 302 is implemented by the hardware of machine 400 such as that illustrated in FIG. 4, which includes processors 410, memory 430, and input / output (I / O) components 450. In this example architecture, the software architecture 302 can be conceptualized as a stack of layers, where each layer can provide a particular functionality. For example, the software architecture 302 includes layers such as an operating system 304, libraries 306, frameworks 308, and applications 310. Operationally, the applications 310 invoke API calls 312 through the software stack and receive messages 314 in response to the API calls 312, consistent with some embodiments. Figure 4

[0256] In various implementations, the operating system 304 manages hardware resources and provides common services. The operating system 304 includes, for example, a kernel 320, services 322, and drivers 324. Consistent with some embodiments, the kernel 320 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 320 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 322 can provide other common services for the other software layers. According to some embodiments, the drivers 324 are responsible for controlling or interfacing with the actual hardware. For instance, the drivers 324 can include display drivers, camera drivers, Bluetooth® or Bluetooth® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth.

[0257] ​In some embodiments, libraries 306 provide a low-level common infrastructure used by the applications 310. Libraries 306 can include system libraries 330 (e.g., C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, libraries 306 can include API libraries 332 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render two dimensional (2D) and three dimensional (3D) graphics in an image context on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. Libraries 306 also include a wide variety of other libraries 334 to provide many other APIs to the applications 310.

[0258] According to some embodiments, frameworks 308 provide a high-level common infrastructure that can be utilized by the applications 310. For example, frameworks 308 provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks 308 can provide a broad spectrum of other APIs that can be utilized by the applications 310, some of which can be specific to a particular operating system 304 or platform.

[0259] In example embodiments, the applications 310 include a home application 350, a contacts application 352, a browser application 354, a book reader application 356, a location application 358, a media application 360, a messaging application 362, a game application 364, and a broad assortment of other applications such as a third party application 366. According to some embodiments, the applications 310 are programs that execute functions defined in the programs. Various programming languages can be employed to create one or more of the applications 310, a web site, a service, and / or other remote

[0260] Figure 4 A diagrammatic representation of a machine 400 in the form of a computer system is shown within which a set of instructions may be executed to cause the machine 400 to perform any one or more of the methodologies discussed herein, according to an example embodiment. Figure 4 A diagrammatic representation of a machine 400 in the example form of a computer system is shown within which instructions 416 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed for causing the machine 400 to perform any one or more of the methodologies discussed herein. For example, the instructions 416 may cause the machine 400 to perform Figure 2 Additionally or alternatively, instruction 416 may implement Figures 1-2 The instructions 416 transform a general-purpose, unprogrammed machine 400 into a specific machine 400 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 400 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 400 can operate as a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook computer, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network appliance, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 416 specifying the actions to be taken by the machine 400. Further, while a single machine 400 is illustrated, the term "machine" shall also be taken to include any collection of machines 400 that individually or jointly execute instructions 416 to perform any one or more of the methodologies discussed herein.

[0261] Machine 400 may include processor 410, memory 430, and I / O components 450, which may be configured to communicate with each other, for example, via bus 402. In an example embodiment, processor 410 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 412 and processor 414, which may execute instructions 416. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions 416 concurrently. AlthoughFigure 4 Multiple processors 410 are shown, but the machine 400 can include a single processor 412 with single core, a single processor 412 with multiple cores (e.g., a multi-core processor 412), multiple processors 412, 414 with single cores, multiple processors 412, 414 with multiple cores, or any combination thereof.

[0262] The memory 430 can include main memory 432, static memory 434, and storage unit 436 each accessible to the processors 410, such as via a bus 402. The main memory 432, static memory 434, and storage unit 436 store the instructions 416 embodying any one or more of the methodologies or functions described herein. The instructions 416 can also reside, completely or partially, within the main memory 432, within the static memory 434, within the storage unit 436, within at least one of the processors 410 (e.g., within a cache of the processor), or any suitable combination thereof, during

[0263] The I / O components 450 can include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I / O components 450 that are included in the particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 450 can include many other components that are not shown in FIG. 4. The I / O components 450 are grouped as shown primarily to simplify the following discussion and are not Figure 4 not limiting. In various example embodiments, the I / O components 450 can include output components 452 and input components 454. The output components 452 can include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components 454 can include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0264] In further example embodiments, the I / O components 450 can include biometric components 456, motion components 458, environmental components 460, or positioning components 462, among a myriad of other components. For example, the biometric components 456 can include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion components 458 can include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. The environmental components 460 can include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to a surrounding physical environment. The positioning components 462 can include location sensor components (e.g., a Global Positioning System (GPS) receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0265] Communication can be implemented using a wide variety of technologies. The I / O components 450 can include communication components 464 operable to couple the machine 400 to a network 480 or to devices 470 via coupling 482 and coupling 472, respectively. For example, the communication components 464 can include a network interface component or another suitable device to interface with a network 480. In further examples, the communication components 464 can include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth®components (e.g., Bluetooth®Low Energy), Wi-Fi®components, and other communication components to provide communication via other modalities. The devices 470 can be another machine or any of a wide array of devices (e.g., a

[0266] Moreover, the communication components 464 can detect identifiers or include components operable to detect identifiers. For example, the communication components 464 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information can be derived via the communication components 464, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that can indicate a particular location, and so forth.

[0267] The various memories (e.g., 430, 432, 434, and / or the memory of the processor 410) and / or the storage unit 436 can store one or more sets of instructions 416 and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 416), when executed by the processor 410, cause various operations to implement the disclosed embodiments.

[0268] As used herein, the terms “machine-storage medium,” “device-storage medium,” and “computer-storage medium” mean the same thing and can be used interchangeably. The terms “machine-storage medium,” “device-storage medium,” and “computer-storage medium” specifically exclude propagated signals and include both volatile and non-volatile, removable and non-removable media implemented in a method or technology for storage of information such as computer readable instructions, data structures, or program instructions. The computer-readable medium, machine-readable medium, and device-readable medium are tangible and non-transitory and include memory, including volatile and non-volatile, removable and non-removable media implemented using any technology for the

[0269] In various example embodiments, one or more portions of network 480 can be a self-organizing network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WW AN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 480 or a portion of network 480 can include a wireless or cellular network, and coupling 482 can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 482 can implement any of a variety of types of data transfer techniques for which a wireless or cellular network is appropriate, such as Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data Rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standards setting organizations, other long range protocols, or other data transfer techniques.

[0270] The instructions 416 can be transmitted or received using a transmission medium via a network interface device and utilizing any one of a number of well-known transfer protocols (e.g., HTTP). Similarly, the instructions 416 can be transmitted or received using a transmission medium via the coupling 472 (e.g., a peer-to-peer coupling). The term “transmission medium” and “signal medium” mean the same thing and can be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying the instructions 416 for execution by the machine 400, and includes digital or analog communications signals or other intangible media to facilitate communication of such software. Hence, the terms “transmission medium” and “signal medium” shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0271] The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and can be used interchangeably in this disclosure. The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” are defined to include both machine-storage media and transmission media. Thus, the terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” include both storage devices / media and carrier waves / modulated data signals.

Claims

1. A system comprising: at least one hardware processor; and A computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context of the user interface screen, the data context comprising an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large language model (LLM) hint based on the request, the identifier, and the data type, the LLM hint instructing the LLM to generate an expression list to perform a modification on the data; sending the LLM reminder to the LLM; receiving the expression list from the LLM; parsing and evaluating expressions to perform said modification of said data in said user interface screen; and The user interface screen is caused to be displayed to the user with the modified data.

2. The system according to claim 1, wherein: The values ​​contained in the plurality of data fields are not sent to the LLM.

3. The system according to claim 1, wherein: The expression list is in the abstract syntax tree (AST) format.

4. The system according to claim 1, wherein: The forming includes adding a system hint to the request, the identifier, and the data type.

5. The system according to claim 1, wherein: The request is an explicit request from the user received via input by the user in the user interface.

6. The system according to claim 1, wherein: The request is inferred based on a user action within the user interface and based on the data context.

7. The system according to claim 1, wherein: The operations are performed on an artificial intelligence processor on a client device that also includes the user interface.

8. A method comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context of the user interface screen, the data context comprising an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large language model (LLM) hint based on the request, the identifier, and the data type, the LLM hint instructing the LLM to generate an expression list to perform a modification on the data; sending the LLM reminder to the LLM; receiving the expression list from the LLM; parsing and evaluating expressions to perform said modification of said data in said user interface screen; and The user interface screen is caused to be displayed to the user with the modified data.

9. The method according to claim 8, wherein The values ​​contained in the plurality of data fields are not sent to the LLM.

10. The method according to claim 8, wherein The expression list is in the abstract syntax tree (AST) format.

11. The method according to claim 8, wherein The forming includes adding a system hint to the request, the identifier, and the data type.

12. The method according to claim 8, wherein The request is an explicit request from the user received via input by the user in the user interface.

13. The method according to claim 8, wherein The request is inferred based on a user action within the user interface and based on the data context.

14. The method according to claim 8, wherein The method is executed on an artificial intelligence processor on a client device that also includes the user interface.

15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context of the user interface screen, the data context comprising an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large language model (LLM) hint based on the request, the identifier, and the data type, the LLM hint instructing the LLM to generate an expression list to perform the modification on the data; sending the LLM reminder to the LLM; receiving the expression list from the LLM; parsing and evaluating expressions to perform said modification of said data in said user interface screen; and The user interface screen is caused to be displayed to the user with the modified data.

16. The non-transitory machine-readable medium of claim 15, wherein: The values ​​contained in the plurality of data fields are not sent to the LLM.

17. The non-transitory machine-readable medium of claim 15, wherein: The expression list is in the abstract syntax tree (AST) format.

18. The non-transitory machine-readable medium of claim 15, wherein: The forming includes adding a system hint to the request, the identifier, and the data type.

19. The non-transitory machine-readable medium of claim 15, wherein: The request is an explicit request from the user received via input by the user in the user interface.

20. The non-transitory machine-readable medium of claim 15, wherein: The request is inferred based on a user action within the user interface and based on the data context.

Images