Data correction using large-scale language model

Abstract

To modify data using a large-scale language model.SOLUTION: According to an examplary embodiment, instead of sending an entire UI screen state to an LLM to generate and modify one or a plurality of values on the UI screen, an identifier and data type of data on the UI screen are collected and sent to the LLM. The LLM is instructed to return a list of expressions defining what needs to be done to satisfy a user prompt. Then, a calling process evaluates the list of expressions and actually operates the data content.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This document relates generally to computer systems. More particularly, this document relates to using large-scale language models (LLMs) to modify data. [Background technology]

[0002] Enterprise resource planning (ERP) software integrates the various processes used to run an organization, such as finance, manufacturing, human resources, supply chain, service, and procurement, into a single system. These processes typically provide intelligence, visibility, and efficiency across most, if not all, aspects of the organization. An example of ERP software is SAP® S / 4 HANA from SAP SE of Walldorf, Germany.

[0003] LLM refers to artificial intelligence (AI) systems that are trained on massive datasets to understand and generate human language. These models are designed to process and understand natural language in ways that enable them to answer questions, engage in conversations, generate sentences, and perform a variety of language-related tasks. Summary of the Invention [Means for solving the problem]

[0004] 1. A system comprising: at least one hardware processor; a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large scale language model (LLM) prompt from the request, the identifier, and the data type that instructs an LLM to generate a list of expressions for implementing the modification of the data; sending the LLM prompt to the LLM; receiving the list of representations from the LLM; analyzing and evaluating the representation to effect the modification of the data in the user interface screen; and causing the display of the user interface screen with the modified data to a user.

[0005] The present disclosure is illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like reference numerals indicate similar elements. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram illustrating a system for manipulating data in a user interface screen using an LLM, according to an example embodiment. [Figure 2] 1 is a flow diagram illustrating a method for using an LLM to modify data on a user interface screen, according to an example embodiment. [Figure 3] FIG. 1 is a block diagram illustrating a software architecture that may be installed on any one or more of the devices described above. [Figure 4] FIG. 1 illustrates a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein, according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] The following description discusses illustrative systems, methods, techniques, instruction sequences, and computing machine program products. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various exemplary embodiments of the present subject matter. However, it will be apparent to those skilled in the art that various exemplary embodiments of the present subject matter may be practiced without these specific details.

[0008] In many software applications, users can enter data for storage or later manipulation. One example is that a user of an ERP system can enter data about products or transactions into an ERP client application for eventual storage in an associated database.

[0009] Various tools have been developed over the years to make data entry easier for users. However, all of these tools still rely on the user knowing or generating the data to be entered. For example, a transaction screen may allow a user to enter the amount of product sold in a transaction. Tools such as simple user interfaces, voice recognition, etc. may make it easier for a user to communicate that, for example, 100 products were sold in a transaction, but these tools still require the user to actually know or derive that 100 products were sold during the transaction.

[0010] One solution to this is to use artificial intelligence, particularly an LLM, to generate the data content to be entered. In such a case, for example, a user interface may be provided with a button that a user can select that causes the LLM to generate a value for a particular field of data. The user interface (UI) may then send the state of the UI screen (also known as context) on which the user is entering data to the LLM, along with instructions for generating the value for the particular field. The LLM can then generate the value and return it to the UI, which can then automatically enter it into the field. Similar processing may be performed to modify data already entered into a field or to perform mathematical operations (such as an average) on the data on the screen.

[0011] However, such an approach presents several technical challenges. First, potentially sensitive data is sent to the LLM. This can create security risks, as some of the data on the UI screen is private and should not, or even legally should not, be shared with third parties such as the LLM. Furthermore, in scenarios where there is a large amount of data to be sent to the LLM, such as when the field to be entered or modified is part of a very large table with a large amount of data, data communication constraints and constraints on LLM input size (typically based on the number of input data or tokens entered at the LLM prompt) make this solution unworkable. Third, with regard to mathematical operations, while the LLM can perform mathematical operations, its effectiveness at doing so is subpar, and its accuracy cannot be trusted.

[0012] In an exemplary embodiment, rather than sending the entire state of a UI screen to the LLM to generate or modify one or more values ​​on the UI screen, identifiers and data types of the data on the UI screen are collected and sent to the LLM. The LLM is instructed to return a list of expressions that define what needs to be done to satisfy the user prompt. A calling process then evaluates the list of expressions to actually manipulate the data content.

[0013] LLMs used to generate information are commonly called generative artificial intelligence (GAI) models. GAI models can be implemented as generative pre-trained transformer (GPT) models or bidirectional encoders. GPT models are a type of machine learning model that uses the Transformer architecture, a type of deep neural network that excels at processing sequential data such as natural language.

[0014] A bidirectional encoder is a type of neural network architecture in which an input sequence is processed in two directions: forward and backward. The forward direction starts at the beginning of the sequence and processes one input token at a time, while the backward direction starts at the end of the sequence and processes the input in reverse order.

[0015] By processing the input sequence in both directions, a bidirectional encoder can capture more contextual information and word-to-word dependencies, resulting in higher performance.

[0016] A bidirectional encoder can be implemented as a Bidirectional Long Short-Term Memory (BiLSTM) or BERT (Bidirectional Encoder Representations from Transformers) model.

[0017] Each direction has its own hidden state, and the final output is a combination of the two hidden states.

[0018] Long Short-Term Memories (LSTMs) are a type of recurrent neural network (RNN) designed to overcome the vanishing gradient problem in traditional RNNs, which can make it difficult to learn long-term dependencies in sequential data.

[0019] An LSTM contains a cell state, which acts as a memory that stores information over time. The cell state is controlled by three gates: an input gate, a forget gate, and an output gate. The input gate determines how much new information is added to the cell state, while the forget gate determines how much old information is discarded. The output gate determines how much of the cell state is used to calculate the output. Each gate is controlled by a sigmoid activation function, which outputs a value between 0 and 1 that determines the amount of information that passes through the gate.

[0020] In BiLSTM, there are separate LSTMs for the forward and backward directions. At each time step, the forward and backward LSTM cells receive the current input token and hidden state from the previous time step. The forward LSTM processes the input tokens from left to right, while the backward LSTM processes them from right to left.

[0021] The output of each LSTM cell at each time step is a combination of the input token and the previous hidden state, which allows the model to capture both short-term and long-term dependencies between the input tokens.

[0022] BERT applies bidirectional training of models known as Transformers to language modeling, in contrast to prior art solutions that look at text sequences either left-to-right or a combination of left-to-right and right-to-left. Bidirectionally trained language models have a deeper sense of linguistic context and flow than unidirectional language models.

[0023] More specifically, a Transformer encoder is considered bidirectional because it reads an entire sequence of information at once (although one could argue that it is actually non-directional). This property allows the model to learn the context of the information based on everything around it.

[0024] In other exemplary embodiments, a generative adversarial network (GAN) embodiment may be used. A GAN is a supervised machine learning model with two submodels: a generative model that is trained to generate new examples, and a discriminative model that attempts to classify examples as either real or generated. These two models are trained together in an adversarial manner (using a zero-sum game according to game theory) until the discriminative model is fooled roughly once every two times (meaning the generative model is generating plausible examples).

[0025] A generative model takes fixed-length random vectors as input and generates samples in the domain of interest. The vectors are drawn randomly from a Gaussian distribution, and the vectors are used to seed the generative process. After training, points in this multidimensional vector space correspond to points in the domain of interest, forming a condensed representation of the data distribution. This vector space is called a latent space or a vector space of latent variables. Latent or hidden variables are variables that are important to the domain but are not directly observable.

[0026] A discriminative model takes examples from a domain as input (real or generated) and predicts a binary classification label: real or fake (generated).

[0027] While generative modeling is an unsupervised learning problem, a clever property of GAN architectures is that training a generative model is framed as a supervised learning problem.

[0028] Two models, a generator and a classifier, are trained together: the generator produces batches of samples, which, along with real examples from the domain, are fed to the classifier to be classified as real or fake.

[0029] The classifier is then updated to become better at distinguishing between real and fake samples in the next round, and importantly, the generator is updated based on how well the generated samples fool the classifier.

[0030] In another exemplary embodiment, the GAI model is a variational algorithmic encoder (VAE) model. A VAE comprises an encoder network that compresses input data into a low-dimensional representation called a latent code, and a decoder network that generates new data from the latent code. In either case, the GAI model includes a generative classifier, which may be implemented, for example, as a naive Bayes classifier.

[0031] Although this solution works with any type of GAI model, an implementation is described specifically for use with LLM.

[0032] 1 is a block diagram illustrating a system 100 for manipulating data in a user interface screen using an LLM 102, according to an example embodiment. A user interface 104 running on a client device 106 behaves as if it were displaying a screen in which data may be entered or manipulated. Note that the term "screen" in this disclosure should be interpreted broadly to mean any data structure that is at least partially displayed on the display of some computing device. It is not necessary that the entire screen be actually displayed at one time. For example, a UI screen may include a large 100x100 table, only 10% of which may be visible at any one time.

[0033] An artificial intelligence (AI) handler 108 on the client device 106 then retrieves the list of identifiers and data types of the UI screens from the user interface 104. In some exemplary embodiments, the list of identifiers and data types may be extracted from a JavaScript Object Notation (JSON) file associated with the UI screens.

[0034] The AI ​​Handler 108 then appends the list of identifiers and data types to a system prompt and sends the combined prompt to the LLM 102. The system prompt instructs the LLM to generate a list of representations in an abstract syntax tree (AST) to effect the creation or modification of data in the UI screen. In particular, the LLM is not instructed to directly create or modify the data itself, but to generate a list of representations to effect such creation or modification.

[0035] The list of expressions is then returned to the AI ​​handler 108, which parses them and evaluates the expressions to modify the data in the UI screen. The modified UI screen is then returned to the user interface 104 for display.

[0036] An AST is a tree representation of the abstract syntax structure of source code written in a programming language. Each node in the tree represents a construct present in the source code. The syntax is "abstract" in that it does not represent every detail that appears in the actual syntax, but only structural or content-related details. An AST contains nodes, edges, and leaves. Nodes are the basic elements of an AST. Each node represents a part of a program, such as a statement, expression, or declaration. Nodes are connected in a way that reflects the syntactic structure of the code. Edges represent parent-child relationships between nodes. Nodes representing control structures may have child nodes that represent, for example, the state and body of the structure. Nodes that have no children are called leaves. They represent the most basic elements, such as identifiers, characters, or operators.

[0037] Essentially, the LLM 102 converts a "user prompt" into a list of expressions using a specific list of identifiers and their data types, as well as specific instructions contained in the system prompt. A user prompt is information about what a user of the user interface 104 is trying to accomplish. This information can take many forms. In some exemplary embodiments, the user may explicitly specify what they want to do ("Set order to match the 1st row and the column labeled "sold" of overview"). In other exemplary embodiments, the AI ​​handler 108 may infer what the user wants to do based on context, such as by pressing a button labeled "autofill" next to a blank field the user wants to fill in.

[0038] Below are examples of system prompts that may be used in an exemplary embodiment. It is necessary to convert the expressions contained in the user's natural language input into an abstract syntax tree in JSON format. The only operators allowed in an abstract syntax tree are: - JavaScript operators that have the same meaning as in JavaScript, and -Math provides mathematical functions like those in Math.js

[0039] Below is an example of a user prompt and the corresponding listing of the AST-formatted representation generated by LLM 102, as well as the results of parsing and processing the representation. 1. User prompt: Set Order to row 1, column Sold of Overview Result: { "type":"Program", "body":[ { "type":"AssignmentExpression", "operator":"=", "left":{ "type":"Identifier", "name":"Order" }, "right":{ "type":"MemberExpression", "object":{ "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Overview" }, "property":{ "type":"Literal", "value":"Row:1" }, }, "property":{ "type":"Identifier", "name":"Column:Sold" }, } } ] } Which is: Order = Overview[1][Sold] 2. User prompt: Set 10 percent of Stock to Order, round Result: { "type":"Program", "body":[ { "type":"AssignmentExpression", "operator":"=", "left":{ "type":"Identifier", "name":"Order" }, "right":{ "type":"CallExpression", "callee":{ "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Math" }, "property":{ "type":"Identifier", "name":"round" }, }, "arguments":[ { "type":"BinaryExpression", "operator":"*", "left":{ "type":"Identifier", "name":"Stock" }, "right":{ "type":"Literal", "value":0.1 } } ] } } ] } Which is: Order = Math.round ($Stock * 0.1) 3. User prompt: Set the average of column Sold to Order and round the result Result: { "type":"Program", "body":[ { "type":"AssignmentExpression", "operator":"=", "left":{ "type":"Identifier", "name":"Order" }, "right":{ "type":"CallExpression", "callee":{ "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Math" }, "property":{ "type":"Identifier", "name":"round" }, }, "arguments":[ { "type":"CallExpression", "callee":{ "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Math" }, "property":{ "type":"Identifier", "name":"avg" }, }, "arguments":[ { "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Overview" }, "property":{ "type":"Identifier", "name":"Column:Sold" }, } ] } ] } } ] } Which is: Order = Math.round (Math.avg (Overview.Sold))

[0040] As another example, a user interface screen may contain a table of car sales, with each sale having an identification of the sale, the name of the product sold, and the quantity. A user may wish to enter in a field for the estimated number of cars to be deducted from inventory for incoming orders ("order estimate"), and may therefore indicate that the field should have a value representing the average number of cars sold in each order. In that case, the user prompt might be "set order estimate to average order sold." Identifiers and data types for user interface screens may be as follows: Stock.type = Integer; Order.type = Integer; Delivery.type = Date; Product.type = String; Save.type = Boolean; Cancel.type = Boolean; Overview.type = Array [[],[]]; Overview.columns = (["Id", "Name", "Sold"]); Overview.columns.type = ([Integer, String, Integer]);

[0041] The LLM prompt may then cause the following list of expressions to be generated in AST format: { "type":"Program", "body":[ { "arguments":[ { "type":"CallExpression", "callee":{ "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Math" }, "property":{ "type":"Identifier", "name":"avg" }, }, "arguments":[ { "type":"MemberExpression", "object":{ "type":"Identifier", "name":"Overview" }, "property":{ "type":"Identifier", "name":"Column:Sold" } }

[0042] As previously mentioned, the above-described solution improves security by not transmitting the actual values ​​of any of the data fields in the user interface screen to the LLM (or otherwise to any external third party). In some exemplary embodiments, it may be desirable to further improve security by obscuring or otherwise hiding the actual identifiers of the data fields. This may be achieved, for example, by replacing some or all of the identifiers with dummy identifiers. Thus, for example, a UI element "Stock.type=String" indicating that the field identifier "Stock" has a type of "String" may be replaced with "A.type=String" before including it in the LLM prompt. When the list of representations is received from the LLM and ultimately parsed and processed, the dummy identifier "A" in the list of representations may be replaced with the actual identifier "Stock," thus preserving the privacy of the data field identifiers.

[0043] 2 is a flow diagram illustrating a method 200 of using an LLM to modify data in a user interface screen, according to an example embodiment. Step 210 comprises receiving a request to modify data in a user interface screen of a user interface. Step 220 comprises accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen. Step 230 comprises forming, from the request, the identifier, and the data type, a large-scale language model (LLM) prompt that instructs the LLM to generate a list of expressions for implementing the data modification.

[0044] Step 240 comprises sending an LLM prompt to the LLM. Step 250 comprises receiving a list of expressions from the LLM. Step 260 comprises parsing and evaluating the expressions to effect modification of data in the user interface screen. Step 270 comprises causing display of the user interface screen with the modified data to the user.

[0045] In view of the above disclosure, various examples are described below. It should be noted that one or more features of the examples may be considered separately or in combination in the disclosure of the present application.

[0046] Example 1 is a system comprising at least one hardware processor and a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations including receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming, from the request, the identifier, and the data type, a large scale language model (LLM) prompt that instructs an LLM to generate a list of expressions to perform the modification of the data; sending the LLM prompt to the LLM; receiving the list of expressions from the LLM; parsing and evaluating the expressions to perform the modification of the data in the user interface screen; and causing display of the user interface screen with the modified data to a user.

[0047] In Example 2, the subject matter of Example 1 includes not transmitting values ​​contained in multiple data fields to the LLM.

[0048] In Example 3, the subject matter of Examples 1-2 includes the list of expressions being in Abstract Syntax Tree (AST) format.

[0049] In Example 4, the subject matter of Examples 1-3 includes wherein forming includes adding a system prompt to the request, the identifier, and the data type.

[0050] In Example 5, the subject matter of Examples 1-4 includes the request being an explicit request from a user received via input by the user at a user interface.

[0051] In Example 6, the subject matter of Examples 1-5 includes that the request is inferred based on the user's movements within the user interface and based on the data context.

[0052] In Example 7, the subject matter of Examples 1-6 includes operations being performed in an artificial intelligence handler of a client device that also includes a user interface.

[0053] Example 8 is a method that includes receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large scale language model (LLM) prompt from the request, the identifier, and the data type that instructs an LLM to generate a list of expressions to implement the modification of the data; sending the LLM prompt to the LLM; receiving the list of expressions from the LLM; parsing and evaluating the expressions to implement the modification of the data in the user interface screen; and causing display of the user interface screen with the modified data to a user.

[0054] In Example 9, the subject matter of Example 8 includes not transmitting values ​​contained in the plurality of data fields to the LLM.

[0055] In Example 10, the subject matter of Examples 8-9 includes the list of expressions being in abstract syntax tree (AST) format.

[0056] In Example 11, the subject matter of Examples 8-10 includes, wherein forming includes adding a system prompt to the request, the identifier, and the data type.

[0057] In Example 12, the subject matter of Examples 8-11 includes the request being an explicit request from a user received via input by the user at a user interface.

[0058] In Example 13, the subject matter of Examples 8-12 includes that the request is inferred based on a user's movements within a user interface and based on a data context.

[0059] In Example 14, the subject matter of Examples 8-13 includes the method being implemented in an artificial intelligence handler of a client device that also includes a user interface.

[0060] Example 15 is a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large scale language model (LLM) prompt from the request, the identifier, and the data type that instructs an LLM to generate a list of expressions to perform the modification of the data; sending the LLM prompt to the LLM; receiving the list of expressions from the LLM; parsing and evaluating the expressions to perform the modification of the data in the user interface screen; and causing display of the user interface screen with the modified data to a user.

[0061] In Example 16, the subject matter of Example 15 includes not transmitting values ​​included in the plurality of data fields to the LLM.

[0062] In Example 17, the subject matter of Examples 15-16 includes the list of expressions being in abstract syntax tree (AST) format.

[0063] In Example 18, the subject matter of Examples 15-17 includes wherein forming includes adding a system prompt to the request, the identifier, and the data type.

[0064] In Example 19, the subject matter of Examples 15-18 includes the request being an explicit request from a user received via input by the user at a user interface.

[0065] In Example 20, the subject matter of Examples 15-19 includes the requirement being inferred based on a user's movements within a user interface and based on a data context.

[0066] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-20.

[0067] Example 22 is an apparatus equipped with means for realizing any one of Examples 1 to 20.

[0068] Example 23 is a system for realizing any one of Examples 1 to 20.

[0069] Example 24 is a method for realizing any of Examples 1 to 20.

[0070] FIG. 3 is a block diagram 300 illustrating a software architecture 302 that may be installed on any one or more of the devices described above. It will be understood that FIG. 3 is merely a non-limiting example of a software architecture, and that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, software architecture 302 is implemented by hardware, such as machine 400 of FIG. 4, which includes a processor 410, memory 430, and input / output (I / O) components 450. In this exemplary architecture, software architecture 302 may be conceptualized as a stack of layers, with each layer providing a particular function. For example, software architecture 302 includes layers such as operating system 304, libraries 306, frameworks 308, and applications 310. Optionally, according to some embodiments, applications 310 invoke API calls 312 through the software stack and receive messages 314 in response to API calls 312.

[0071] In various implementations, the operating system 304 manages hardware resources and provides general services. The operating system 304 includes, for example, a kernel 320, services 322, and drivers 324. According to some embodiments, the kernel 320 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 320 provides memory management, processor management (e.g., scheduling), component management, networking, and security configuration, among other functions. The services 322 may provide other general services for the other software layers. According to some embodiments, the drivers 324 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 324 may include a display driver, a camera driver, a BLUETOOTH® or BLUETOOTH® Low-Energy driver, a flash memory driver, a serial communications driver (e.g., a Universal Serial Bus (USB) driver), a Wi-Fi® driver, an audio driver, a power management driver, etc.

[0072] In some embodiments, libraries 306 provide low-level, general infrastructure utilized by applications 310. Libraries 306 may include system libraries 330 (e.g., the C standard library) that can provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, libraries 306 may include API libraries 332, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., the OpenGL framework used for rendering in two dimensions (2D) and three dimensions (3D) in a graphical context on a display), a database library (e.g., SQLite for providing various relational database functions), a web library (e.g., WebKit for providing web browsing functions), etc. Libraries 306 may also include a wide variety of other libraries 334 for providing many other APIs to application 310.

[0073] According to some embodiments, framework 308 provides a high-level, general infrastructure that can be utilized by applications 310. For example, framework 308 provides various graphical user interface features, high-level resource management, high-level location services, etc. Framework 308 can provide a wide range of other APIs that can be utilized by applications 310, some of which may be specific to a particular operating system 304 or platform.

[0074] In one exemplary embodiment, applications 310 include a home application 350, a contacts application 352, a browser application 354, an e-book application 356, a location application 358, a media application 360, a messaging application 362, a game application 364, and a wide variety of other applications, such as a third-party application 366. According to some embodiments, applications 310 are programs that perform functions defined in the program. Various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language), may be utilized to create one or more of applications 310, which may be structured in various ways. In one particular example, third-party application 366 (e.g., an application developed using the ANDROID® or IOS™ Software Development Kit (SDK) by an entity other than the vendor of a particular platform) may be mobile software that runs on a mobile operating system, such as IOS™, ANDROID®, WINDOWS® Phone, or another mobile operating system. In this example, the third-party application 366 can invoke API calls 312 provided by the operating system 304 to facilitate the functionality described herein.

[0075] FIG. 4 shows a diagrammatic representation of a machine 400 in the form of a computer system within which a set of instructions may be executed to cause the machine 400 to perform any one or more of the methodologies discussed herein, according to an exemplary embodiment. Specifically, FIG. 4 shows a diagrammatic representation of the machine 400 in the exemplary form of a computer system within which instructions 416 (e.g., software, programs, applications, applets, apps, or other executable code) may be executed to cause the machine 400 to perform any one or more of the methodologies discussed herein. For example, the instructions 416 may cause the machine 400 to perform method 200 of FIG. 2. Additionally or alternatively, the instructions 416 may implement FIGS. 1-2, etc. The instructions 416 transform a general, unprogrammed machine 400 into a specific machine 400 that is programmed to perform the functions described and illustrated in the described manner. In alternative embodiments, the machine 400 may operate as a standalone device or be coupled (e.g., networked) to other machines. In a networked deployment, machine 400 may operate as a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 400 may comprise, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a notebook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a mobile phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, instructions 416 that specify actions to be taken by machine 400.Furthermore, although only a single machine 400 is shown, the term "machine" should be interpreted to include a collection of machines 400 that individually or jointly execute instructions 416 to perform any one or more of the methodologies discussed herein.

[0076] Machine 400 may include processor 410, memory 430, and I / O components 450, which may be configured to communicate with each other via bus 402 or the like. In an example embodiment, processor 410 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 412 and processor 414, which may execute instructions 416. The term “processor” is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as “cores”) that may execute instructions 416 simultaneously. Although FIG. 4 shows multiple processors 410, the machine 400 may include a single processor 412 having a single core, a single processor 412 having multiple cores (e.g., a multi-core processor 412), multiple processors 412, 414 having a single core, multiple processors 412, 414 having multiple cores, or any combination thereof.

[0077] Memory 430 may include a main memory 432, a static memory 434, and a storage unit 436, each of which is accessible to processor 410, such as via bus 402. Main memory 432, static memory 434, and storage unit 436 store instructions 416 that embody any one or more of the methods or functions described herein. Instructions 416 may also reside, completely or partially, within main memory 432, within static memory 434, within storage unit 436, within at least one of processors 410 (e.g., within a processor's cache memory), or any suitable combination thereof, during their execution by machine 400.

[0078] I / O components 450 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 450 included in a specific machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be understood that I / O components 450 may include many other components not shown in FIG. 4 . I / O components 450 are grouped according to function solely to simplify the following discussion, and this grouping is in no way limiting. In various exemplary embodiments, I / O components 450 may include output components 452 and input components 454. Output components 452 may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube [CRT]), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. Input components 454 may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to accept alphanumeric input, an optical keyboard, or other alphanumeric input component), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or another pointing device), tactile input components (e.g., physical buttons, a touchscreen that provides the position and / or force of a touch or touch gesture, or other tactile input component), audio input components (e.g., a microphone), etc.

[0079] In further exemplary embodiments, I / O component 450 may include a biometric component 456, a motion component 458, an environmental component 460, or a positioning component 462, among other different components. For example, biometric component 456 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, respiratory rate, or brain waves), identifying people (e.g., voice identification, retinal identification, face identification, fingerprint identification, or brain wave-based identification), etc. Motion component 458 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental components 460 may include, for example, an illumination sensor component (e.g., a light meter), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the air), or other components that may provide indicators, measurements, or signals corresponding to the surrounding physical environment. The positioning components 462 may include a location sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects atmospheric pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0080] Communication may be implemented using a wide variety of technologies. I / O component 450 may include a communication component 464 operable to couple machine 400 to network 480 or device 470 via coupling 482 and coupling 472, respectively. For example, communication component 464 may include a network interface component or another suitable device for interfacing with network 480. In further examples, communication component 464 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a Bluetooth® component (e.g., Bluetooth® Low Energy), a Wi-Fi® component, and other communication components for providing communication via other modalities. Device 470 may be another machine or any of a wide variety of peripheral devices (e.g., coupled via USB).

[0081] Moreover, the communication component 464 may include a component that detects an identifier or is operable to detect an identifier. For example, the communication component 464 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multidimensional barcodes such as QR Code, Aztec Code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). Additionally, various information may be derived via the communication component 464, such as location determination via Internet Protocol (IP) geolocation, location determination via Wi-Fi signal triangulation, location determination via detecting NFC beacon signals that may indicate a particular location, etc.

[0082] Various memories (e.g., 430, 432, 434, and / or memory of processor 410) and / or storage unit 436 may store one or more sets of instructions 416 and data structures (e.g., software) that embody or are utilized by any one or more of the methods or functions described herein. These instructions (e.g., instructions 416), when executed by processor 410, cause various operations to implement the disclosed embodiments.

[0083] As used herein, the terms “mechanical storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and may be used interchangeably. These terms refer to single or multiple storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions and / or data. These terms should be interpreted accordingly to include solid-state memory, including, but not limited to, memory internal or external to a processor, as well as optical and magnetic media. Specific examples of mechanical storage media, computer storage media, and / or device storage media include semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field programmable gate arrays (FPGAs), and non-volatile memory, including, by way of example, flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "mechanical storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media" discussed below.

[0084] In various exemplary embodiments, one or more portions of network 480 may be an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular network, a wireless network, a Wi-Fi network, another type of network, or a combination of two or more such networks. For example, network 480 or a portion of network 480 may include a wireless or cellular network, and connection 482 may be a code division multiple access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless connection. In this example, coupling 482 may implement any of various types of data transfer technologies, such as Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, Third Generation Partnership Project (3GPP®) including 3G, Fourth Generation Wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE) standards, others defined by various standards-setting organizations, other long-range protocols, or other data transfer technologies.

[0085] The instructions 416 may be transmitted or received over the network 480 using a transmission medium via a network interface device (e.g., a network interface component included in the communications component 464) and utilizing any one of several well-known transfer protocols (e.g., HTTP). Similarly, the instructions 416 may be transmitted or received using a transmission medium via a coupling 472 to the device 470 (e.g., a peer-to-peer coupling). The terms “transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be interpreted to include any intangible medium capable of storing, encoding, or carrying the instructions 416 for execution by the machine 400, including digital or analog communications signals or other intangible media for facilitating communication of such software. Accordingly, the terms “transmission medium” and “signal medium” should be interpreted to include any form of modulated data signal, carrier wave, or the like. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0086] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both mechanical storage media and transmission media. Thus, these terms include both storage devices / media and carrier wave / modulated data signals. [Explanation of symbols]

[0087] 100 systems 102 LLM 104 User Interface 106 client devices 108 AI Handler 302 Software Architecture 304 Operating System 306 Library 308 Framework 310 Applications 312 API Call 314 Messages 320 Kernel 322 Service 324 Driver 330 System 332 API 334 other libraries 350 Home Applications 352 Contacts Application 354 Browser Applications 356 e-book applications 358 Location Applications 360 Media Applications 362 messaging application 364 Game Application 366 Third-Party Applications 400 machines 402 Bus 410 processor 412 processors 414 processor 416 Command 430 memory 432 main memory 434 Static Memory 436 Memory Unit 450 I / O Components 452 Output Component 454 Input Component 456 Biometric Components 458 Motion Components 460 Environmental Components 462 Positioning Component 464 Communication Components 470 devices 472 Combine 480 Network 482 Combination

Claims

1. 1. A system comprising: at least one hardware processor; a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations comprising: receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large scale language model (LLM) prompt from the request, the identifier, and the data type that instructs an LLM to generate a list of expressions for implementing the modification of the data; sending the LLM prompt to the LLM; receiving the list of representations from the LLM; analyzing and evaluating the representation to effect the modification of the data in the user interface screen; causing a display of the user interface screen with the modified data to a user; and Including, the system.

2. The system of claim 1 , wherein values ​​included in the plurality of data fields are not transmitted to the LLM.

3. The system of claim 1 , wherein the list of representations is in an abstract syntax tree (AST) format.

4. The system of claim 1 , wherein the forming includes adding a system prompt to the request, the identifier, and the data type.

5. The system of claim 1 , wherein the request is an explicit request from the user received via input by the user at the user interface.

6. The system of claim 1 , wherein the request is inferred based on user movements within the user interface and based on the data context.

7. The system of claim 1 , wherein the actions are performed in an artificial intelligence handler of a client device that also includes the user interface.

8. receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large scale language model (LLM) prompt from the request, the identifier, and the data type that instructs an LLM to generate a list of expressions for implementing the modification of the data; sending the LLM prompt to the LLM; receiving the list of expressions from the LLM; analyzing and evaluating the representation to effect the modification of the data in the user interface screen; causing a display of the user interface screen with the modified data to a user.

9. The method of claim 8 , wherein values ​​included in the plurality of data fields are not transmitted to the LLM.

10. The method of claim 8 , wherein the list of representations is in Abstract Syntax Tree (AST) format.

11. The method of claim 8 , wherein the forming step includes adding a system prompt to the request, the identifier, and the data type.

12. The method of claim 8 , wherein the request is an explicit request from the user received via input by the user at the user interface.

13. The method of claim 8 , wherein the request is inferred based on user movements within the user interface and based on the data context.

14. The method of claim 8 , wherein the method is implemented in an artificial intelligence handler of a client device that also includes the user interface.

15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: receiving a request to modify data in a user interface screen of a user interface; accessing a data context for the user interface screen, the data context including an identifier and a data type for each of a plurality of data fields in the user interface screen; forming a large scale language model (LLM) prompt from the request, the identifier, and the data type that instructs an LLM to generate a list of expressions for implementing the modification of the data; sending the LLM prompt to the LLM; receiving the list of representations from the LLM; analyzing and evaluating the representation to effect the modification of the data in the user interface screen; and causing a display of the user interface screen having the modified data to a user.

16. 16. The non-transitory machine-readable medium of claim 15, wherein values ​​included in the plurality of data fields are not transmitted to the LLM.

17. 16. The non-transitory machine-readable medium of claim 15, wherein the list of representations is in an Abstract Syntax Tree (AST) format.

18. 16. The non-transitory machine-readable medium of claim 15, wherein the forming includes adding a system prompt to the request, the identifier, and the data type.

19. 16. The non-transitory machine-readable medium of claim 15, wherein the request is an explicit request from the user received via input by the user at the user interface.

20. 16. The non-transitory machine-readable medium of claim 15, wherein the request is inferred based on a user's movements within the user interface and based on the data context.

Images