Systems and methods for utilizing weak learners on large language models

Abstract

The disclosure relates to systems and methods for utilizing weak learners on large language models. A computer-implemented method includes converting tabular data into a textual representation, generating metadata associated with the textual representation of the tabular data, outputting one or more natural language data descriptions indicative of the tabular data in response to zero sample hints utilizing a Large Language Model (LLM) and the metadata and textual representation of the tabular data, and outputting one or more natural language data descriptions indicative of the tabular data using the LLM. The method includes receiving tabular data, outputting one or more natural language data descriptions, and adding prompts to output one or more summaries on the one or more natural language data descriptions, in response to a single summary having a minimum verification rate, selecting a single summary of the one or more summaries, receiving a query associated with the tabular data, outputting one or more predictions associated with the query, and outputting one or more predictions associated with the query. And in response to one or more predictions generated from one or more iterations satisfying a convergence threshold, outputting a final prediction associated with the query.

Description

Technical Field

[0001] The present disclosure relates to a generative system for responding to text queries or human speech input. Background Art

[0002] A weak learner can be a classifier that can achieve better performance than random chance by a given margin on any specified distribution of training data. One of the early breakthroughs in machine learning was the determination that weak learners are sufficient for arbitrarily strong classification via an ensembling procedure. This led to the development of boosting algorithms, a class of methods that continue to perform very well, especially on tabular datasets that lack regularity in the input space for vision or language tasks.

[0003] In a seemingly independent area of ​​research, large language models (LLMs) based on transformer architectures have come to dominate many natural language domains in recent years. These models are typically fine-tuned on data from new downstream tasks, but in recent years have also been shown to exhibit strong performance as zero-shot or few-shot learning alone, via prompting the model with a string of context. Summary of the invention

[0004] A first embodiment is a computer-implemented method for natural language processing, comprising receiving tabular data associated with one or more records, converting the tabular data into a textual representation indicating the tabular data, generating metadata associated with the textual representation of the tabular data, wherein the metadata indicates a description of the tabular data. The method comprises, for one or more iterations, outputting one or more natural language data descriptions indicating the tabular data in response to a zero-shot prompt using a large language model (LLM) and the metadata and textual representation of the tabular data, wherein the LLM comprises a neural network having a plurality of parameters. In addition, for one or more iterations, the method comprises outputting one or more summaries using the LLM and appending a prompt to the one or more natural language data descriptions, wherein the one or more summaries comprise less text than the one or more natural language data descriptions, for one or more iterations, selecting a single summary of the one or more summaries in response to a single summary having a minimum validation rate, receiving a query associated with the tabular data, for one or more iterations, outputting one or more predictions associated with the query using the LLM for the single summary and the query, and in response to the one or more predictions generated from the one or more iterations satisfying a convergence threshold, outputting a final prediction associated with the query, wherein the final prediction is selected in response to a weighted majority vote of all the one or more predictions generated from the one or more iterations.

[0005] A second embodiment discloses a system comprising an input interface to the system, wherein the input interface is configured to receive data associated with the system, and one or more processors in communication with the input interface, the one or more processors being programmed to receive tabular data associated with one or more records, convert the tabular data into a textual representation indicating the tabular data, generate metadata associated with the textual representation of the tabular data, wherein the metadata indicates a description of the tabular data. The processor is further programmed to, in response to a zero-shot prompt using a large language model (LLM) and the metadata and textual representation of the tabular data, output one or more natural language data descriptions indicating the tabular data, wherein the LLM includes a neural network having a plurality of parameters, output one or more summaries using the LLM and the one or more natural language data descriptions, wherein the one or more summaries include less text than the one or more natural language data descriptions, for one or more iterations, in response to a single summary having a minimum validation rate, select a single summary from the one or more summaries, receive a query associated with the tabular data, for one or more iterations, output one or more predictions associated with the query using the LLM for the single summary and the query; and in response to one or more predictions generated from the one or more iterations of outputting the one or more predictions satisfying a convergence threshold, output a final prediction associated with the query.

[0006] A third embodiment discloses a computer-implemented method for natural language processing, the method comprising receiving tabular data associated with one or more records, converting the tabular data into a text representation indicating the tabular data, generating metadata associated with the text representation of the tabular data, wherein the metadata indicates a description of the tabular data, in response to a zero-shot prompt using a large language model (LLM) and the metadata and text representation of the tabular data, outputting one or more natural language data descriptions indicating the tabular data; outputting one or more summaries using the LLM and appending a prompt to the one or more natural language data descriptions, wherein the one or more summaries include less text than the one or more natural language data descriptions, for one or more iterations, in response to a single summary with a minimum validation rate, selecting a single summary from the one or more summaries, creating a subset of the single summaries, receiving a query associated with the tabular data, for one or more iterations, using the LLM for one of the single summaries of the subset of the single summaries and the query, outputting one or more predictions associated with the query, creating a subset of the one or more predictions, and in response to the subset of the one or more predictions satisfying a convergence threshold, outputting a final prediction associated with the query, wherein the final prediction is selected in response to a weighted majority vote of the subset of the one or more predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1is a diagram of an example of a system having a framework for querying tasks according to an example embodiment of the present disclosure.

[0008] Figure 2 is a conceptual diagram of an example of a framework for a query task of a machine learning system according to an example embodiment of the present disclosure.

[0009] Figure 3 According to an exemplary embodiment of the present disclosure, Figure 1 Figure 2. An example of a control system of a machine learning system.

[0010] Figure 4 is about robot and / or automated personal assistant technology according to an example embodiment of the present disclosure Figure 3 FIG. 1 is a diagram of an example of a control system.

[0011] Figure 5 FIG. 1 shows a mobile machine technology according to an exemplary embodiment of the present disclosure. Figure 3 FIG. 1 is a diagram of an example of a control system.

[0012] Figure 6 The figure shows an embodiment of a systematic transformation of data points.

[0013] Figure 7 The diagram illustrates an embodiment of a process for generating a summary and using the summary to make predictions on new data. DETAILED DESCRIPTION

[0014] A central concept in practical and theoretical machine learning is a weak learner, i.e., a classifier that achieves better than random performance (on any given distribution of data), even by a small margin. Such weak learners form the practical basis of canonical machine learning methods such as boosting. In the embodiments disclosed below, the system and method illustrate that a large language model based on a prompt can operate effectively as the weak learner. Specifically, the system and method can illustrate the use of a large language model (LLM) as a weak learner in a boosting algorithm applied to tabular data. The system can show that by providing a text description of a tabular data sample (appropriately sampled according to the distribution of interest), the LLM can generate a summary of the sample, which is used as a template for classification and achieves the purpose of acting as a weak learner in the task. The system and method can combine these models into a boosting method, which in some settings can use the knowledge within the LLM to outperform traditional tree-based boosting. The model outperforms few-shot learning, and sometimes even outperforms more involved fine-tuning processes, especially for tasks involving a small number of data points. The results illustrate the potential of the prompt-based LLM not only as a few-shot learner itself, but also as a component of a larger machine learning pipeline.

[0015] The following disclosure is an example of the use of LLM as a weak learner in a boosting framework, particularly on tabular data, to show that by appropriately converting the tabular data into textual form, and asking the LLM to summarize a set of carefully selected examples from the data, the system can produce a summary of the examples that can be used as a template (e.g., a hint) for a tabular data classifier, and generally achieve this weak learning purpose. This enables us to incorporate this set of LLM-generated weak learners into the boosting framework accordingly.

[0016] The system can show that the resulting approach performs well in many settings, and outperforms zero-shot and few-shot classification, as well as "one-shot" summaries generated by LLMs. This is all done via a system that may not require any retraining or fine-tuning of the LLMs themselves, but rather simply via prompts. Furthermore, in some domains (particularly those with very few examples, where exploiting the prior knowledge built into the LLMs will be particularly important), the system can show that the approach can even outperform traditional tree-based boosting and LLM-based fine-tuning approaches, and that its performance may improve as the power of the LLMs increases. Overall, the system can highlight the potential for incorporating LLMs as subroutines of larger machine learning systems.

[0017] The following systems and methods can utilize LLMs to generate weak learners, and then use these weak learners within a boosting framework. This can be referred to as LLM summary boosting, because an embodiment can include an embodiment in which the core learning process can be a process of using a language model to create summaries of (particularly selected) samples from a data set. These summaries can serve as hints through which the system can make predictions for new examples. Finally, the system can use boosting to build an ensemble of these summaries that provides an overall prediction for new data points.

[0018] Figure 1 is a diagram of an example of a system 100 having a neural symbolic framework 200 for query tasks according to an example embodiment. The system 100 is configured to pre-train (or train) a machine learning system 210 via the neural symbolic framework 200. In addition, the system 100 is an example of a system configured to execute on the machine learning system 210 via the framework 200. After undergoing pre-training (or pre-training and zero-shot testing), the system 100 can be configured to employ the machine learning system 210 for use. Alternatively, the system 100 can be configured to enable the pre-trained (or pre-trained and zero-shot testing) machine learning system 210 to be employed and / or deployed in another system (e.g., Figure 3 system 300) for use.

[0019] System 100 includes at least a processing system 140. Processing system 140 includes at least an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any suitable processing technology, or any number and combination thereof. As described herein, processing system 140 is operable to provide the functionality of framework 200 and machine learning system 210.

[0020] The system 100 includes at least a memory system 120, which is operably connected to the processing system 140. In an example embodiment, the memory system 120 includes at least one non-transitory computer-readable medium configured to store and provide access to various data to enable at least the processing system 140 to perform operations and functions with respect to the framework 200 and the corresponding machine learning system 210, as disclosed herein. In an example embodiment, the memory system 120 includes a single computer-readable storage device or multiple computer-readable storage devices. The memory system 120 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable storage technology that can operate with the system 100. For example, in an example embodiment, the memory system 120 may include random access memory (RAM), read-only memory (ROM), flash memory, a disk drive, a memory card, an optical storage device, a magnetic storage device, a memory module, any suitable type of memory device, or any number and any combination thereof. With respect to the processing system 140 and / or other components of the system 100, the memory system 120 is local, remote, or a combination thereof (e.g., partially local and partially remote). For example, memory system 120 may include at least a cloud-based storage system (eg, a cloud-based database system) that is remote from processing system 140 and / or other components of system 100 .

[0021] The memory system 120 includes at least a framework 200, a machine learning system 210, machine learning data 220, and other related data 230, which are stored on and accessible from the memory system 120. The framework 200 includes computer-readable data that, when executed by the processing system 140, is configured to generate at least one training set with an appropriate number of query tasks for the machine learning system 210. In addition, the framework 200 includes computer-readable data that, when executed by the processing system 140, is configured to implement a zero-shot testing process (or a zero-shot evaluation process) to evaluate the pre-trained (or trained) machine learning system 210 on various common sense tasks. The computer-readable data may include instructions, codes, routines, various related data, any software technology, or any number and combination thereof.

[0022] In an example embodiment, the machine learning system 210 includes at least a machine learning model. More specifically, the machine learning system 210 includes at least one language model. For example, the machine learning system 210 includes a large language model (LLM), or any number of language models and combinations thereof.

[0023] In an example embodiment, the machine learning data 220 includes various data that the framework 200 uses to train, test, and develop the machine learning system 210. For example, the machine learning data 220 includes a global knowledge graph 220A. The global knowledge graph 220A is generated by combining various knowledge graphs 220B. The machine learning data 220 may also include one or more knowledge bases associated with one or more knowledge graphs 220B. The machine learning data 220 also includes a set of common sense task data sets 220C that cover a set of different tasks. In addition, the machine learning data 220 may also include various annotations, various loss data, various parameter data, and any related data that enable the neural symbolic framework 200 and the machine learning system 210 to perform the functions described herein while meeting certain performance criteria. At the same time, other related data 230 provides various data (e.g., operating system, etc.), which enables the system 100 to perform the functions discussed here.

[0024] In an example embodiment, if Figure 1 As shown, the system 100 is configured to include at least one human-machine interface (HMI) system 110. The HMI system 110 includes at least one user interface, at least one HMI device, or any combination thereof. For example, the HMI system 110 may include a visual user interface, an auditory user interface, a tactile user interface, any suitable user interface, or any number and combination thereof. The HMI system 110 is operable to communicate with an I / O system 130. The HMI system 110 is also operable to communicate with one or more other components of the system 100 (e.g., a processing system 140, a memory system 120, etc.). More specifically, for example, the processing system 140 is configured to obtain or extract a query or query task directly or indirectly from the HMI system 110, the memory system 120, and / or the I / O system 130. Upon receiving the query or query task, the processing system 140 is configured to provide a predicted answer to the query or query task via the machine learning system 210.

[0025] Additionally, system 100 includes other components that facilitate the training and / or execution of framework 200 and machine learning system 210. For example, Figure 1As shown, the memory system 120 is also configured to store other related data 230, which is related to the operation of the system 100 related to one or more components (e.g., the sensor system 110, the I / O system 130, and the other functional modules 150). In addition, the I / O system 130 may include an I / O interface, and may include one or more devices (e.g., a microphone, a keyboard device, a touch display device, a microphone, a mouse, a speaker device, etc.). In addition, the system 100 includes other functional modules 150, such as any appropriate hardware technology, software technology, or combination thereof that helps or facilitates the operation of the system 100. For example, the other functional modules 150 include communication technologies that enable the components of the system 100 to communicate with each other as described herein. Therefore, at least utilizing Figure 1 , system 100 is configured to execute framework 200 to pre-train (or train) machine learning system 210 to perform well on various query tasks (e.g., question answering tasks) in a zero-shot setting or when deployed / adopted for use in an application.

[0026] Figure 2 2 is a conceptual diagram of an embodiment of a neural symbolic framework 200 for a machine learning system 210. In an example embodiment, the framework 200 includes at least a query task generator 200A. The query task generator 200A may be configured to obtain a data structure (e.g., a triple) from a global knowledge graph 220A. Figure 2 As shown, the global knowledge graph 220A includes a plurality of different knowledge graphs 220B, wherein the total number of knowledge graphs 220B is Figure 2 denoted by "N". In this regard, "N" represents an integer at least greater than 2. The query task generator 200A is configured to generate query tasks based on the data structure of the global knowledge graph 220A. The query task generator 200A is configured to create a training set including an appropriate number of query tasks. The query task generator 200A is configured to pre-train or train the machine learning system 210 with at least one training set. The query task generator 200A is also configured to calculate at least one score of the machine learning system 210, and to fine-tune the machine learning system 210, for example, based on score data, loss data, and / or any other relevant data. The query task generator 200A ensures that the machine learning system 210 is pre-trained or trained to perform well on various common sense tasks in a zero-shot setting and / or when deployed / adopted for use in an application.

[0027] In addition, the framework 200 is configured to include a zero-shot evaluator 200B. The zero-shot evaluator 200B is configured to perform zero-shot testing on the machine learning system 210. Figure 2As indicated, the zero-shot evaluator 200B is configured to perform zero-shot testing during a post-training phase. The post-training phase refers to any phase that occurs after pre-training (or training) of the machine learning system 210, which has at least one training set generated by the query task generator 200A. The zero-shot evaluator 200B is configured to test the machine learning system 210 with the common sense task datasets 220C in a zero-shot manner. At this point, the machine learning system 210 is configured to process each common sense task dataset 220C without prior observation of the common sense task dataset 220C. The zero-shot evaluator 200B is configured to obtain a set of common sense task datasets 220C and apply each common sense task dataset 220C to the machine learning system 210. The set of common sense task datasets 220C includes a plurality of common sense task datasets 220C, wherein the total number of common sense task datasets 220C is in Figure 2 denoted by “M” in the query task generator 200A. In this regard, “M” represents an integer that is at least greater than 2. For example, each common sense task dataset 220C is different from other common sense task datasets 220C in the group with respect to the format of the query task and / or the type of knowledge associated with the query task. Utilizing the various common sense task datasets 220C, the zero-shot evaluator 200B is advantageously configured to demonstrate the effectiveness of the pre-training (or training) of the machine learning system 210 based on the training set generated by the query task generator 200A. In this regard, the zero-shot evaluator 200B is configured to provide a robust measure of the reasoning capability of the machine learning system 210. The zero-shot evaluator 200B is also configured to evaluate the machine learning system 210 and the impact of pre-training on the various common sense task datasets 220C.

[0028] As described above, the group of common sense task data sets 220C includes various common sense task data sets 220C. Each common sense task data set 220C is different from the training set generated by the query task generator 200A. The group of common sense task data sets 220C is a data set that the machine learning system 210 does not observe at all during its pre-training phase or training phase. In this regard, the group of common sense task data sets 220C is selected to cover different task sets, for example, at least about format (e.g., question answering, pronoun resolution, natural language reasoning, etc.), knowledge type (e.g., social knowledge, physical knowledge, etc.), or both format and knowledge type. For example, there may be a task data set 220C including a natural reasoning task, in which the beginning and end of the story are given, and wherein the task is to select a more credible hypothesis from a set of response options. In addition, there may be a task data set 220C including a wide range of common sense aspects, in which the task is to respond to a question by selecting one of five response options. As another example, there may be a task data set 220C focusing on physical reasoning, in which the task is to select a more reasonable response option from two possible continuations. In addition, there may be a task dataset 220C that focuses on reasoning based on social interactions, where the task includes some context, a question, and a set of response options. As another example, there may be a task dataset 220C involving pronoun resolution, where the task includes some context, emphasized pronouns, and response options provided as possible references. In addition, the set of common sense task datasets 220C is not limited to the aforementioned common sense task datasets 220C, but may include any task dataset 220C suitable for performing zero-shot testing on the machine learning system 210.

[0029] Figure 3 is a diagram of a system 300 that is configured to include at least a pre-trained (or trained) machine learning system 210. In this regard, the system 300 includes at least an HMI system 310, a control system 320, and an actuator system 330. The system 300 is configured such that the control system 320 controls the actuator system 330 based on input received from the HMI system 310. More specifically, the HMI system 310 includes one or more user interfaces and / or devices that communicate with one or more I / O devices of an I / O system 370. Once the input is obtained, the HMI system 310 is operable to communicate with the control system 320 via the input / output (I / O) system 370 and / or other functional modules 350 that include communication technology.

[0030] The control system 320 is configured to obtain input from the HMI system 310. Once the input is received, the control system 320 is operable to process the input via the processing system 340. In this regard, the processing system 340 includes at least one processor. For example, the processing system 340 includes an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a processing circuit, any suitable processing technology or any combination thereof. When at least processing the input received from the HI system 310, the processing system 340 is operable to provide a query or query task to the machine learning system 210 based on the input. The processing system 340 is also configured to generate a predicted answer via the machine learning system 210. The processing system 340 is configured to generate output data based on the predicted answer. The processing system 340 is configured to provide output data and / or predicted answers to the user via the I / O system 370 and / or the HMI system 310. In addition, the processing system 340 is operable to generate actuator control data based on the output data and / or predicted answers. The control system 320 is configured to control the actuator system 330 according to the actuator control data.

[0031] The memory system 360 is a computer or electronic storage system that is configured to store various data and provide access to various data to implement at least the operations and functions disclosed herein. The memory system 360 includes a single device or multiple devices. The memory system 360 includes electrical, electronic, magnetic, optical, semiconductor, electromagnetic, any suitable memory technology or any combination thereof. For example, the memory system 360 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drive, memory card, optical storage device, magnetic storage device, memory module, any suitable type of memory device, or any number and combination thereof. In an example embodiment, with respect to the control system 320 and / or the processing system 340, the memory system 360 is local, remote, or a combination thereof (e.g., partially local and partially remote). For example, the memory system 360 may be configured to include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from the processing system 340 and / or other components of the control system 320.

[0032] The memory system 360 includes the machine learning system 210, which has been implemented via the framework 200 ( Figure 1-2 ) is pre-trained (or trained). The pre-trained or trained machine learning system 210 is configured to be implemented, executed and / or employed via the processing system 340. At this point, the machine learning system 210 is configured to receive and process queries or query tasks as input data. The machine learning system 210 is configured to provide predicted answers in response to the queries or query tasks. At this point, the machine learning system 210 is configured to perform question answering.

[0033] In addition, the memory system 360 includes a query response application system 380. The query-response application system 380 is configured to ensure that queries or query tasks are provided to the machine learning system 210 as input data. In this regard, the processing system 340 is configured to process input from the HMI system 310 via the query response application system 380. If deemed necessary, the query-response application system 380 is configured to generate queries or query tasks when processing input from the HMI system 310. Furthermore, in some cases, the query response application system 380 is configured to generate output data based on predicted answers obtained from the machine learning system 210. In general, the query-response application system 380 enables the machine learning system 210 to operate seamlessly as part of the control system 320 for a desired application.

[0034] In addition, if Figure 3 As shown, system 300 includes other components that facilitate the operation of control system 320 associated with HMI system 310 and actuator system 330. For example, Figure 3 As shown, the memory system 360 is also configured to store other relevant data 390 related to the operation of the system 300. In addition, as shown in FIG. Figure 3 As shown, the control system 320 includes an I / O system 370, which includes one or more I / O devices associated with the system 100. In addition, the control system 320 is configured to provide other functional modules 350, such as any appropriate hardware technology, software technology, or any combination thereof that helps and / or facilitates the operation of the system 300. For example, the other functional modules 350 include an operating system and communication technology that enables the components of the system 300 to communicate with each other as described herein. In addition, the components of the system 300 are not limited to this configuration, but can include any suitable configuration as long as the system 300 performs the functions described herein. For example, the HMI system 310 can be a more complete part of the I / O system 370 and / or the control system 320. Therefore, the system 300 is useful in a variety of applications.

[0035] For example, as a non-limiting example, the system 300 may be a dialogue system for providing customer service and / or troubleshooting assistance. In this case, the system 300 does not further include the actuator system 330. In this regard, for example, the HMI system 310 may include a user interface that operates with the I / O system 370, such as a touch screen device, to receive input from a user. When input data is input into the touch screen device, the processing system 340 is configured to provide a query or query task to the pre-trained or trained machine learning system 210. In response to the query or query task, the processing system 340 is configured to provide a predicted answer via the machine learning system 210. The processing system 340 is configured to directly or indirectly provide a predicted answer as output data, which is received by the user via the touch screen device.

[0036] Figure 4 4 is a diagram of an example of an application of a system 300 for automated personal assistant technology 400 according to an example embodiment. For example, as a non-limiting example, the automated personal assistant technology 400 is a robot 410 that is configured to receive input that may directly or indirectly contain a query or query task. The automated personal assistant technology 400 is configured to process the input and provide the query or query task to the machine learning system 210. In response to the query or query task, the automated personal assistant technology 400 is configured to provide a predicted answer via a pre-trained or trained machine learning system 210. In addition, the automated personal assistant technology 400 is configured to control an appliance such as a washing machine, a stove, a vacuum cleaner, an oven, a microwave oven, a dishwasher, another type of household appliance, any suitable device, or any number and combination thereof. The HMI system 310 includes at least one user interface that operates with an I / O system 370 (e.g., a microphone, a touch screen, a keyboard, a display technology, a gesture technology, a camera, a sensor, or any suitable technology) to obtain input.

[0037] The control system 320 is configured to obtain input (e.g., audio commands, touch screen commands, etc.) from a user 420 via the HMI system 310 and / or the I / O system 370. The control system 320 is configured to process the input. The control system 320 is configured to provide a query or query task based on the input. In addition, the control system 320 is configured to provide a predicted answer in response to the query or query task via the pre-trained or trained machine learning system 210. The control system 320 is configured to generate output data based on the predicted answer. The control system 320 is configured to provide the predicted answer and / or output data to the I / O system 370 and / or the HMI system 310. The control system 320 is configured to generate actuator control data based on the predicted answer and / or output data. In addition, as a non-limiting example, in response to the actuator control data, the control system 320 is configured to control the actuator system 530.

[0038] Figure 5 is a diagram of an example application of a system 300 for mobile machine technology according to an exemplary embodiment. As a non-limiting example, the mobile machine technology includes a vehicle 500 that is at least partially autonomous or fully autonomous. Figure 7 In the embodiment of the present invention, the vehicle 500 includes an HMI system 310 configured to receive input. Based on the input, the control system 320 is configured to provide at least a query or query task to the machine learning system 210. The machine learning system 210 is configured to provide a predicted answer in response to the query or query task. The control system 320 is configured to generate actuator control data, which is based at least on the predicted answer. For example, as a non-limiting example, the actuator system 330 is configured to at least actuate the braking system to stop the vehicle 500 when the actuator control data is received. At this point, the actuator system 330 is configured to include a braking system, a propulsion system, an engine, a transmission system, a steering system, or any number of actuators and combinations thereof of the vehicle 500. The actuator system 330 can be configured to control the vehicle 500 so that the vehicle 500 follows the rules of the road and avoids collisions based at least on the predicted answers provided by the machine learning system 210.

[0039] Figure 6 An embodiment of a system transformation of data points on an embodiment of a wholesale customer data set is disclosed. In order to utilize a large language model (LLM) with tabular data 602, it may be necessary to first convert the records 602 into natural language descriptions. These transformations may be referred to as data descriptions 604. Data descriptions 604 may be performed using a data processor 603. Template matching, commonly used in previous approaches, inserts attribute values ​​into predefined templates. However, this approach may often produce unnatural descriptions that differ from how a human might describe the data. Depending on the data set, manually designing templates can also be challenging. To overcome this, the system may utilize LLM 601 as a more appropriate solution. As shown below in Figure 6 As illustrated, the system can obtain these data descriptions with little effort by zero-shot prompting the LLM 601 with information about the dataset (which is typically available as metadata 601 for a tabular dataset) and a textual representation 606 of the table records (e.g., parsed JSON). For example, the metadata 601 can indicate information about the dataset, such as a description. The textual representation 606 can be text indicating a record or tabular data. Specifically, to ensure that the examples can be used as training data and query input, the system extracts descriptions of the features and concatenates them with a target label using a delimiter tag 608. Interestingly, the descriptions generated by the LLM 601 in this way generally perform better than those from the template.

[0040] In one embodiment, a challenge in this process may be how to effectively encode numeric attributes; naively including numeric values ​​in descriptions can lead to poor performance in subsequent learning tasks. To address this, the system can take a straightforward approach, such as binning all numeric features into percentiles and encoding them descriptively as "low," "medium," and "high" in the text representation 506. Overall, data descriptions can be automatically generated with minimal human engineering.

[0041] The typical approach to performing few-shot learning using large language models (LLMs) involves providing a small number of demonstrations of the intended task as prompts, and then asking the model to generate answers. For example, in the few-shot setting, one can simply present the natural language description above and generate predictions for new examples. However, for tabular data, there may be a large number of data points that do not fit within the context of an LLM. Furthermore, naively increasing the number of examples in context does not always improve performance, and there is no obvious way to manage the weighted distribution over examples, as is required in boosting methods. These observations necessitate alternative approaches to weak learning via LLMs. An example of a cluster sampling algorithm is shown below.

[0042] Algorithm 1 Cluster Sampling:

[0043] 1: Input: X, all training data; y, all training label; r, ratio of classes; p, AdaBoost weights of the current round: s.target number of samples. is the proportion of examples in class I.2:S ←new empty set

[0044] 3: w←new array with same length as Xfilled with-1 is probability of sampling example i.

[0045] 4: for k=1to number of target classes in y do

[0046] 5: E←GPTEMbedding(X[y==k])Erefers to the embeddings of the datadescriptions

[0047] 6: C←AgglomerativeClustering(E).Cjis set of data indices present inthej th cluster.

[0048] 7: c←new empty array same size asC.c[j] will store sampling probability of clusterj

[0049] 8: forj = 1 to len(C) do

[0050] 9:

[0051] 10: end for

[0052] 11: for i = 1 to len(X) do

[0053] 12: w[i]←c[j], such that, i∈C j

[0054] 13: end for

[0055] 14: w←Normalize(Normalize(w)*p) Normalize turns weights to aprobability distribution.

[0056] 15: Sample s*r[c]examples from X using categorical distribution w andappend to S.

[0057] 16: end for

[0058] 17: Return S

[0059] (1: Input: X, all training data; y, all training labels; r, class ratio; p, AdaBoost weight of the current round; s, target number of samples. is the proportion of examples of the first class I

[0060] 2: S ← new empty set

[0061] 3: w← a new array with the same length as X and filled with -1, is the probability of sampling sample j

[0062] 4: Corresponding to the number of target classes from k = 1 to y,

[0063] 5: E←GPTEmbedding(X[y==k]), where E refers to the embedding of the data description

[0064] 6:C←AgglomerativeClustering(E).C j is the data index set existing in the jth cluster

[0065] 7: c←new empty array of the same size as C. c[j] will store the sampling probability of cluster j

[0066] 8: For j = 1 to len(C), perform

[0067] 9:

[0068] 10: End, for

[0069] 11: for i = 1 to len(X), proceed

[0070] 12: w[i]←c[j], so that i∈C j

[0071] 13: End, for

[0072] 14:w←Normalize(Normalize(w)*p) Normalize the round weights to a probability distribution

[0073] 15: Sample s*r[c] from X using the class distribution w and append to S

[0074] 16: End for

[0075] 17: Return to S).

[0076] Figure 7 A process for generating summaries and using the summaries to make predictions on new data is disclosed. Alternatively, the system may propose that generating summaries of sets of examples can serve as a powerful proxy for learning models based on some examples. Specifically, given a set of data descriptions, the system may first perform summarization 707 on the data by invoking LLM 709 (e.g., by concatenating a list of examples in natural language form and appending a prompt "tldr"). The resulting summary 707 can be considered a hypothesis because it provides an explanation for the data. By using the summary as a prompt, LLM 709 in turn uses the hypothesis rather than the original data description to perform inferences (e.g., Figure 7As shown in Figure 2). Since the sampled summaries may sometimes be noisy, the system may generate a fixed number of summaries 711 and pick the one with the smallest validation error rate in step 713. In the case of a tie, the system may choose the summary with the higher training error, e.g., lower generalization gap. Several methods of building such summaries are possible, but simple methods, such as the "tldr" method, may be as effective as more complex alternatives.

[0077] Algorithm 2 Summary Improvement [Compression]

[0078] 1: Input: X, all training data; y, all training label; T: maximum number of rounds; s: size of the sampling subset; r: ratio of classes. denotes the proportion of examples in class k.

[0079] 2: h, ∈, α ← new empty arrays of length T.

[0080] 3: N←len(X); K←number of target classes in y.

[0081] 4: w←new array of length N filled with is the data distribution

[0082] 5: for r = 1 to T do

[0083] 6:(X s ,y s )←ClusterSampling(X, y, r, w, s) sample s training examples from distribution w.

[0084] 7: h[r]←Summary X s ,y s is the weak learner in the current roundr.

[0085] 8: is the weighted error at round r.

[0086] 9: if then

[0087] 10:Goto Step 6.

[0088] 11: end if

[0089] 12: refers to coefficient ofthehypothesis at round r.

[0090] 13: for i = 1 to N do

[0091] 14:

[0092] 15: end for

[0093] 16: w←Normalize(w)

[0094] 17: end for

[0095] 18: Return h, α

[0096] (1: Input: X, all training data; y, all training labels; T: maximum number of rounds; s: sampling subset; r: class ratio; represents the proportion of examples in class k

[0097] 2: h,∈,α←new empty array of length T

[0098] 3: N←len(X); the number of target classes in K←y

[0099] 4: w←filled with a new array of length N, Is the data distribution

[0100] 5: For r = 1 to T

[0101] 6:(X s ,y s )←ClusterSampling(X,y,r,w,s) Sample s is a training sample from distribution w

[0102] 7: h[r]←Summary X s ,y s is the weak learner in the current round r

[0103] 8: The weighted error in round r

[0104] 9: If Then

[0105] 10: Go to step 6

[0106] 11: End if

[0107] 12: Refers to the coefficient of the assumption in wheel r

[0108] 13: For i=1 to N, perform

[0109] 14:

[0110] 15: End for

[0111] 16:w←Normalize(w)

[0112] 17: End for

[0113] 18: Return h, α).

[0114] Figure 7 The process shown can split hypothesis generation 701 in the upper half to describe how weak learning hypotheses (summaries) are generated. The lower half is an illustrative embodiment of a system that performs inference 750. Records in text form can be used for hierarchical cluster sampling. In one embodiment, the system and method can utilize weighted cluster sampling 705, or hierarchical cluster sampling 705. Since the context size of existing LLMs is still limited, the system may not be able to put the entire data set into the context of the summary 707. The summary prompt 707 can also include relevant metadata 703 of the record or text. In addition, the boosting algorithm requires that the system can provide weak learners on weighted samples of the training set, effectively guiding the boosting process to focus on "harder" examples as the boosting process continues. Therefore, the system can only utilize a representative subset of the data set instead of trying to summarize the entire data set. The size of this subset is governed by the maximum context size and the size of the data description. In order to select this representative subset, the system can use weighted hierarchical sampling, which uses a subpopulation defined by the language embedding clustering of each data description. The language embedding can be a sentence representation generated by GPT-3, GPT-4 or other language prediction models. Specifically, the system can use hierarchical agglomerative clustering to identify clusters in the embedding. Such a process is shown in Algorithm 1 above. Such a process can consistently produce weak learners and can improve on random guessing under the distribution of interest (represented by the input p of the algorithm).

[0115] Finally, the system can utilize an adaptive boosting ("AdaBoost") algorithm to produce an ensemble with a collection of these summary-based weak learners. Adaptive boosting can utilize the weights reassigned to each instance, assigning higher weights to incorrectly classified instances. Boosting is used to reduce bias, as well as variance in supervised learning. It works by growing learners sequentially. Except for the first learner, each subsequent learner is grown from the previously grown learner. Thus, weak learners are transformed into strong learners. One idea of ​​AdaBoost is to adapt a series of weak learners to repeatedly modified versions of the data. The algorithm can be executed in T rounds, where the weights of the training data points are adjusted based on the training error.

[0116] Given a new data point or query 751, the predictions from all rounds of classifiers can then be combined by weighted majority voting to produce a final prediction 777 or answer 777. The system uses the error on the holdout validation set to determine the number of rounds t. Algorithm 2 gives a condensed version of this process. In this algorithm, the summary method summarizes the examples in the prompt via the discussed process related to weak learning via summaries. Each summary can be viewed as a hypothesis that can classify new data.

[0117] However, unlike the summary process where the system can resample multiple times to find the best learner, the boosting process returns immediately when a summary is found whose error rate is better than random guessing (or chance). The system and method can utilize cluster sampling to subsample mini-batches of examples that fit within the maximum context length of the LLM.

[0118] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of the present disclosure may be required.

[0119] As used herein, "a", "an", and "the" refer to the singular and the plural, unless the context clearly indicates otherwise. By way of example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor that are collectively programmed to perform each of the various functions.

[0120] Although exemplary embodiments are described above, this does not mean that these embodiments describe all possible forms included in the claims. The words used in the specification are descriptive words, not restrictive, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As mentioned above, the features of the various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations with respect to one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, with respect to one or more features, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A computer-implemented method for natural language processing, comprising: receiving tabular data associated with one or more records; converting tabular data into a textual representation indicative of the tabular data; generating metadata associated with the textual representation of the tabular data, wherein the metadata indicates a description of the tabular data; For one or more iterations, in response to a zero-shot prompt utilizing a large language model (LLM) and metadata and a textual representation of the tabular data, outputting one or more natural language data descriptions indicative of the tabular data, wherein the LLM comprises a neural network having a plurality of parameters; For one or more iterations, outputting one or more summaries using the LLM, wherein the one or more summaries include less text than the one or more natural language data descriptions; For one or more iterations, in response to a single summary having a minimum validation rate, selecting a single summary of the one or more summaries; receiving a query associated with the tabular data; For one or more iterations, outputting one or more predictions associated with the query using the LLM for the single summary and the query; and In response to one or more predictions generated from the one or more iterations satisfying a convergence threshold, a final prediction associated with the query is output, wherein the final prediction is selected in response to a weighted majority vote of all of the one or more predictions generated from the one or more iterations.

2. The method of claim 1, wherein the LLM does not utilize fine-tuning or building a new language model to output a final prediction. The method of claim 1 , wherein the plurality of parameters exceeds one billion. The method of claim 1 , wherein the LLM is GPT-4. The method of claim 1 , wherein the text representation is a parsed JSON file.

6. The method of claim 1, wherein any numerical features associated with the tabular data are encoded as percentiles.

7. The method of claim 1, wherein outputting one or more summaries using the LLM comprises appending hints to the one or more natural language data descriptions.

8. The method of claim 1, wherein the one or more predictors are weak learners.

9. The method of claim 1, wherein the method comprises selecting a single summary using weighted stratified sampling.

10. A system comprising: an input interface to the system, wherein the input interface is configured to receive data associated with the system; and one or more processors in communication with the input interface, the one or more processors being programmed to: receiving tabular data associated with one or more records; converting tabular data into a textual representation indicative of the tabular data; generating metadata associated with the textual representation of the tabular data, wherein the metadata indicates a description of the tabular data; outputting one or more natural language data descriptions indicative of the tabular data in response to a zero-shot prompt utilizing a large language model (LLM) and metadata and a textual representation of the tabular data, wherein the LLM comprises a neural network having a plurality of parameters; outputting one or more summaries using the LLM and the one or more natural language data descriptions, wherein the one or more summaries include less text than the one or more natural language data descriptions; For one or more iterations, in response to a single summary having a minimum validation rate, selecting a single summary of the one or more summaries; receiving a query associated with the tabular data; For one or more iterations, outputting one or more predictions associated with the query using the LLM for the single summary and the query; and In response to one or more predictions generated from the one or more iterations of outputting the one or more predictions satisfying a convergence threshold, a final prediction associated with the query is output.

11. The system of claim 10, wherein the processor is further programmed to output a final prediction in response to a weighted majority vote of all of the one or more predictions generated from one or more iterations.

12. The system of claim 10, wherein the numerical features associated with the tabular data are encoded as percentiles.

13. The system of claim 10, wherein the one or more processors are programmed to utilize the LLM and append hints to the one or more natural language data descriptions to output one or more summaries.

14. The system of claim 10, wherein the one or more processors are collectively programmed to perform steps.

15. The system of claim 10, wherein the one or more processors comprises a single processor programmed to perform steps.

16. The system of claim 10, wherein the one or more processors are programmed to select a representative subset of the one or more summaries using weighted stratified sampling.

17. The system of claim 10, wherein the one or more processors are programmed to select a representative subset of the one or more summaries using weighted stratified sampling.

18. The system of claim 10, wherein the one or more processors are programmed to select a representative subset of the one or more summaries from a fixed number of iterations.

19. The system of claim 10, wherein the processor is programmed to, for one or more iterations, output the one or more natural language data descriptions and output the one or more summaries.

20. A computer-implemented method for natural language processing, comprising: receiving tabular data associated with one or more records; converting tabular data into a textual representation indicative of the tabular data; generating metadata associated with the textual representation of the tabular data, wherein the metadata indicates a description of the tabular data; In response to a zero-shot prompt utilizing a large language model (LLM) and metadata and a textual representation of the tabular data, outputting one or more natural language data descriptions indicative of the tabular data; Outputting one or more summaries using the LLM and appending prompts to the one or more natural language data descriptions, wherein the one or more summaries include less text than the one or more natural language data descriptions; For one or more iterations, in response to a single summary having a minimum validation rate, selecting a single summary of the one or more summaries; Create subsets of a single profile; receiving a query associated with the tabular data; For one or more iterations, outputting one or more predictions associated with the query using one of the single summaries of the subset of the single summaries and the LLM of the query; Create one or more subsets of forecasts; and In response to the subset of the one or more predictions satisfying a convergence threshold, a final prediction associated with the query is output, wherein the final prediction is selected in response to a weighted majority vote of the subset of the one or more predictions.

Images