System and method for utilizing weak learners on large language models

Abstract

To provide a computer-implemented method for natural language processing.SOLUTION: The method includes: converting tabular data to a text representation; generating metadata associated with the text representation of the tabular data; outputting one or more natural language data descriptions indicating the tabular data in response to utilizing a Large Language Model (LLM) and zero-shot prompting of the metadata and text representation of the tabular data; outputting one or more summaries utilizing the LLM and appending a prompt on the one or more natural language data descriptions; selecting a single summary of the one or more summaries in response to the single summary having the smallest validation rate; receiving a query associated with the tabular data; and outputting one or more predictions associated with the query, and in response to meeting a convergence threshold with the one or more predictions generated from one or more iterations, outputting a final prediction associated with the query.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the creation of systems for responding to text queries or human voice input. [Background technology]

[0002] background A weak learner can refer to a classifier that can achieve better performance than random chance by some given margin for an arbitrarily specified distribution over the training data. One of the early breakthroughs in machine learning established that weak learning is sufficient to make arbitrary strong classifications through ensemble procedures. This led to the development of boosting algorithms, a class of approaches that continue to perform extremely well even on tabular datasets that lack the regularity of the input space, especially for vision or language tasks.

[0003] In a seemingly separate stream of research, large-scale language models (LLMs) based on modern transducer architectures have become dominant in many natural language domains. While such models are often fine-tuned based on data from new downstream tasks, it has also been shown in recent years to show strong performance as zero-shot or few-shot learning by simply prompting the model with a portion of the context string. Summary of the Invention [Means for solving the problem]

[0004] overview According to a first embodiment, a computer-implemented method for natural language processing includes receiving tabular data associated with one or more records, converting the tabular data into a textual representation indicative of the tabular data, and generating metadata indicative of a description of the tabular data associated with the textual representation of the tabular data. The method includes outputting, for one or more iterations, one or more natural language data descriptions indicative of the tabular data in response to utilizing a large-scale language model (LLM) and zero-shot prompting of the metadata and the textual representation of the tabular data, where the LLM includes a neural network having a plurality of parameters. The method further includes: for the one or more iterations, outputting one or more summaries utilizing the LLM and adding prompts to the one or more natural language data descriptions, where the one or more summaries include less text than the one or more natural language data descriptions; for the one or more iterations, selecting a single summary of the one or more summaries in response to the single summary having a minimum validation rate; receiving a query associated with the tabular data; for the one or more iterations, outputting one or more predictions associated with the query utilizing the LLM for the single summary and the query; and in response to a convergence threshold being met by the one or more predictions generated from the one or more iterations, outputting a final prediction associated with the query, where the final prediction is selected in response to a weighted majority vote of all of the one or more predictions generated from the one or more iterations.

[0005] A second embodiment discloses a system comprising an input interface to the system configured to receive data associated with the system, and one or more processors in communication with the input interface, the one or more processors being programmed to receive tabular data associated with one or more records, convert the tabular data into a text representation indicative of the tabular data, and generate metadata indicative of a description of the tabular data associated with the text representation of the tabular data. The processor is further programmed to: output one or more natural language data descriptions indicative of the tabular data in response to utilizing a large-scale language model (LLM) and zero-shot prompting of the metadata and the textual representation of the tabular data, where the LLM includes a neural network having a plurality of parameters; output one or more summaries utilizing the LLM and the one or more natural language data descriptions, where the one or more summaries include less text than the one or more natural language data descriptions; select a single summary of the one or more summaries in response to a single summary having a minimum validation rate, for one or more iterations; receive a query associated with the tabular data; output one or more predictions associated with the query, for one or more iterations, utilizing the LLM for the single summary and the query; and output a final prediction associated with the query in response to a convergence threshold being met by the one or more predictions generated from the one or more iterations outputting the one or more predictions.

[0006] A third embodiment is a computer-implemented method for natural language processing comprising: receiving tabular data associated with one or more records; converting the tabular data into a textual representation indicative of the tabular data; generating metadata indicative of a description of the tabular data associated with the textual representation of the tabular data; outputting one or more natural language data descriptions indicative of the tabular data in response to utilizing a large scale language model (LLM) and zero-shot prompting of the metadata and the textual representation of the tabular data; outputting one or more summaries and adding prompts to the one or more natural language data descriptions utilizing the LLM, wherein the one or more summaries include less text than the one or more natural language data descriptions; or for multiple iterations, in response to a single summary having a minimum validation rate, selecting a single summary of the one or more summaries; creating a subset of the single summaries; receiving a query associated with the tabular data; for one or more iterations, utilizing an LLM for one of the single summaries of the subset of single summaries and the query to output one or more predictions associated with the query; creating the subset of one or more predictions; and in response to a convergence threshold being met by the subset of the one or more predictions, outputting a final prediction associated with the query, wherein the final prediction is selected in response to a weighted majority vote of the subset of the one or more predictions. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a system having a framework for query tasks, according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates a conceptual diagram of an example framework for query tasks for a machine learning system, according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 2 illustrates an example of a control system using the machine learning system of FIG. 1 in accordance with an exemplary embodiment of the present disclosure. [Figure 4]FIG. 4 illustrates an example of the control system of FIG. 3 for robotic and / or automated personal assistant technology, according to an exemplary embodiment of the present disclosure; [Diagram 5] 4 illustrates an example of the control system of FIG. 3 for mobile device technology, according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates one embodiment of a system transformation for data points. [Figure 7] FIG. 1 illustrates one embodiment of a process for generating and using summaries to make predictions on new data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description A central concept in practical and theoretical machine learning is the notion of weak learners, classifiers that achieve better performance than random (based on any given distribution over the data), even by a small margin. Such weak learners form the practical basis of standard machine learning methods such as boosting. In the embodiments disclosed below, the system and method show that a large-scale language model based on prompts can effectively operate like the weak learners described above. Specifically, the system and method can show the use of a large-scale language model (LLM) as a weak learner in a boosting algorithm applied to tabular data. By forming textual descriptions (suitably sampled according to the distribution of interest) of tabular data samples, the system can show that the LLM can form a summary of the samples that serves as a template for classification and achieves the goal of acting as a weak learner for the task. The system and method can incorporate these models into a boosting approach, which in some settings leverages the knowledge in the LLM to achieve superiority over traditional tree-based boosting. The models outperform both few-shot learning and potentially multiple complex fine-tuning procedures, especially for tasks involving a small number of data points. These results demonstrate the potential for prompt-based LLM to function as a component of larger machine learning pipelines, not just as a few-shot learner on its own.

[0009] The following disclosure is an example of an LLM functioning as a weak learner in a boosting framework, particularly for tabular data, and shows that by appropriately converting the tabular data to a textual format and requesting the LLM to summarize a carefully selected set of examples from the data, the system can generate a summary of the examples that can serve as templates (e.g., prompts) for a tabular data classifier, which typically achieves the weak learning goal. This allows such a collection of weak learners generated by the LLM to be appropriately incorporated into a boosting framework.

[0010] The system can show that the resulting approach performs well in many settings, outperforming zero-shot and few-shot classifications as well as "single-shot" summaries produced by LLMs. All of this is accomplished by a system that does not require any retraining or fine-tuning of the LLM itself, but only by prompting. Furthermore, in certain domains (especially those with very few examples where it is particularly important to leverage the prior knowledge embedded in the LLM), the system can show that the approach can outperform traditional tree-based boosting and LLM-based fine-tuning methods, and that its performance can improve as the capabilities of the LLM improve. Overall, the system can highlight the possibility of incorporating LLMs as subroutines in larger machine learning systems.

[0011] The following systems and methods can utilize LLM to generate weak learners, which can then be used within a boosting framework. This can be referred to as LLM summary boosting, since the core learning process can include in an embodiment a process that can use a language model to create summaries of (specifically selected) samples from a dataset. These summaries can serve as prompts from which the system can make predictions for new examples. Finally, the system can use boosting to build an ensemble of such summaries that provide an overall prediction for new data points.

[0012] FIG. 1 illustrates an example of a system 100 having a neuro-symbolic framework 200 for a query task, according to an exemplary embodiment. The system 100 is configured to pre-train (or train) a machine learning system 210 via the neuro-symbolic framework 200. Moreover, the system 100 is an example of a system configured to run on the machine learning system 210 via the framework 200. After undergoing pre-training (or after undergoing both pre-training and zero-shot testing), the system 100 can be configured to utilize the machine learning system 210 for use. Alternatively, the system 100 can be configured such that the pre-trained (or pre-trained and zero-shot tested) machine learning system 210 can be made available and / or deployed for use in other systems (e.g., system 300 of FIG. 3).

[0013] System 100 includes at least one processing system 140. Processing system 140 includes at least an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any suitable processing technology, or any multiple thereof and any combination thereof. Processing system 140 is operable to provide the functionality of framework 200 and machine learning system 210 as described herein.

[0014] System 100 includes at least one memory system 120 operably connected to processing system 140. In an exemplary embodiment, memory system 120 includes at least one non-transitory computer-readable medium configured to store and provide access to various data to enable at least processing system 140 to perform operations and functions related to framework 200 and corresponding machine learning system 210, as disclosed herein. In an exemplary embodiment, memory system 120 includes a single computer-readable storage device or multiple computer-readable storage devices. Memory system 120 may include electrical storage technologies, electronic storage technologies, magnetic storage technologies, optical storage technologies, semiconductor storage technologies, electromagnetic storage technologies, or any suitable storage technologies operable with system 100. For example, in an exemplary embodiment, memory system 120 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any multiple thereof and any combination thereof. Memory system 120 may be local or remote, or a combination thereof (e.g., partially local and partially remote), relative to processing system 140 and / or other components of system 100. For example, memory system 120 may include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from processing system 140 and / or other components of system 100.

[0015] The memory system 120 includes at least a framework 200, a machine learning system 210, machine learning data 220, and other associated data 230 stored therein and accessible therefrom. The framework 200 includes computer readable data configured to generate at least one training set having an appropriate number of query tasks for the machine learning system 210 when executed by the processing system 140. Additionally, the framework 200 includes computer readable data configured to implement a zero-shot testing process (or a zero-shot evaluation process) that evaluates the pre-trained (or trained) machine learning system 210 on various common sense tasks when executed by the processing system 140. The computer readable data may include instructions, code, routines, various associated data, any software techniques, or any combination and plurality of these.

[0016] In an exemplary embodiment, the machine learning system 210 includes at least one machine learning model. More specifically, the machine learning system 210 includes at least one language model. For example, the machine learning system 210 includes a large language model (LLM) or any number of language models and combinations thereof.

[0017] In an exemplary embodiment, the machine learning data 220 includes various data that the framework 200 uses to train, test, and develop the machine learning system 210. For example, the machine learning data 220 includes a global knowledge graph 220A. The global knowledge graph 220A is generated by combining various knowledge graphs 220B. The machine learning data 220 may also include one or more knowledge bases associated with one or more knowledge graphs 220B. The machine learning data 220 also includes a collection of common sense task datasets 220C that encompass a diverse set of tasks. In addition, the machine learning data 220 may include various annotations, various loss data, various parameter data, and any related data that enables the neurosymbol framework 200 and the machine learning system 210 to perform the functions described herein while meeting predetermined performance criteria. In contrast, the other related data 230 provides various data (e.g., operating system, etc.) that enables the system 100 to perform the functions described herein.

[0018] In an exemplary embodiment, as shown in FIG. 1, system 100 is configured to include at least one human-machine interface (HMI) system 110. HMI system 110 includes at least one user interface, at least one HMI device, or any number of combinations thereof. For example, HMI system 110 may include a visual user interface, an auditory user interface, a tactile user interface, any suitable user interface, or any number of and any combinations thereof. HMI system 110 is operable to communicate with I / O system 130. HMI system 110 is also operable to communicate with one or more other components of system 100 (e.g., processing system 140, memory system 120, etc.). More specifically, for example, processing system 140 is configured to obtain or extract queries or query tasks directly or indirectly from HMI system 110, memory system 120, and / or I / O system 130. In response to receiving the query or query task, the processing system 140 is configured to form a predicted answer to the query or query task via the machine learning system 210.

[0019] Additionally, the system 100 includes other components that contribute to training and / or execution of the framework 200 and the machine learning system 210. For example, as shown in FIG. 1, the memory system 120 is also configured to store other relevant data 230 regarding the operation of the system 100 associated with one or more components (e.g., the sensor system 110, the I / O system 130, and other functional modules 150). Additionally, the I / O system 130 may include an I / O interface and may include one or more devices (e.g., a microphone, a keyboard device, a touch display device, a microphone, a mouse, a speaker device, etc.). The system 100 also includes other functional modules 150, e.g., any suitable hardware technology, software technology, or combination thereof, that support or contribute to the functionality of the system 100. For example, the other functional modules 150 include communication technology that allows the components of the system 100 to communicate with each other as described herein. Thus, using at least the components shown in FIG. 1, system 100 executes a framework 200 to pre-train (or train) a machine learning system 210 so that it performs well across a variety of query tasks (e.g., question answering tasks) in a zero-shot setting or when deployed / used for use in an application.

[0020] FIG. 2 is a conceptual diagram of an embodiment of a neuro-symbolic framework 200 for a machine learning system 210. In an exemplary embodiment, the framework 200 includes at least a query task generator 200A. The query task generator 200A is configurable to obtain a data structure (e.g., triples) from a global knowledge graph 220A. As shown in FIG. 2, the global knowledge graph 220A includes multiple separate knowledge graphs 220B, where the total number of knowledge graphs 220B is represented by "N" in FIG. 2. In this regard, "N" represents an integer of at least 3 or more. The query task generator 200A is configured to generate a query task based on the data structure of the global knowledge graph 220A. The query task generator 200A is configured to create a training set including an appropriate number of query tasks. The query task generator 200A is configured to pre-train or train the machine learning system 210 using at least one training set. The query task generator 200A is also configured to calculate at least one score of the machine learning system 210 and fine-tune the machine learning system 210, for example, based on the score data, the loss data, and / or any other relevant data. The query task generator 200A ensures that the machine learning system 210 is pre-trained or trained to perform well across a variety of common sense tasks when tested in a zero-shot setting and / or when deployed / used for use in an application.

[0021] Further, the framework 200 is configured to include a zero-shot evaluator 200B. The zero-shot evaluator 200B is configured to perform zero-shot testing on the machine learning system 210. As shown in FIG. 2, the zero-shot evaluator 200B is configured to perform zero-shot testing during a post-training phase. The post-training phase refers to any phase that occurs after pre-training (or training) of the machine learning system 210 using at least one training set generated by the query task generator 200A. The zero-shot evaluator 200B is configured to test the machine learning system 210 in a zero-shot manner using the common sense task dataset 220C. In this regard, the machine learning system 210 is configured to process each common sense task dataset 220C without observing the common sense task dataset 220C in advance. The zero-shot evaluator 200B is configured to obtain a set of common sense task datasets 220C and apply each common sense task dataset 220C to the machine learning system 210. The collection of common sense task datasets 220C includes a plurality of common sense task datasets 220C, where the total number of common sense task datasets 220C is represented by "M" in FIG. 2. In this regard, "M" represents an integer of at least 3 or more. Each common sense task dataset 220C differs from the other common sense task datasets 220C of the collection, for example, in terms of the format of the query task and / or the type of knowledge associated with the query task. By using the various common sense task datasets 220C, the zero-shot evaluator 200B is advantageously configured to demonstrate the effectiveness of the pre-training (or training) of the machine learning system 210 based on the training set generated by the query task generator 200A. In this regard, the zero-shot evaluator 200B is configured to provide a robust measure of the inference capability of the machine learning system 210.The zero-shot evaluator 200B is also configured to evaluate the machine learning system 210 along with the impact of pre-training across various common sense task datasets 220C.

[0022] As described above, the set of common sense task datasets 220C includes various common sense task datasets 220C. Each common sense task dataset 220C is different from the training set generated by the query task generator 200A. The set of common sense task datasets 220C is a dataset that the machine learning system 210 did not observe at all during its pre-training phase or training phase. In this regard, the set of common sense task datasets 220C is selected to include a diverse set of tasks, for example, at least in terms of format (e.g., question answering, pronoun resolution, natural language reasoning, etc.) or in terms of knowledge type (e.g., social knowledge, physical knowledge, etc.) or in terms of both format and knowledge type. For example, there may be a task dataset 220C that includes a natural reasoning task, where a story beginning and end are given and the task is to select a more plausible hypothesis from a set of response options. Furthermore, a task dataset 220C that includes a wide range of common sense aspects may be provided, where the task is to answer a question by selecting one of five response options. As another example, there may be a task dataset 220C that focuses on physical reasoning, where the task is to select a more plausible response option from two possible continuations. There may also be a task dataset 220C that focuses on social interaction-based reasoning, where the task includes some context, a question, and a set of response options. As yet another example, there may be a task dataset 220C that includes pronoun resolution, where the task includes some context, an emphasis pronoun, and a response option that is provided as a possible reference. It should be noted that the set of common sense task datasets 220C is not limited to the common sense task datasets 220C described above, but may include any task dataset 220C that is suitable for performing zero-shot testing on the machine learning system 210.

[0023] 3 illustrates a system 300 configured to include at least a pre-trained (or trained) machine learning system 210. In this regard, the system 300 includes at least an HMI system 310, a control system 320, and an actuator system 330. The system 300 is configured such that the control system 320 controls the actuator system 330 based on input received from the HMI system 310. More specifically, the HMI system 310 includes one or more user interfaces and / or devices that communicate with one or more I / O devices of an I / O system 370. Once the input is obtained, the HMI system 310 is operable to communicate with the control system 320 via the input / output (I / O) system 370 and / or other functional modules 350 including communication technologies.

[0024] The control system 320 is configured to obtain inputs from the HMI system 310. Once the inputs are received, the control system 320 is operable to process the inputs via the processing system 340. In this regard, the processing system 340 includes at least one processor. For example, the processing system 340 includes an electronic processor, a central processing unit (CPU), a graphic processing unit (GPU), a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a processing circuit, any suitable processing technology, or any combination thereof. Once the inputs received from at least the HMI system 310 are processed, the processing system 340 is operable to provide a query or query task to the machine learning system 210 based on the inputs. The processing system 340 is also configured to generate a predicted answer via the machine learning system 210. The processing system 340 is configured to generate output data based on the predicted answer. The processing system 340 is configured to provide the output data and / or the predicted answer to a user via the I / O system 370 and / or the HMI system 310. Further, the processing system 340 is operable to generate actuator control data based on the output data and / or the predicted response. The control system 320 is configured to control the actuator system 330 according to the actuator control data.

[0025] Memory system 360 is a computer or electronic storage system configured to store and provide access to various data to enable at least the operations and functions disclosed herein. Memory system 360 may include a single device or multiple devices. Memory system 360 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, any suitable memory technology, or any combination thereof. For example, memory system 360 may include random access memory (RAM), read only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any multiple and any combination thereof. In an exemplary embodiment, memory system 360 is local or remote, or a combination thereof, with respect to control system 320 and / or processing system 340 (e.g., partly local and partly remote). For example, memory system 360 can be configured to include at least a cloud-based storage system (e.g., a cloud-based database system) that is located remotely from other components of processing system 340 and / or control system 320.

[0026] The memory system 360 includes a machine learning system 210 that has been pre-trained (or trained) via the framework 200 (FIGS. 1-2). The pre-trained or trained machine learning system 210 herein is configured to be implemented, executed, and / or used via the processing system 340. In this regard, the machine learning system 210 is configured to receive and process a query or query task as input data. The machine learning system 210 is configured to form a predicted answer in response to the query or query task. In this regard, the machine learning system 210 is configured to perform question answering.

[0027] Further, the memory system 360 includes a query response application system 380. The query response application system 380 is configured to ensure that the machine learning system 210 is provided with queries or query tasks as input data. In this regard, the processing system 340 is configured to process inputs from the HMI system 310 via the query response application system 380. If deemed necessary, the query response application system 380 is configured to generate queries or query tasks after the inputs from the HMI system 310 are processed. Furthermore, in some examples, the query response application system 380 is configured to generate output data based on predicted answers obtained from the machine learning system 210. Overall, the query response application system 380 enables the machine learning system 210 to seamlessly operate as part of the control system 320 for a desired application.

[0028] 3, the system 300 includes other components that contribute to the operation of the control system 320 associated with the HMI system 310 and the actuator system 330. For example, as shown in FIG. 3, the memory system 360 is further configured to store other relevant data 390 associated with the operation of the system 300. Also, as shown in FIG. 3, the control system 320 includes an I / O system 370 that includes one or more I / O devices associated with the system 100. Also, the control system 320 is configured to provide other functional modules 350, such as any suitable hardware technology, software technology, or any combination thereof, that support and / or contribute to the functionality of the system 300. For example, the other functional modules 350 include an operating system and communication technology that allows the components of the system 300 to communicate with each other as described herein. Also, the components of the system 300 are not limited to such configurations and may include any suitable configurations as long as the system 300 performs the functions described herein. For example, HMI system 310 may be a more integrated part of I / O system 370 and / or control system 320. Thus, system 300 is useful in a variety of applications.

[0029] For example, and by way of non-limiting example, the system 300 may be a dialogue system used to provide customer service and / or troubleshooting assistance. In this case, the system 300 further does not include the actuator system 330. In this regard, for example, the HMI system 310 may include a user interface, such as a touch screen device, that operates with the I / O system 370 to receive input from a user. When input data is entered into the touch screen device, the processing system 340 is configured to provide a query or query task to the pre-trained or trained machine learning system 210. In response to the query or query task, the processing system 340 is configured to form a predicted answer via the machine learning system 210. The processing system 340 is configured to provide the predicted answer directly or indirectly as output data received by the user via the touch screen device.

[0030] FIG. 4 illustrates an example of an application of the system 300 for an automated personal assistant technology 400, according to an exemplary embodiment. For example, as a non-limiting example, the automated personal assistant technology 400 is a robot 410 configured to receive input that may directly or indirectly include a query or query task. The automated personal assistant technology 400 is configured to process the input and provide the query or query task to the machine learning system 210. In response to the query or query task, the automated personal assistant technology 400 is configured to provide a predicted answer via the pre-trained or trained machine learning system 210. Furthermore, the automated personal assistant technology 400 is configured to control a home appliance device, such as a washing machine, a stove, a vacuum cleaner, an oven, a microwave oven, a dishwasher, other types of household appliances, any suitable device, or any multiple of these and any combination thereof. The HMI system 310 includes at least one user interface that operates with the I / O system 370 (e.g., a microphone, a touch screen, a keyboard, a display technology, a gesture technology, a camera, a sensor, or any suitable technology) to obtain input.

[0031] The control system 320 is configured to obtain input (e.g., audio commands, touch screen commands, etc.) from the user 420 via the HMI system 310 and / or the I / O system 370. The control system 320 is configured to process the input. The control system 320 is configured to form a query or query task based on the input. Furthermore, the control system 320 is configured to form a predicted answer responsive to the query or query task via the pre-trained or trained machine learning system 210. The control system 320 is configured to generate output data based on the predicted answer. The control system 320 is configured to provide the predicted answer and / or the output data to the I / O system 370 and / or the HMI system 310. The control system 320 is configured to generate actuator control data based on the predicted answer and / or the output data. Also, by way of non-limiting example, the control system 320 is configured to control the actuator system 330 in response to the actuator control data.

[0032] FIG. 5 illustrates an example of an application of the system 300 with respect to mobile machine technology, according to an exemplary embodiment. As a non-limiting example, the mobile machine technology includes a vehicle 500 that is at least partially autonomous or fully autonomous. In FIG. 5, the vehicle 500 includes an HMI system 310 configured to receive an input. Based on the input, the control system 320 is configured to provide at least one query or query task to the machine learning system 210. The machine learning system 210 is configured to form a predicted answer in response to the query or query task. The control system 320 is configured to generate actuator control data based at least on the predicted answer. For example, as a non-limiting example, the actuator system 330 is configured to operate at least a braking system to stop the vehicle 500 upon receiving the actuator control data. In this regard, the actuator system 330 is configured to include actuators of the braking system, propulsion system, engine, drive train, steering system, or any number and combination of the vehicle of the vehicle. The actuator system 330 can be configured to control the vehicle 500 such that the vehicle 500 obeys road traffic rules and avoids collisions based on at least the predicted answers formed by the machine learning system 210.

[0033] FIG. 6 discloses an embodiment of a system transformation for data points in an embodiment of a wholesale customer dataset. To utilize a large-scale language model (LLM) with tabular data 602, it may be necessary to first transform the records 602 into natural language descriptions. This may be referred to as data descriptions 604. The data descriptions 604 may be performed using a data processor 603. In template matching, which is commonly used in prior approaches, attribute values ​​are inserted into predefined templates. However, such approaches often produce unnatural descriptions that are different from how humans would describe data. For some datasets, manual template design may also be difficult. To overcome this, the system may utilize an LLM 601 as a more suitable solution. As shown below with reference to FIG. 6, the system may obtain the data descriptions with less effort by zero-shot prompting the LLM 601 with information about the dataset and a textual representation 606 of the tabular records (e.g., parsed JSON) (which is commonly available as metadata 601 for the tabular dataset). For example, the metadata 601 may indicate information about the dataset, such as a description. The text representation 606 can be text representing records or tabular data. Specifically, to ensure that examples serve as both training data and query inputs, the system extracts feature descriptions and concatenates them with target labels using separator tokens at 608. Interestingly, the descriptions generated by the LLM 601 in this manner often perform better than descriptions from templates.

[0034] In one embodiment, a challenge in the process is how to effectively encode numerical attributes, as simply including numerical values ​​in the description may degrade performance in the subsequent learning task. To address this, the system can take a straightforward approach, for example binning all numerical features into percentiles and encoding these with the description as "low", "medium" and "high" in the text representation 506. Overall, data descriptions can be generated automatically with minimal manual engineering.

[0035] A typical method for performing short-shot learning with large-scale language models (LLMs) involves providing a small number of demonstrations of the target task as prompts and then requesting the model to generate an answer. For example, in a short-shot setting, simply presenting the natural language description above can generate predictions for new examples. However, for tabular data, there may be many data points that do not fit within the LLM context. Moreover, simply increasing the number of examples in the context does not necessarily improve performance, and there was no clear scheme to manage the weight distribution across examples, as required by boosting methods. These observations call for an alternative approach to weak learning via LLMs. An example of a cluster sampling algorithm is shown below.

[0036]

number

[0037] In FIG. 7, a process of generating summaries and using the summaries to form predictions on new data is disclosed. Alternatively, the system can propose that generating summaries of a collection of examples can serve as a strong proxy for learning a model based on several examples. Specifically, given a set of data descriptions, the system can first perform summaries 707 on the data by invoking the LLM 709 (e.g., by concatenating a list of examples in the form of natural language and adding the prompt "tldr"). The resulting summary 707 provides a data description and can be considered as a hypothesis. By using the summary as a prompt, the LLM 709 makes inferences using the hypothesis instead of the raw data description (as shown in FIG. 7). Because sampled summaries can sometimes have a lot of noise, the system can generate a fixed number of summaries 711 and select the summary 711 with the smallest validation error rate in step 713. During matching, the system can select the summary with a higher training error, e.g., a lower generalization gap. Several ways of constructing such summaries are possible, but simple approaches such as the "tldr" approach can work as more sophisticated alternatives.

[0038]

number

[0039] The process shown in FIG. 7 can be divided, with the top half showing hypothesis generation 701 to explain how weak learning hypotheses (summaries) are generated. The bottom half is an exemplary embodiment of a system performing inference 750. Records as text can be used to perform stratified cluster sampling. The system and method can utilize weighted cluster sampling 705 in one embodiment, or can utilize stratified cluster sampling 705. Since the context size of existing LLMs is still limited, the system may not be able to fit the entire data set into the context for summarization 707. The summarization prompt 707 may also include associated metadata 703 of the records or text. Note that boosting algorithms may require that for weighted samples of the training set, the system can provide a weak learner that effectively guides the boosting process to focus on "harder" examples as the boosting process continues. Thus, instead of attempting to summarize the entire data set, the system can utilize only a representative subset of the data set. The size of the subset is governed by the maximum context size and the size of the data description. To select such a representative subset, the system can use weighted stratified sampling, using subpopulations defined by clusters of the language embeddings of each data description. The language embeddings can be sentence representations generated by GPT-3, GPT-4, or other language prediction models. In particular, the system can use hierarchical agglomerative clustering to identify clusters in the embeddings. The process is shown in Algorithm 1 above. The process can consistently generate weak learners that can improve random estimation under the distribution of interest (denoted by the input p to the algorithm).

[0040] Finally, the system can utilize the Adaptive Boosting ("AdaBoost") algorithm to generate an ensemble with the above collection of summary-based weak learners. Adaptive Boosting can utilize the weights assigned to each instance again, but now a larger weight is assigned to the misclassified instances. Boosting is used to reduce bias and variance in supervised learning. It works based on a method of sequentially growing learners. Except for the first learner, each subsequent learner is grown from the previous grown learner. Thus, the weak learner is transformed into a strong learner. One of the ideas of AdaBoost is to fit a sequence of weak learners to iteratively modified versions of the data. The algorithm can be run for T rounds, where the weights of the training data points are adjusted based on the training error.

[0041] Given a new data point or query 751, the classifier predictions from all rounds are combined by weighted voting to generate a final prediction 777 or answer 777. The system uses the error on the holdout validation set to determine the number of rounds T. A compact version of this process is presented in Algorithm 2. In this algorithm, a summary method summarizes the examples in the prompt by the process discussed in connection with Weak Learning via Summarization. Each summary can be treated as a hypothesis that may classify the new data.

[0042] However, unlike the summarization process, where the system may resample multiple times to find the best learner, the boosting process returns immediately if a summary is found that has an error rate better than random estimation (or chance).The system and method can utilize cluster sampling to subsample mini-batches of examples that fit the maximum context length of the LLM.

[0043] The embodiments of the present disclosure have been described herein. However, it should be understood that the disclosed embodiments are merely examples, and that other embodiments may take various alternative forms. The figures are not necessarily to scale, and some features may be exaggerated or minimized to show the details of certain components. Thus, the specific structural and functional details disclosed herein should not be interpreted as limiting, but rather as a representative basis to teach one of ordinary skill in the art to variously utilize the embodiments. As will be understood by one of ordinary skill in the art, various features illustrated and described with reference to any one of the figures may be combined with features shown in one or more other figures to form an embodiment not explicitly illustrated or described. The combination of the illustrated features provides a representative embodiment for a typical application. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for a particular application or implementation.

[0044] As used herein, "a", "an", and "the" refer to both the singular and plural of the referent, unless the context clearly indicates otherwise. By way of example, a "processor" programmed to perform various functions may refer to one processor programmed to perform each function, or may collectively refer to two or more processors programmed to perform each of the various functions.

[0045] Although exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms encompassed by the claims. It is to be understood that the terms used herein are descriptive rather than limiting, and that various modifications are possible without departing from the spirit and scope of the present disclosure. As described above, features of various embodiments may be combined to form further embodiments of the present invention not expressly described or shown. Although various embodiments have been described as offering advantages or being more preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those skilled in the art will recognize that, depending on the particular application and implementation, one or more features or characteristics may be compromised to obtain overall desirable system attributes. These attributes may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, and the like. Thus, even if any embodiment is described as less desirable than other embodiments or prior art embodiments with respect to one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for a particular application.

Claims

1. 1. A computer-implemented method for natural language processing, comprising: Receiving tabular data associated with one or more records; converting the tabular data into a textual representation representative of the tabular data; generating metadata indicative of a description of the tabular data associated with a textual representation of the tabular data; outputting, for one or more iterations, one or more natural language data descriptions representing the tabular data in response to utilizing a large scale language model (LLM) and zero-shot prompting of the metadata and the textual representation of the tabular data, the LLM including a neural network having a plurality of parameters; utilizing the LLM to output one or more summaries for one or more iterations, the one or more summaries including less text than the one or more natural language data descriptions; selecting a single summary of the one or more summaries in response to a single summary having a minimum validation rate for one or more iterations; receiving a query associated with the tabular data; utilizing the single summary and the LLM for the query for one or more iterations to output one or more predictions associated with the query; outputting a final prediction associated with the query in response to a convergence threshold being met by the one or more predictions generated from the one or more iterations, the final prediction being selected in response to a weighted majority vote of all of the one or more predictions generated from the one or more iterations; The method includes:

2. The method of claim 1 , wherein the LLM does not utilize fine-tuning or building of a new language model to output the final prediction.

3. The method of claim 1 , wherein the plurality of parameters is greater than 1 billion.

4. The method of claim 1, wherein the LLM is GPT-4.

5. The method of claim 1 , wherein the textual representation is a parsed JSON file.

6. The method of claim 1 , wherein any numerical features associated with the tabular data are coded into percentiles.

7. The method of claim 1 , wherein utilizing the LLM to output one or more summaries comprises adding prompts to the one or more natural language data descriptions.

8. The method of claim 1 , wherein the one or more predictors are weak learners.

9. The method of claim 1 , wherein the method comprises utilizing weighted stratified sampling to select the single summary.

10. 1. A system comprising: an input interface to the system configured to receive data associated with the system; one or more processors in communication with the input interface; In a system comprising: The one or more processors: receiving tabular data associated with one or more records; converting the tabular data into a textual representation of the tabular data; generating metadata associated with a textual representation of the tabular data indicating a description of the tabular data; outputting one or more natural language data descriptions representing the tabular data in response to utilizing a large scale language model (LLM) and zero-shot prompting of the metadata and the textual representation of the tabular data, wherein the LLM includes a neural network having a plurality of parameters; utilizing the LLM and the one or more natural language data descriptions to output one or more summaries, where the one or more summaries include less text than the one or more natural language data descriptions; selecting a single summary of the one or more summaries in response to a single summary having a minimum validation rate for one or more iterations; receiving a query associated with the tabular data; for one or more iterations, utilizing the single summary and the LLM for the query to output one or more predictions associated with the query; outputting one or more predictions; and in response to a convergence threshold being met by the one or more predictions generated from the one or more iterations that output one or more predictions, outputting a final prediction associated with the query. The system is programmed to do so.

11. 11. The system of claim 10, wherein the processor is further programmed to output the final prediction in response to a weighted majority vote for all of the one or more predictions generated from the one or more iterations.

12. The system of claim 10 , wherein the numerical features associated with the tabular data are encoded into percentiles.

13. 11. The system of claim 10, wherein the one or more processors are programmed to utilize the LLM to output one or more summaries and add prompts to the one or more natural language data descriptions.

14. The system of claim 10 , wherein the one or more processors are collectively programmed to perform each step.

15. The system of claim 10 , wherein the one or more processors include a single processor programmed to perform each step.

16. The system of claim 10 , wherein the one or more processors are programmed to select a representative subset of the one or more summaries using weighted stratified sampling.

17. The system of claim 10 , wherein the one or more processors are programmed to select a representative subset of the one or more summaries using weighted stratified sampling.

18. The system of claim 10 , wherein the one or more processors are programmed to select a representative subset of the one or more summaries from a fixed number of iterations.

19. 11. The system of claim 10, wherein the processor is programmed to output the one or more natural language data descriptions and the one or more summaries for one or more iterations.

20. 1. A computer-implemented method for natural language processing, comprising: Receiving tabular data associated with one or more records; converting the tabular data into a textual representation representative of the tabular data; generating metadata indicative of a description of the tabular data associated with a textual representation of the tabular data; outputting one or more natural language data descriptions representing the tabular data in response to utilizing a large scale language model (LLM) and zero-shot prompting of the metadata and the textual representation of the tabular data; utilizing the LLM to output one or more summaries and add prompts to the one or more natural language data descriptions, the one or more summaries including less text than the one or more natural language data descriptions; selecting a single summary of the one or more summaries in response to a single summary having a minimum validation rate for one or more iterations; creating a single summary subset; receiving a query associated with the tabular data; outputting, for one or more iterations, one or more predictions associated with the query using the LLM for one of the single summaries of the subset of single summaries and the query; generating a subset of one or more predictions; in response to a convergence threshold being satisfied by the subset of the one or more predictions, outputting a final prediction associated with the query, the final prediction being selected in response to a weighted majority vote of the subset of the one or more predictions; The method includes:

Images