INFORMATION EXTRACTION WITH LARGE LANGUAGE MODELS

Abstract

Information extraction methods and systems include configuring (304) a language model with an information extraction instruction prompt and at least one prompt with a labeled example. The language model configuration is validated (306) using at least one validation prompt. Errors made by the language model in response to at least one validation prompt are corrected (308) using a correction prompt. Information extraction is performed (310) on an unlabeled sentence using the language model to identify a relation from the unlabeled sentence. An action is carried out (210) in response to the identified relation.

Description

INFORMATION ABOUT RELATED REGISTRATIONS

[0001] This application claims priority over U.S. Patent Application No. 63 / 500,664, filed on May 8, 2023, U.S. Patent Application No. 63 / 522,731, filed on June 23, 2023, and U.S. Patent Application No. 18 / 649,145, filed on April 29, 2024, each of which is incorporated herein by reference in its entirety. BACKGROUND Technical area

[0002] The present invention relates to information extraction and in particular the use of large language models for extracting information from text. Description of the state of the art

[0003] Information extraction is a natural language task that analyzes unstructured text to identify the types of information it contains. For example, information extraction can identify named entities within the input text and furthermore identify relationships between such named entities. SUMMARY

[0004] An information extraction procedure involves configuring a language model with an information extraction instruction prompt and at least one labeled example prompt. The language model configuration is validated using at least one validation prompt. Errors made by the language model in response to the validation prompt are corrected using a correction prompt. Information extraction is performed on an unlabeled sentence using the language model to identify a relation within the unlabeled sentence. An action is then performed in response to the identified relation.

[0005] An information extraction system comprises a hardware processor and memory that stores a computer program. When the computer program is executed by the hardware processor, it causes the hardware processor to configure a language model with an information extraction instruction prompt and at least one prompt with a labeled example, to validate the configuration of the language model using at least one validation prompt, to correct errors made by the language model in response to the at least one validation prompt using a correction prompt, to perform information extraction on an unlabeled sentence using the language model to identify a relation from the unlabeled sentence, and to perform an action in response to the identified relation.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The disclosure will provide details in the following description of preferred embodiments with reference to the following figures, wherein: Fig. 1 a block / flow diagram of an information extraction task performed on a natural language input by a large language model, according to an embodiment of the present invention; Fig. 2 a block / flow diagram of an information extraction task performed on a natural language input by a large language model, according to an embodiment of the present invention; Fig. 3 a block / flow diagram of a series of prompts to a large language model that configure the large language model to perform domain-specific information extraction, according to an embodiment of the present invention; Fig. 4 is a block diagram showing the use of information extraction in the context of medical decision-making in a healthcare facility, according to an embodiment of the present invention; Fig. 5 a block diagram of a computing device that can configure and use a large language model to perform information extraction, according to an embodiment of the present invention; Fig. 6 is a block diagram showing an exemplary neural network architecture that can be used as part of the large language model according to an embodiment of the present invention; and Fig. Figure 7 is a block diagram showing an exemplary deep neural network architecture that can be used as part of the large language model, according to an embodiment of the present invention. DETAILED DESCRIPTION OF PREFERRED EXECUTION FORMS

[0008] Large language models can be used to perform a wide variety of natural language tasks, including information extraction. However, using a general-purpose language model to perform a specific task can lead to inaccuracies because the model is not specifically trained to perform that task. Similarly, the model may lack access to domain-specific information, reducing accuracy for domain-specific inputs. Furthermore, the design of the model's prompt has a significant impact on output quality.

[0009] To this end, domain-specific open information extraction using a large language model can be enhanced through in-context learning. Given an input sentence, the key relations can be extracted, including a subject, an action, and an object. A set of domain-specific sentences with ground-truth labels (reference truth labels) and associated confidence labels can be used, with the ground-truth labels including a subject, an action, and an object. An initial prompt can be designed for a general-purpose language model, including a description of the information extraction task. A series of prompts can be provided, some containing the domain-specific sentences and their associated labels, to provide the language model with domain-specific knowledge.Additional prompts can be used to test the language model's understanding and provide corrections if the model produces erroneous results. The language model can then be used to extract information from unlabeled new sentences and provide a confidence score regarding the correctness of its output.

[0010] Referring to Fig. Figure 1 presents an exemplary information extraction task. Input text 102 is provided to a large language model 104. The input sentences 102 are made part of one or more prompts that instruct the large language model 104 to perform information extraction. In this example, the information extraction task specifically extracts subjects, actions, and objects and produces an output 106, but it should be understood that any suitable information extraction task can be performed instead. For example, some embodiments can further extract adverbials.

[0011] Any suitable language model can be used for the large language model. Exemplary language models include ChatGPT, GPT3, and GPT4. The large language model can be implemented as a machine learning model, such as a transformer neural network model, which has parameters that are pre-trained on a large corpus of textual material.

[0012] Referring to Fig. Section 2 presents a procedure for performing information extraction with a large language model. Block 202 applies training prompts to language model 104, defining the information extraction task and providing examples from a set of labeled training data. The training data may relate to a specific domain that was not used in the original training of language model 104.

[0013] Block 204 can then perform error correction, for example, by prompting the language model 104 to extract information from additional examples in the labeled training data and comparing its predictions with the predetermined labels. Block 204's error correction may include further prompts to the language model 104, instructing it on the correct response in cases where it performs the task inaccurately.

[0014] Once the language model 104 has been properly instructed regarding the task, Block 206 performs information extraction on a new sentence input, and Block 208 determines a confidence level associated with the extracted information. The confidence level can be provided by a human annotator, such as a subject matter expert. Based on the output of the language model 104, Block 210 can perform a responsive action. For example, this action might occur in the context of medical treatment in a healthcare facility, as described in more detail below. The responsive action can be made dependent on a confidence threshold, such as at least 90%.

[0015] For a set of training examples x, there can be a set of sentences S = {S1,S2,...,S N}, give, where N is the number of sentences in the set, and a set of corresponding ground-truth labels Y = {y1,..,y N}, where each yn={(sn1,an1,on1),…,(snK,anK,onK)}, where snk a subject represents ank represents an action onk an object is represented and K is the number of relations for the set S. n is.

[0016] The training prompts include a definition of the system's role. For example, this can be expressed in natural language as a command or instruction, such as: "You are an AI assistant for open information extraction." Subsequent training prompts provide in-context learning by, for example, defining a format for the relations to be extracted. For instance, such a training prompt might be expressed as:

[0017] "Identify all combinations of subject, action, and object for all given sentences and present them in the form of triples: (subject, action, object). Here are some examples:"

[0018] Sentence: ,domain-specific sentence: S1'

[0019] Relations: 1.(s11,a11,o11)2.(s12,a12,o12)…K.(s1K,a1K,o1K)

[0020] Sentence: ,domain-specific sentence: S2'

[0021] Relations: 1.(s21,a21,o21)2.(s22,a22,o22)…K.(s2K,a2K,o2K)…

[0022] Sentence: ,domain-specific sentence: S m '

[0023] Relations: 1.(sm1,am1,om1)2.(sm2,am2,om2)…K.(smK,amK,omK)

[0024] Please now identify all combinations of subject, action, and object for the m sentences.

[0025] Based on this in-context learning prompt, language model 104 learns domain-specific knowledge and predicts open information for an input sentence. Thus, for an input sentence p, the extracted information can be represented here as r. p be designated.

[0026] The error correction of block 204 helps language model 104 adapt to the domain of the training data. Given training examples {S, Y}, language model 104 predicts the open information contained therein. Given a sentence s n , is the language model prediction r p A set of extracted relations. An error correction prompt can be generated, for example as:

[0027] "No, the correct combinations of relations are: 1.(sm1,am1,oM11)2.(sm2,am2,om2)…K.(smK,amK,omK)”

[0028] Based on the error correction prompt, the large language model 104 is able to generate predictions with higher accuracy for information extraction, here referred to as r̂ p .

[0029] Blocks 202 and 204 can also provide instructions to language model 104 regarding the prediction of confidence values. For example, a prompt can be constructed as:

[0030] "For this sentence p, identify all combinations of subject, action and object for all given sentences and also provide the confidence for each combination."

[0031] This prompt generates a confidence value c along with the prediction r. p The confidence value can be expressed as c = [c1, ..., c k] can be defined, and the ground truth confidence can be represented as ĉ. A further prompt can then be issued to perform error correction, for example as:

[0032] "No, your confidence level is incorrect. The correct confidence level is ĉ."

[0033] After the confidence correction, language model 104 can provide its confidence prediction via a further prompt:

[0034] "Please state p, the confidence of the relation r, for this sentence." p ready"

[0035] Referring now to Fig. Figure 3 shows a sequence of prompts. The first prompt is a task definition prompt 302, which instructs the language model 104 what type of information to extract. This is followed by a prompt with a set of examples taken from the training data. These two prompts provide the language model 104 with information on how to respond to subsequent prompts and also provide contextual information regarding any domain-specific knowledge that may be encoded in the examples.

[0036] The initial setup is followed by a validation prompt 306, which pulls examples from the labeled training data and prompts the prepared language model 104 to predict the predetermined labels for these examples. For any examples where the language model 104 provides incorrect answers, a correction prompt 308 is used to identify which example the language model 104 incorrectly predicted and what the correct answers should have been.

[0037] After the correction has been made, additional validation prompts 306 can be used to ensure that language model 104 is working correctly. If so, a test prompt 310 with a new sentence or sentences can be applied to language model 104 to extract information and generate a confidence score.

[0038] Referring now to Fig. Figure 4 shows a diagram of information extraction in the context of a healthcare facility. Information extraction can be used to process patient information, for example, by identifying relevant facts from patient history information and medical records. Information extraction can be implemented with a large language model using a set of domain-specific training examples labeled according to their use in a medical context. A large amount of patient information can be processed quickly in this way to identify important information, such as exposure to allergens, medications taken, and the circumstances of an injury.

[0039] The healthcare facility may include one or more healthcare professionals (402) who review information extracted from a patient's medical records (406) to determine their health and treatment needs. These medical records (406) may include self-reported information from the patient, test results, and notes made by healthcare professionals that have been added to the patient's file. Treatment systems (404) may also monitor patient status to generate medical records (406) and may be designed to automatically administer and adjust treatments as needed.

[0040] Based on information obtained from Information Extraction 408, the medical professionals 402 can then make medical decisions regarding the patient's healthcare that are appropriate for the patient's needs. For example, the medical professionals 402 can diagnose the patient's health condition and prescribe specific medications, surgeries, and / or therapies.

[0041] The various elements of the healthcare facility 400 can communicate with each other via a network 410, for example, using any suitable wired or wireless communication protocol and medium. Thus, the information extraction unit 408 receives textual information from healthcare professionals 402, from treatment systems 404, and from medical records 406, and updates the medical records 406 with the output of the speech model. In some cases, it can coordinate with treatment systems 404 to automatically administer or modify a treatment. For example, if the information extraction unit 408 indicates a potential allergic reaction to a treatment or medication, the treatment systems 404 can automatically stop administering the treatment.

[0042] Referring now to Fig. Figure 5 shows an exemplary computing device 500 according to an embodiment of the present invention. The computing device 500 is configured to perform tissue analysis.

[0043] The Computing Device 500 may be implemented as any type of computing or computer device capable of performing the functions described herein, including, without limitation, a computer, server, rack-based server, blade server, workstation, desktop computer, laptop computer, notebook computer, tablet computer, mobile computing device, portable computing device, network appliance, web appliance, distributed computing system, processor-based system, and / or consumer electronics device. Additionally or alternatively, the Computing Device 500 may be implemented as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically disaggregated computing device.

[0044] As in Fig. As shown in Figure 5, the computing device 500 includes, for illustrative purposes, the processor 510, an input / output subsystem 520, a memory 530, a data storage device 540, and a communication subsystem 550, and / or other components and devices commonly found in a server or similar computing device. In other embodiments, the computing device 500 may include other or additional components, such as those commonly found in a server computer (e.g., various input / output devices). Additionally, in some embodiments, one or more of the illustrative components may be integrated into another component or otherwise form part of it. For example, in some embodiments, the memory 530, or parts thereof, may be integrated into the processor 510.The Processor 510 can be implemented as any type of processor capable of performing the functions described herein. The Processor 510 can be implemented as a single processor, multiple processors, one central processing unit (CPU), one graphics processing unit (GPU), one single-core or multi-core processor, one digital signal processor, one microcontroller, or other processor or processing / control circuitry.

[0045] The Memory 530 can be implemented as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the Memory 530 can store various data and software used during the operation of the Computing Device 500, such as operating systems, applications, programs, libraries, and drivers. The Memory 530 is communicatively coupled to the Processor 510 via the I / O Subsystem 520, which can be implemented as circuitry and / or components to facilitate input / output operations with the Processor 510, the Memory 530, and other components of the Computing Device 500. For example, the I / O Subsystem 520 can be implemented as memory controller hubs, input / output control hubs, platform controller hubs, integrated control circuitry, firmware devices, communication links (e.g.,Point-to-point connections, bus connections, wires, cables, optical fibers, circuit board traces, etc.) and / or other components and subsystems may be implemented to facilitate or otherwise include input / output operations. In some embodiments, the I / O subsystem 520 may form part of a system-on-a-chip (SoC) and be integrated on a single integrated circuit chip together with the processor 510, the memory 530, and other components of the computing device 500.

[0046] The data storage device 540 can be implemented as any type of device or devices configured for short-term or long-term data storage, such as storage devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device 540 can store program code 540A for configuring a model, 540B for performing information extraction, and / or 540C for performing diagnostics and treatment. One or all of these program code blocks can be contained in a given computing system. The communication subsystem 550 of the computing device 500 can be implemented as any network interface controller or any other communication circuit, device, or collection thereof capable of enabling communication between the computing device 500 and other remote devices over a network.The Communication Subsystem 550 can be configured to use one or more communication technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

[0047] As shown, the computing device 500 can also include one or more peripheral devices 560. The peripheral devices 560 can include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, the peripheral devices 560 can include a display, a touchscreen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0048] Of course, the computing device 500 can also include other elements (not shown), as a person skilled in the art can readily imagine, as well as omit certain elements. For example, various other sensors, input devices, and / or output devices can be included in the computing device 500, depending on its specific implementation, as a person skilled in the art can readily understand. For example, different types of wireless and / or wired input and / or output devices can be used. Furthermore, additional processors, controllers, memory, and so on can also be used in various configurations. These and other variations of the processing system 500 are readily apparent to a person skilled in the art in light of the teachings of the present invention provided herein.

[0049] Referring now to Fig. 6 and Fig.Section 7 presents exemplary neural network architectures that can be used to implement parts of the models presented here, such as the large language model 104. A neural network is a generalized system that improves its performance and accuracy through exposure to additional empirical data. The neural network is trained by exposure to the empirical data. During training, the neural network stores and adjusts a variety of weights that are applied to the incoming empirical data. By applying the adjusted weights to the data, the data can be identified as belonging to a specific predefined class from a set of classes, or a probability that the input data belongs to each of the classes can be output.

[0050] The empirical data, also known as training data, from a set of examples can be formatted as a string of values ​​and fed into the input of the neural network. Each example can be associated with a known outcome or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output.

[0051] The input data can encompass a variety of different data types and can include multiple different values. The network can have an input node for each value that constitutes the example's input data, and a separate weight can be applied to each input value. The input data can be formatted as a vector, an array, or a string, for example, depending on the architecture of the neural network being constructed and trained.

[0052] The neural network "learns" by comparing the output it generates from the input data with known values ​​from the examples and adjusting the stored weights to minimize the differences between the output values ​​and the known values. These adjustments to the stored weights can be made through backpropagation, where the effect of the weights on the output values ​​can be determined by calculating the mathematical gradient and adjusting the weights in a way that shifts the output toward a minimal difference. This optimization, known as gradient descent, is a non-restrictive example of how training can be performed. A subset of examples with known values, not used for training, can be used to test and validate the accuracy of the neural network.

[0053] During operation, the trained neural network can be used with new data not previously used in training or validation through generalization. The adjusted weights of the neural network can be applied to the new data, with the weights estimating a function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0054] In layered neural networks, nodes are arranged in the form of layers. An exemplary simple neural network has an input layer 620 consisting of source nodes 622 and a single compute layer 630 with one or more compute nodes 632 that also function as output nodes, with a single compute node 632 for each possible category into which the input sample could be classified. An input layer 620 can have a number of source nodes 622 equal to the number of data values ​​612 in the input data 610. The data values ​​612 in the input data 610 can be represented as a column vector. Each compute node 632 in the compute layer 630 generates a linear combination of weighted values ​​from the input data 610 fed into input nodes 620 and applies a differentiable, nonlinear activation function to the sum.The exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0055] A deep neural network, such as a multi-layered perceptron, can have an input layer 620 consisting of source nodes 622, one or more computation layers 630 with one or more computation nodes 632, and an output layer 640, where there is a single output node 642 for each possible category into which the input sample could be classified. An input layer 620 can have a number of source nodes 622 equal to the number of data values ​​612 in the input data 610. The computation nodes 632 in the computation layer(s) 630 can also be called hidden layers because they lie between the source nodes 622 and output nodes 642 and are not directly observed.Each node 632, 642 in a computation layer generates a linear combination of weighted values ​​from the values ​​output by the nodes in a previous layer and applies a nonlinear activation function that is differentiable over the domain of the linear combination. The weights applied to the value from each previous node can be denoted, for example, by w1, w2, ... w. n-1 , w n The output layer provides the network's overall response to the input data. A deep neural network can be fully connected, where every node in a computational layer is connected to every other node in the previous layer, or it can have other configurations of connections between layers. If connections between nodes are missing, the network is said to be partially connected.

[0056] Training a deep neural network can involve two phases: a forward phase in which the weights of each node are fixed and the input is propagated through the network, and a backward phase in which an error value is propagated backward through the network and weight values ​​are updated.

[0057] The computation nodes 632 in the one or more (hidden) computation layer(s) 630 perform a nonlinear transformation on the input data 612, which generates a feature space. The classes or categories can be separated more easily in the feature space than in the original data space.

[0058] The embodiments described herein may comprise entirely hardware, entirely software, or both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0059] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer or any command-execution system. A computer-usable or computer-readable medium may include any device that stores, communicates, propagates, or transports the program for use by or in conjunction with the command-execution system, device, or apparatus. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or apparatus), or a propagation medium.The medium can include a computer-readable storage medium, such as semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, optical disk, etc.

[0060] Any computer program can be tangibly stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk), readable by a general or special programmable computer, for configuring and controlling the operation of a computer when the storage medium or device is read by the computer to perform the procedures described herein. The system according to the invention can also be considered as embodied in a computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0061] A data processing system capable of storing and / or executing program code may include at least one processor, which is directly or indirectly coupled to memory elements via a system bus. The memory elements may include local memory, used during the actual execution of the program code; mass storage; and cache memory, which provides temporary storage of at least some program code to reduce the number of times code is retrieved from mass storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intermediary I / O controllers.

[0062] Network adapters can also be connected to the system to allow the data processing system to connect to other data processing systems or remote printers or storage devices through intermediary private or public networks. Modems, cable modems, and Ethernet cards are just some of the types of network adapters currently available.

[0063] As used herein, the term “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combinations thereof, working together to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be contained within a central processing unit, a graphics processing unit, and / or a separate processor- or computationally-element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.).In some embodiments, the hardware processor subsystem may include one or more memories, which may be on-board or off-board, or which may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.).

[0064] In some embodiments, the hardware processor subsystem can include and execute one or more software elements. These software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0065] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0066] These and other variations of a hardware processor subsystem are also considered in accordance with embodiments of the present invention.

[0067] Reference in the description to "an embodiment" or "a form" of the present invention, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth, described in connection with the embodiment, is included in at least one embodiment of the present invention. Thus, the appearance of the phrase "in an embodiment" or "in one form," as well as any other variations appearing at different points in the description, does not necessarily all refer to the same embodiment. However, it should be noted that features from one or more embodiments may be combined, given the teachings of the present invention provided herein.

[0068] It should be noted that the use of each of the following “ / ”, “and / or”, and “at least one of”, for example in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to include the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B).As another example, in the cases of "A, B and / or C" and "at least one of A, B and C", such wording is intended to include the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A, B and C). This can be extended to as many listed items as needed.

[0069] The foregoing is to be understood in every respect as illustrative and exemplary, but not limiting, and the scope of the invention disclosed herein is not to be determined from the detailed description, but rather from the claims as interpreted to the fullest extent permitted by patent law. It is to be understood that the embodiments shown and described herein are only illustrative of the present invention and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the art could implement various other combinations of features without departing from the scope and spirit of the invention.Having thus described aspects of the invention, with the details and specificity required by patent law, what is claimed and desired to be protected by Letters Patent is set out in the attached claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 500,664

[0001] US 63 / 522,731

[0001] US 18 / 649,145

[0001]

Claims

[1] A computer-implemented method for information extraction, comprising: Configure (304) a language model with an information extraction instruction prompt and at least one prompt with a labeled example; Validate (306) the configuration of the language model using at least one validation prompt; Correcting (308) errors made by the language model in response to at least one validation prompt, using a correction prompt; Performing (310) information extraction on an unlabeled sentence using the language model to identify a relation from the unlabeled sentence; and Performing (210) an action in response to the identified relation. [2] The method according to claim 1, wherein the information extraction instruction prompt and the at least one prompt with a characterized example include a textual description of an information extraction task, including a definition of a relation format. [3] The method according to claim 1, wherein the at least one input prompt with a characterized example is taken from a set of training data containing sentences and associated relations. [4] The method according to claim 3, wherein the at least one validation input prompt is also taken from the set of training data. [5] The method according to claim 4, wherein the correction prompt identifies a response to the at least one validation prompt that does not match a label of the at least one validation prompt from the training data, and provides the label to the language model. [6] The method according to claim 1, wherein the at least one input prompt with a characterized example contains a confidence value and wherein the input of the test input prompt into the language model further determines a confidence value associated with the relation. [7] The method according to claim 1, wherein the unmarked sentence refers to the medical condition of a patient. [8] The method according to claim 7, wherein performing the action includes automatically adjusting the treatment of a patient based on the identified relation. [9] The method according to claim 7, wherein the identified relation is stored in a medical history of the patient to assist in medical decision-making by a medical professional. [10] The method according to claim 1, wherein the language model is a pre-trained large language model based on a machine learning model [11] An information extraction system comprising: a 510 hardware processor; and a memory 540 that stores a computer program which, when executed by the hardware processor, causes the hardware processor to: Configure (304) a language model with an information extraction instruction prompt and at least one prompt with a labeled example; Validate (306) the configuration of the language model using at least one validation prompt; Correcting (308) errors made by the language model in response to at least one validation prompt, using a correction prompt; Performing (310) information extraction on an unlabeled sentence using the language model to identify a relation from the unlabeled sentence; and Performing (210) an action in response to the identified relation. [12] The system according to claim 11, wherein the information extraction instruction prompt and the at least one prompt with a characterized example include a textual description of an information extraction task, including a definition of a relation format. [13] The system according to claim 11, wherein the at least one input prompt with a characterized example is taken from a set of training data containing sentences and associated relations. [14] The system according to claim 13, wherein the at least one validation prompt is also taken from the set of training data. [15] The system according to claim 14, wherein the correction prompt identifies a response to the at least one validation prompt that does not match a label of the at least one validation prompt from the training data, and provides the label to the language model. [16] The system according to claim 11, wherein the at least one input prompt with a characterized example contains a confidence value and wherein the input of the test input prompt into the language model further determines a confidence value associated with the relation. [17] The system according to claim 11, wherein the unlabeled phrase refers to the medical condition of a patient. [18] The system according to claim 17, wherein the computer program further causes the hardware processor to automatically adjust the treatment of a patient based on the identified relation. [19] The system according to claim 17, wherein the identified relation is stored in a medical history of the patient to assist in medical decision-making by a medical professional. [20] The system according to claim 11, wherein the language model is a pre-trained large language model based on a machine learning model.

Images