Information extraction using large-scale language models

Abstract

A method and system for information extraction includes configuring a language model using an information extraction instruction prompt and at least one labeled example prompt (304). The configuration of the language model is validated using at least one validation prompt (306). Errors made by the language model in response to at least one validation prompt are corrected using a correction prompt (308). Information extraction is performed on unlabeled sentences using the language model (310) to identify relationships from the unlabeled sentences. Actions are performed according to the identified relationships (210).

Description

[Technical Field]

[0001] Related application information This application claims priority to U.S. Patent Application No. 63 / 500,664 filed on 8 May 2023, U.S. Patent Application No. 63 / 522,731 filed on 23 June 2023, and U.S. Patent Application No. 18 / 649,145 filed on 29 April 2024, which are incorporated herein by reference in their entirety. [Background technology]

[0002] This invention relates to information extraction, and more particularly to the use of large-scale language models for extracting information from text. Explanation of related technologies

[0003] Information extraction is a natural language task that analyzes unstructured text to identify the types of information it contains. For example, information extraction can identify named entities in the input text and further identify the relationships between such named entities. [Overview of the project]

[0004] The information extraction method includes constructing a language model using an information extraction instruction prompt and at least one labeled example prompt. The construction of the language model is validated using at least one validation prompt. Errors made by the language model in response to at least one validation prompt are corrected using a correction prompt. Information extraction is performed on unlabeled sentences using the language model to identify relationships from the unlabeled sentences. Actions are performed according to the identified relationships.

[0005] The information extraction system includes a hardware processor and memory for storing a computer program. When executed by the hardware processor, the computer program causes the hardware processor to construct a language model using information extraction command prompts and at least one labeled example prompt; to verify the construction of the language model using at least one validation prompt; to correct errors made by the language model in response to at least one validation prompt using correction prompts; to perform information extraction on unlabeled sentences using the language model to identify relationships from the unlabeled sentences; and to perform actions according to the identified relationships.

[0006] These and other features and advantages will become apparent from the following detailed description of the exemplary embodiment, which will be read in conjunction with the attached drawings. [Brief explanation of the drawing]

[0007] This disclosure provides further details in the following description of preferred embodiments with reference to the following figures.

[0008] [Figure 1] This is a block / flow diagram of an information extraction task performed on natural language input using a large-scale language mode, according to one embodiment of the present invention.

[0009] [Figure 2] This is a block / flow diagram of an information extraction task performed on natural language input using a large-scale language mode, according to one embodiment of the present invention.

[0010] [Figure 3] This is a block / flow diagram of a series of prompts for a large language model, which configures the large language model to perform domain-specific information extraction, according to one embodiment of the present invention.

[0011] [Figure 4]This is a block diagram showing the use of information extraction in the context of medical decision-making in a healthcare facility, according to embodiments of the present invention.

[0012] [Figure 5] This is a block diagram of a computing device that can be configured and used to perform information extraction, according to one embodiment of the present invention.

[0013] [Figure 6] This block diagram shows an exemplary neural network architecture that can be used as part of a large-scale language model according to an embodiment of the present invention.

[0014] [Figure 7] This block diagram shows an exemplary deep neural network architecture that can be used as part of a large-scale language model according to embodiments of the present invention. [Modes for carrying out the invention]

[0015] Large-scale language models can be used to perform a variety of natural language tasks, including information extraction. However, using a general-purpose language model to perform a specific task can lead to inaccuracies because the model has not been specifically trained to perform that task. Similarly, language models may lack access to domain-specific information, resulting in reduced accuracy for domain-specific inputs. Furthermore, the way the model prompts is created significantly impacts the quality of the output.

[0016] Therefore, open information extraction in a specific domain using large language models can potentially be enhanced by in-context learning. When an input sentence is given, the most important relationships including the subject, action, and object are extracted. A set of domain-specific sentences with confidence labels related to ground-truth labels may be used, where the ground-truth labels include the subject, action, and object. The initial prompt may be designed for a general language model that includes an explanation of the information extraction task. A series of prompts are provided along with domain-specific sentences and their associated labels to provide the language model with domain-specific knowledge. Additional prompts can be used to test the language model's understanding and provide corrections in case the model generates incorrect results. The language model can be used to extract information from new unlabeled sentences and provide a confidence score regarding the correctness of its output.

[0017] Referring now to FIG. 1, an exemplary information extraction task is shown. Input sentence 102 is provided to large language model 104. Input sentence 102 is made part of one or more prompts that instruct large language model 104 to perform information extraction. In this example, the information extraction task, among other things, extracts the subject, action, and object and generates output 106, although it should be understood that any suitable information extraction task may instead be performed. For example, in some embodiments, adverbs may be further extracted.

[0018] Any suitable language model can be used for large language model 104. Exemplary language models include ChatGPT, GPT3, and GPT4. The large language model can be implemented as a machine learning model such as a transformational neural network model and has parameters pre-trained on a corpus of large text materials.

[0019] Next, referring to FIG. 2, a method of performing information extraction using a large language model is shown. Block 202 applies a training prompt to the language model 104, defines an information extraction task, and provides examples from a set of labeled training data. The training data may relate to a specific domain that was not used in the original training of the language model 104.

[0020] Block 204 can then perform error correction by, for example, prompting the language model 104 to perform information extraction on additional examples from the labeled training data and comparing its predictions to a predetermined label. The error correction of block 204 can include additional prompts to the language model 104 that indicate the correct answer when the task was performed inaccurately.

[0021] Once the language model 104 is properly instructed about the task, block 206 performs information extraction on the input of the new sentence, and block 208 determines the confidence associated with the extracted information. The confidence score may be provided by a human annotator such as a subject expert. Based on the output of the language model 104, block 210 can perform a response action. For example, this action may be performed in the context of a medical treatment in a medical facility, as will be described in more detail below. The response action can be conditioned on a confidence threshold such as at least 90%.

[0022] For a set of training examples x, there is a set of sentences S = {S1, S2,..., S }, where N is the number of sentences in the set, and a set of respective ground truth labels Y = {y1,..., y N}. Here, each

Number

Number

Number

number

[0023] The training prompt includes a definition of the system's role. For example, this can be expressed in natural language as a command or instruction such as "You are an AI assistant for open information extraction." Subsequent training prompts provide contextual learning, such as defining the format of the relationships to be extracted. For example, such a training prompt could be expressed as follows:

[0024] Identify all subject-action-object combinations in a given sentence and represent them in the form of a triplet (subject, action, object). Here are m examples.

[0025] Text: "Field-specific text: S1"

[0026] relationship:

[0027]

number

[0028]

number

[0029] ...

[0030]

number

[0031] Text: "Field-specific text: S2"

[0032] relationship:

[0033]

number

[0034]

number

[0035] ...

[0036]

number

[0037] ...

[0038] Sentence: “Field-specific sentence: S m "

[0039] relationship:

[0040]

number

[0041]

number

[0042] ...

[0043]

number

[0044] Now, identify all the subject-action-object combinations in m sentences.

[0045] Based on this contextual learning prompt, the language model 104 learns domain-specific knowledge and predicts open information in the input sentence. Therefore, for the input sentence p, the extracted information is referred to as r in this specification. p It can be expressed as follows.

[0046] Error correction in block 204 helps the language model 104 adapt to the domain of the training data. Given a training example {S,Y}, the language model 104 predicts the open information contained therein. Sentence S n Given this, the language model predicts r p This is the set of extracted relationships. An error correction prompt is generated, for example, as follows:

[0047] "No, the correct combination of relationships is,

[0048]

number

[0049]

number

[0050] ...

[0051]

number

[0052] Based on error correction prompts, the large-scale language model here,

number

[0053] Blocks 202 and 204 may further provide instructions to the language model 104 regarding the prediction of confidence scores. For example, a prompt can be constructed as follows:

[0054] "For this sentence p, identify all possible combinations of subject, action, and object, and indicate your confidence level for each combination."

[0055] This prompt generates a predicted r p along with a confidence score c. The confidence score is defined as c = [c1,..., c k , and the ground truth confidence is

Number

[0056] "No, your confidence is incorrect. The correct confidence is

Number

[0057] After confidence correction, the language model 104 can provide a confidence prediction via additional prompts.

[0058] "For this sentence p, provide the confidence for the relationship r p ."

[0059] Next, referring to Figure 3, a sequence of prompts is shown. The first prompt is the task definition prompt 302, which instructs the type of information to be extracted by the language model 104. Following this prompt, a set of examples drawn from the training data is presented as prompts. These two prompts provide the language model 104 with information on how to respond to subsequent prompts, and also provide contextual information regarding domain-specific knowledge that may be encoded in the examples.

[0060] Following the initial setup, a validation prompt 306 follows, which prompts the system to extract examples from the labeled training data and ask the prepared language model 104 to predict predetermined labels for those examples. For examples where the language model 104 provides an incorrect answer, a correction prompt 308 is used to identify the example where the language model 104 made a wrong prediction and what the correct answer should be.

[0061] After the correction is made, an additional verification prompt 306 may be used to verify that the language model 104 is working correctly. In that case, a new sentence or a test prompt 310 containing a sentence may be applied to the language model 104 to extract information and generate a confidence score.

[0062] Next, referring to Figure 4, a diagram of information extraction in the context of a medical facility 400 is shown. Information extraction 408 can be used to process patient information, for example, by identifying relevant facts from patient medical history information and medical records 406. Information extraction 408 can be implemented in a large-scale language model using a set of domain-specific training examples labeled according to their use in a medical context. This method allows for the rapid processing of large amounts of patient information and the identification of important information such as allergen exposure, medications taken, and injury circumstances.

[0063] A healthcare facility may include one or more healthcare professionals 402 who review information extracted from the patient's medical records 406 and determine the patient's medical and treatment needs. These medical records 406 may include self-reported information from the patient, test results, and notes made by healthcare professionals in the patient's file. The treatment system 404 may also be designed to monitor the patient's condition to generate the medical records 406 and to automatically manage and adjust treatment as needed.

[0064] Based on the information extracted from information extraction 408, the medical professional 402 can make medical decisions regarding patient health management that are appropriate to the patient's needs. For example, the medical professional 402 can diagnose the patient's health condition and prescribe specific medications, surgeries, and / or therapies.

[0065] Different elements of the medical facility 400 can communicate with each other via the network 410, for example, using any suitable wired or wireless communication protocol and medium. Thus, the information extractor 408 can receive text information from medical professionals 402, the treatment system 404, and the medical record 406, update the medical record 406 with the output of the language model, and, if applicable, work with the treatment system 404 to automatically manage or modify treatment. For example, if the information extractor 408 indicates a possible allergic reaction to treatment or medication, the treatment system 404 can automatically stop the administration of the treatment.

[0066] Next, referring to Figure 5, an exemplary computing device 500 according to an embodiment of the present invention is shown. The computing device 500 is configured to perform tissue analysis.

[0067] The computing unit 500 may be embodied as any type of computing or computer device capable of performing the functions described herein, including but not limited to computers, servers, rack-based servers, blade servers, workstations, desktop computers, laptop computers, notebook computers, tablet computers, mobile computing units, wearable computing units, network appliances, web appliances, distributed computing systems, processor-based systems, and / or consumer electronic devices. Furthermore, or alternatively, the computing unit 500 may be embodied as one or more compute reads, memory reads, or other racks, reads, computing chassis, or other components of a physically isolated computing unit.

[0068] As shown in Figure 5, the arithmetic unit 500 exemplary includes a processor 510, an input / output subsystem 520, memory 530, a data storage device 540, and a communication subsystem 550, and / or other components and devices commonly found in a server or similar arithmetic unit. In other embodiments, the arithmetic unit 500 may include other or additional components (e.g., various input / output devices) commonly found in a server computer. Furthermore, in some embodiments, one or more exemplary components may be incorporated into another component or form part of another component. For example, memory 530, or part thereof, may be incorporated into the processor 510 in some embodiments.

[0069] The processor 510 can be embodied as any type of processor capable of performing the functions described herein. The processor 510 may be embodied as a single processor, a multiprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a single or multicore processor, a digital signal processor, a microcontroller, or other processor or processing / control circuit.

[0070] Memory 530 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 530 may store various data and software used during the operation of the arithmetic unit 500, such as operating systems, applications, programs, libraries, and drivers. Memory 530 may be communicably coupled to the processor 510 via the I / O subsystem 520 and may be embodied as circuits and / or components to facilitate input / output operations with the processor 510, memory 530, and other components of the arithmetic unit 500. For example, the I / O subsystem 520 may be embodied as, or otherwise include, a memory controller hub, an input / output control hub, a platform controller hub, an integrated control circuit, a firmware device, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 520 may form part of a system-on-a-chip (SOC) and be integrated into a single integrated circuit chip together with other components of the processor 510, memory 530, and arithmetic unit 500.

[0071] The data storage device 540 can be embodied as any type of device or apparatus configured for short-term or long-term storage of data, such as a memory device and circuit, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage device 540 can store program code 540A for constructing a model, program code 540B for performing information extraction, and / or program code 540C for performing diagnosis and treatment. The communication subsystem 550 of the arithmetic unit 500 can be embodied as any network interface controller or other communication circuit, apparatus, or assembly thereof that can enable communication between the arithmetic unit 500 and other remote devices over a network. The communication subsystem 550 can be configured to achieve such communication using any one or more communication technologies (e.g., wired or wireless) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX®, etc.).

[0072] As shown in the figure, the arithmetic unit 500 may also include one or more peripheral devices 560. Peripheral devices 560 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 560 may include a display, a touchscreen, a graphics circuit, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0073] Of course, the arithmetic unit 500 may include other elements (not shown) as readily conceivable to those skilled in the art, and certain elements may be omitted. For example, various other sensors, input devices, and / or output devices may be included in the arithmetic unit 500, depending on specific implementations of the same, as readily understood to those skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, processors, controllers, memory, etc., may be added to enable various configurations. These and other variations of the processing system 500 are readily conceivable to those skilled in the art, given the teachings of the present invention provided herein.

[0074] Referring to Figures 6 and 7, exemplary neural network architectures are shown, such as the large-scale language model 104, which can be used to implement parts of this model. A neural network is a generalized system whose functionality and accuracy improve with exposure to additional empirical data. Neural networks are learned by being exposed to empirical data. During training, the neural network remembers and adjusts multiple weights that are applied to the input empirical data. By applying the adjusted weights to the data, it can identify whether the data belongs to a predefined class from a set of classes, or output the probability that the input data belongs to each class.

[0075] The empirical data obtained from a series of examples (also called training data) is formatted as a string of values ​​and fed into the neural network. Each example is associated with a known result or output. Each column is represented as a pair (x,y), where x is the input data and y is the known output. The input data can be of various data types and may contain multiple different values. The network can have one input node for each value that makes up the example's input data, and each input value can be assigned a separate weight. The input data can be formatted as a vector, array, or string, for example, depending on the architecture of the neural network being built and trained.

[0076] A neural network "learns" by comparing the neural network output generated from input data with known values ​​from examples, and adjusting the stored weights to minimize the difference between the output and the known values. This adjustment can be performed on the stored weights through backpropagation, and the effect of the weights on the output is determined by calculating a mathematical gradient and adjusting the weights in a way that shifts the output to the minimum difference. This optimization, called gradient descent, is a non-restrictive example of how training takes place. A subset of examples with known values ​​not used in training can be used to test and validate the accuracy of the neural network.

[0077] During operation, the trained neural network can be used on new data that has not been previously used for training or validation through generalization. The weights of the tuned neural network can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function captured by the weights are based on statistical inference.

[0078] In a layered neural network, nodes are arranged in layers. An exemplary simple neural network has an input layer 620 with source nodes 622 and a single computation layer 630 with one or more computation nodes 632 that also function as output nodes, with one computation node 632 for each possible category into which the input example can be classified. The input layer 620 may have a number of source nodes 622 equal to the number of data values ​​612 of the input data 610. The data values ​​612 of the input data 610 can be represented as a column vector. Each computation node 632 in the computation layer 630 generates a linear combination of weighted values ​​from the input data 610 supplied to the input nodes 620 and applies a differentiable nonlinear activation function to the sum. The exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0079] Deep neural networks, such as multilayer perceptrons, can have an input layer 620 with source nodes 622, one or more computation layers 630 with one or more computation nodes 632, and an output layer 640 with one output node 642 for each category into which the input example may be classified. The input layer 620 can have a number of source nodes 622 equal to the number of data values ​​612 of the input data 610. The computation nodes 632 of the computation layer 630 are located between the source nodes 622 and the output nodes 642 and are not directly observed, and are therefore also called hidden layers. Each node 632,642 of the computation layer generates a linear combination of weighted values ​​from the output values ​​of the nodes of the previous layer and applies a differentiable nonlinear activation function over the range of the linear combination. The weights applied to the values ​​from each previous node are, for example, w1, w2, ... w n-1 ,w n It can be represented as follows: The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in the computational layer is connected to all nodes in the previous layer, or the connections between layers can be in other configurations. If there are missing links between nodes, the network is said to be partially connected.

[0080] Training a deep neural network involves two phases: a forward phase in which the weights of each node are fixed and the input is propagated through the network, and a backward phase in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0081] One or more computational (hidden) layers 630 compute nodes 632 perform a nonlinear transformation on the input data 612 that generates the feature space. Classes and categories may be easier to separate in the feature space than in the original data space.

[0082] The embodiments described herein may be entirely hardware, entirely software, or may include both hardware and software elements. In preferred embodiments, the present invention is implemented in software including, but not limited to, firmware, resident software, and microcode.

[0083] Embodiments may include computer program products accessible from computer-enabled or computer-readable media that provide program code for use by or in connection with a computer or any instruction execution system. Computer-enabled or computer-readable media may include any device that stores, communicates, propagates, or transports programs for use by or in connection with an instruction execution system, apparatus, or device. The medium may be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor systems (or apparatus or devices), or propagation media. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random-access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0084] Each computer program can be substantially stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk) that is readable by a general-purpose or special-purpose programmable computer, in order to configure and control the operation of the computer when the storage medium or device is read by the computer in order to perform the procedures described herein. The system of the present invention can also be considered to be implemented on a computer-readable storage medium configured with a computer program, in which case the configured storage medium causes the computer to operate in a specific predetermined manner to perform the functions described herein.

[0085] A data processing system suitable for storing and / or executing program code may include at least one processor directly or indirectly coupled to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory that provides at least some temporary storage for the program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system directly or via an intermediary I / O controller.

[0086] Network adapters can also be integrated into a system to enable a data processing system to connect to other data processing systems or remote printers or storage devices via an intervening private or public network. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0087] As used herein, the terms “hardware processor subsystem” or “hardware processor” may refer to a processor, memory, software, or combination thereof that works together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or arithmetic element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more onboard memories (e.g., caches, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.) that may be onboard or offboard, or that may be dedicated for use by the hardware processor subsystem.

[0088] In some embodiments, a hardware processor subsystem may include and execute one or more software elements. These software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0089] In other embodiments, the hardware processor subsystem may include dedicated circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0090] These and other variations of the hardware processor subsystem are also intended in accordance with embodiments of the present invention.

[0091] In this specification, any reference to “one embodiment” or “one embodiment” of the present invention, and to other modifications, means that certain features, structures, properties, etc., described in relation to the embodiments are included in at least one embodiment of the present invention. Therefore, expressions such as “in one embodiment” or “in one embodiment” appearing elsewhere in this specification, and any other modifications, do not necessarily all refer to the same embodiment. However, it should be understood that, considering the teachings of the present invention provided herein, features of one or more embodiments can be combined.

[0092] For example, in the case of "A / B," the use of any of the following " / ," "and / or," or "at least one," such as "A and / or B" or "at least one of A and B," will be understood as intended to include the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to include the selection of only the first listed option (A), only the second listed option (B), only the third listed option (C), only the first and second listed options (A and B), only the first and third listed options (A and C), only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended as many times as there are listed items.

[0093] The foregoing is to be understood in all respects to be illustrative and not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description but from the claims as interpreted in accordance with the full width permitted by patent law. The embodiments shown and described herein are merely illustrative of the invention, and those skilled in the art should understand that various modifications can be implemented without departing from the scope and spirit of the invention. Those skilled in the art can implement various other combinations of features without departing from the scope and spirit of the invention. Thus, while aspects of the invention have been described with the detail and specificity required by patent law, what is claimed and intended to be protected by the patent is as stated in the appended claims.

Claims

1. A method for extracting information, implemented in a computer. Constructing a language model using an information extraction instruction prompt and at least one labeled example prompt (304), Validating the configuration of the language model using at least one validation prompt (306), Correcting errors made by the language model in response to at least one validation prompt using correction prompts (308), Using the language model described above, information is extracted from the unlabeled sentences to identify relationships from the unlabeled sentences (310), A method comprising (210) performing an action according to the identified relationship.

2. In the method according to claim 1, The information extraction instruction prompt and the at least one labeled example prompt include a method comprising a text description of an information extraction task, including a definition of a relational format.

3. In the method according to claim 1, The method by which the at least one labeled example prompt is derived from a set of training data, which includes sentences and associated relationships.

4. In the method according to claim 3, A method by which the at least one validation prompt is also derived from the training data set.

5. In the method according to claim 4, A method for identifying at least one response to a validation prompt that does not match the label of the at least one validation prompt from the training data, and supplying the label to the language model.

6. In the method according to claim 1, A method for determining the confidence score associated with the relationship, wherein the at least one labeled example prompt includes a confidence score, and inputting a test prompt into the language model further determines the confidence score associated with the relationship.

7. In the method according to claim 1, A method for identifying unlabeled sentences as relating to a patient's medical condition.

8. In the method described in claim 7, A method for performing the aforementioned actions, which includes automatically adjusting the treatment of a patient based on the identified relationship.

9. In the method according to claim 7, The identified relationships are stored in the patient's medical history in order to support medical decision-making by healthcare professionals.

10. In the method according to claim 1, The language model is a large-scale language model pre-trained based on a machine learning model.

11. A system for extracting information, Hardware processor (510), When executed by the aforementioned hardware processor, the hardware processor will Constructing a language model using an information extraction instruction prompt and at least one labeled example prompt (304), Validating the configuration of the language model using at least one validation prompt (306), Correcting errors made by the language model in response to at least one validation prompt using correction prompts (308), Using the language model described above, information is extracted from the unlabeled sentences to identify relationships from the unlabeled sentences (310), A system including (210) an operation that performs an operation according to the aforementioned identified relationship, and (540) a memory that stores a computer program that causes the operation to be performed.

12. In the system according to claim 11, The information extraction instruction prompt and the at least one labeled example prompt are a system that includes a text description of an information extraction task, including a definition of a relational format.

13. In the system according to claim 11, The system draws the at least one labeled example prompt from a set of training data, which includes sentences and associated relationships.

14. In the system described in claim 13, The system also derives at least one of the aforementioned validation prompts from the set of training data.

15. In the system described in claim 14, A system that identifies at least one response to a validation prompt whose correction prompt does not match the label of the at least one validation prompt from the training data, and supplies the label to the language model.

16. In the system according to claim 11, A system in which at least one labeled example prompt includes a confidence score, and inputting a test prompt into the language model further determines the confidence score associated with the relationship.

17. In the system according to claim 11, A system where unlabeled sentences relate to the patient's medical condition.

18. In the system described in claim 17, The computer program further includes a system that causes the hardware processor to automatically adjust the patient's treatment based on the identified relationship.

19. In the system described in claim 17, The identified relationships are stored in the patient's medical history in a system to support medical decision-making by healthcare professionals.

20. In the system according to claim 11, The aforementioned language model is a system that is a large-scale language model pre-trained based on a machine learning model.

Images