Efficient multi-relation prediction with improved context calculation

Abstract

The method includes: accessing a text, wherein within the text, spans are identified and include one or more target span pairs and one or more intermediate context spans; generating an embedded representation of tokens associated with each target span, tokens associated with an entity type of each target span, and tokens associated with each intermediate context span; generating an entity-centered span embedded representation for each target span based on the embedded representation of tokens associated with each target span and the embedded representation of tokens associated with the entity type of the target span; generating an intermediate context embedded representation for each intermediate context span based on the embedded representation of tokens associated with each intermediate context span; and generating a probability distribution for each relation of a set of relations based on the entity-centered span embedded representations of subject and object spans contained in each target pair and the intermediate context embedded representations for intermediate context spans occurring between the subject span and the object span.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 656,831, filed May 7, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to artificial intelligence (AI) technology, and more particularly to techniques for relation extraction with improved contextual computation. Background Technology

[0004] Natural Language Processing (NLP) is a subfield of AI that focuses on using natural language in computer-human interactions. The goal of NLP is to enable machines to understand, interpret, and / or generate human language in a meaningful and context-sensitive manner. NLP can be leveraged in many fields, including but not limited to medical research, clinical health practice, search engines, business intelligence, and digital or virtual assistants. For example, in clinical health practice, NLP can be used to summarize large amounts of text, perform automated doctor-patient speech recognition, and / or identify Personally Identifiable Information (PII) / Protected Health Information (PHI). To achieve this goal independently of the application domain, NLP involves a combination of language and computational methods or tasks that enable computers to understand, interpret, and / or generate human language. These methods or tasks collectively enable computers to process and understand natural language, allowing them to perform various tasks such as text summarization, machine translation, and question answering.

[0005] Named Entity Recognition (NER) and Relation Extraction (RE) are specific methods or tasks within Natural Language Processing (NLP). NER is a task in which the goal is to identify and classify entities within text. Entities are typically real-world objects, such as people's names, organizations, locations, dates, numbers, etc. For example, in the sentence "Oracle Corporation is headquartered in Austin, Texas," NER can identify "Oracle Corporation" as an organization and "Austin, Texas" as a location. RE involves identifying and classifying the relationships between entities mentioned in text. Here, the relationship between entity types "organization" and "location" could be "headquarteredAT". Using RE, this relationship could be encoded as: headquarteredAT(OracleCorporation,Austin). Summary of the Invention

[0006] The techniques disclosed in this paper generally relate to AI techniques, and more particularly to techniques for relation extraction with improved contextual computation.

[0007] In various embodiments, a computer-implemented method is provided, comprising: accessing text comprising a sequence of words, wherein multiple spans are identified within the text, each of the multiple spans comprising one or more words in the word sequence, and the multiple spans comprising one or more target span pairs and one or more mid-context spans, each of the one or more target span pairs being associated with an entity type and comprising a subject span and an object span, each of the one or more mid-context spans appearing between the subject span and object span included in each target pair; generating embeddings of lemmas associated with the one or more words appearing within each target span by a pre-trained machine learning (ML) model. The pre-trained ML model generates an entity-focused span embedding representation for each target span based on the embedding representations of lemmas associated with the entity type of each target span and the embedding representations of lemmas associated with the entity type of that target span; generates an intermediate context embedding representation for each intermediate context span based on the embedding representations of lemmas associated with each intermediate context span; generates an intermediate context embedding representation for each intermediate context span based on the embedding representations of lemmas associated with each intermediate context span; generates a probability distribution for each relation in a relation set for each target pair based on the entity-focused span embedding representations of the subject span and object span contained in each target pair and the intermediate context embedding representations of the intermediate context spans that appear between the subject span and object span of that target pair, the relation set including empty relations; and predicts a given relation for a target span pair based on the probability distributions.

[0008] In some embodiments, the entity-centric span embedding representation is generated by integrating entity-type-specific information with the embedding representation of lemmas associated with each target span using entity-independent span lexical embeddings, span-aligned entity embeddings, and span-independent entity representations.

[0009] In some embodiments, the computer-implemented method further includes: applying activation functions to the entity-centric span embedding representations of the subject span and object span, and the intermediate context embedding representation of the intermediate context span appearing between the subject span and object span of each target pair, by the fully connected layers of the pre-trained ML model, to obtain activated entity-centric span embedding representations of the subject span and object span of each target pair and activated intermediate context embedding representations.

[0010] In some embodiments, the computer-implemented method further includes, for each target pair, generating a concatenation of activated entity-centric span embedding representations of the subject span and object span associated with each target pair by a fully connected layer.

[0011] In some embodiments, a probability distribution for each relation of each target pair is generated based on the cascade.

[0012] In some embodiments, generating a probability distribution for each relation of each target pair takes into account entity-type-specific information integrated into the active entity-centric span embedding representation of the subject span and object span contained in each target pair.

[0013] In some embodiments, the entity-independent span lexical embedding maintains the embedding representation of lexical units associated with one or more words within the target span, the span-aligned entity embedding includes a fixed-length representation of entity types based on soft alignment with lexical units of the target span, and the span-independent entity representation includes a representation of entity types that do not depend on lexical units of the target span.

[0014] In various embodiments, a computer system is provided that includes one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform some or all of the operations and / or methods disclosed herein.

[0015] In various embodiments, one or more non-transitory computer-readable media are provided that store instructions, when executed by one or more processors of a computer system, cause the one or more processors to perform some or all of the operations and / or methods disclosed herein.

[0016] The techniques described herein can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, which are described in more detail below. However, the following implementations and contexts are only a small fraction of the many implementations and contexts. Attached Figure Description

[0017] Figure 1 This is a simplified block diagram of the system determined based on the relationships between various embodiments.

[0018] Figure 2 This is a simplified flowchart of relation extraction technology.

[0019] Figure 3 This is a simplified block diagram of a relation extraction system using R-BERT.

[0020] Figure 4 This is a simplified block diagram of a relation extraction system using R-BERT.

[0021] Figure 5A This is a simplified block diagram of a relation extraction system using a modified R-BERT according to at least one embodiment.

[0022] Figure 5B This is a simplified block diagram of a relationship extraction system using a modified R-BERT for batch processing according to at least one embodiment.

[0023] Figure 6 This is a simplified block diagram of a shared layer architecture for joint NER and RE tasks according to various embodiments.

[0024] Figure 7 This is a simplified block diagram of the processes performed by the system based on various embodiments.

[0025] Figure 8 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0026] Figure 9 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0027] Figure 10 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0028] Figure 11 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0029] Figure 12 This is a block diagram illustrating an example computer system according to at least one embodiment.

[0030] Figure 13 It is a simplified block diagram of the system based on the relationships in a distributed computing environment according to various embodiments.

[0031] Figure 14It is a simplified block diagram of the system determined based on the relationships in the cloud service provider (CSP) infrastructure according to various embodiments. Detailed Implementation

[0032] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0033] I. Introduction

[0034] Artificial intelligence (AI) technology has numerous applications in natural language processing (NLP). In the medical setting, AI can be widely used for drug discovery, disease diagnosis, disease prediction, providing disease prognosis, identifying potential drug interactions, performing medical imaging analysis, and for other purposes. AI technologies such as Natural Language Processing (NER) can be used to identify and classify entities in medical and other texts. Additional NLP techniques, such as Natural Language Analysis (RE), can be combined with NER to enhance the natural language processing of textual information.

[0035] Entity Recognition (also known as Entity Identification (EI) and Entity Extraction) is the task of locating and classifying atomic elements in text into predetermined categories or entity types (such as names of people, organizations, locations, expressions of time, quantities, monetary values, percentages, etc.). For example, in the text "John sold 5 companies in 2002", a trained ML model can identify the entity John as "PERSON", the entity "5 companies" as "QUANTITY", and the entity "2002" as "DATE", where "PERSON", "QUANTITY", and "DATE" are included in a predetermined set of entity types. Additional words ("sold" and "in") can be identified as non-entity or empty entities.

[0036] Accordingly, the NER task of a trained ML model has the objective of finding entities (or names) in text and classifying them by entity type. As mentioned above, the entity types included in the predefined set of entity types can include people, quantities, and dates. However, this is not intended to be restrictive. For example, the predefined set of entity types can also include organizations, locations, etc. In the medical field, the predefined set of entity types could include people, drugs, diseases, side effects, anatomical regions, etc.

[0037] Relation prediction or relation extraction tasks connect entity types through relations contained in a predefined set of relations. For example, given the text "American Airlines, a unit of AMR, immediately matched the price of United Airlines, spokesperson Tim Wagner said. United, a unit of UAL, said that the price increase took effect Thursday," the NER task can identify the entities American Airlines, AMR, United Airlines, Tim Wagner, and UAL. The relation extraction task can then connect these entities (in the form of spans) through relations. For example, the relation extraction task can connect American Airlines and AMR through the relation "subsidiary," where American Airlines is a subsidiary of AMR. Similarly, the relation extraction task can connect Tim Wagner and American Airlines (Tim Wagner is an employee of American Airlines) through the relation "employee," and connect United Airlines and UAL (United Airlines is a subsidiary of UAL) through the relation "subsidiary."

[0038] As used in this article, a span is a set of words. A span can include a single word or several consecutive words in a sequence of words in the input text. A span can be an entity span that can be associated with an entity type. Alternatively, a span can be a non-entity span, i.e., an empty entity.

[0039] As mentioned above, the relationships (e.g., relationship types) included in the predefined set of relationship types can include subsidiaries and employees. However, this is not intended to be restrictive, and the predefined set of relationships can include relationships defined for a particular domain. For example, in the medical field, the predefined set of relationships could include "medication-takenby", "disease-sufferedby", "dosage-of-medicine", etc.

[0040] Relation extraction methods can use pre-trained language models. Initially, the text is lexicalized, and then possible spans are generated using lexical representations.

[0041] After lexicalization, the lexical representations (capturing the spans of interest) are fed into a pre-trained language model, such as the Bidirectional Encoder Representation (BERT) from the Transformer. The BERT-based model generates embeddings in a latent space of a fixed dimension. Different relation extraction models process the span embeddings in different ways and pass the processed embeddings to fully connected layers. The output of the fully connected layers is then passed to a softmax module to compute the probability distribution of the golden relation for the span pairs in question.

[0042] For training, an appropriate log-likelihood loss function is formulated from the probability distribution of relations, which is optimized using a batch process. This results in the parameters of the network (fully connected layers and a BERT-like encoder) in the learning model. A small number of dropouts can be added appropriately. During inference, the probability distribution of relations for each candidate span pair in the context of the associated sentences can be computed. Relation symbols can be assigned to span pairs corresponding to the mapping entity pair that obtains the highest probability (among all relation candidates). A special relation called 'No-Relation' can be used in the computation, which indicates that there may be no relation between span pairs.

[0043] However, the current technology has a drawback: it does not provide entity type information or contextual information corresponding to intermediate words appearing between entities of interest at the output of the pre-trained language model. This may lead to suboptimal relation type predictions.

[0044] Accordingly, different approaches are needed to address these and other challenges.

[0045] In various embodiments, a computer-implemented method is provided, comprising: accessing text comprising a sequence of words, wherein multiple spans are identified within the text, each of the multiple spans comprising one or more words in the word sequence, and the multiple spans comprising one or more target span pairs and one or more intermediate context spans, each of the one or more target span pairs being associated with an entity type, each pair comprising a subject span and an object span, each of the one or more intermediate context spans appearing between a subject span and an object span included in each of the one or more target span pairs; generating, by a pre-trained machine learning (ML) model, embedding representations of lemmas associated with one or more words appearing within each target span in the target spans, embedding representations of lemmas associated with an entity type in each target span in the target spans, and embedding representations of lemmas associated with an entity type appearing within each of the one or more intermediate context spans. The method includes: embedding representations of lemmas associated with multiple intermediate context words; generating entity-centric span embedding representations for each target span based on the embedding representations of lemmas associated with each target span and the embedding representations of lemmas associated with the entity type of the target span; generating intermediate context embedding representations for each intermediate context span based on the embedding representations of lemmas associated with each intermediate context span; generating probability distributions of each relation in a relation set for each of the one or more target span pairs based on entity-centric span embedding representations of subject and object spans contained in each target pair and intermediate context embedding representations of intermediate context spans appearing between the subject and object spans in the pair; and predicting a given relation for a target span pair based on the probability distributions.

[0046] In the disclosed techniques, unlike related domain techniques, entity type-specific information is provided at the output of a pre-trained language model (e.g., BERT) and can be passed to fully connected layers and softmax layers for relation extraction tasks.

[0047] In the disclosed technique, the intermediate context information is calculated according to a specific improvement technique; thereby providing the improved context information of the intermediate context to the fully connected layer and softmax layer of the relation extraction task.

[0048] Compared to current systems and methods, the disclosed techniques can provide richer embedded representations containing contextual information to fully connected layers, leading to substantial improvements in the accuracy of the probability distribution of relations, which can be integrated into a framework for predicting multiple relations in a single pass.

[0049] Furthermore, since the disclosed technique is able to predict the relationship of all target span pairs in an input text block in a single transmission, latency can be significantly reduced while conserving computational resources.

[0050] II. Relationship Determination System

[0051] Figure 1 This is a simplified block diagram of a relationship determination system 100 according to various embodiments. The relationship determination system 100 can be implemented using one or more computer systems, each having one or more processors. The relationship determination system 100 may include multiple components and subsystems communicatively coupled to each other via one or more communication mechanisms. For example, in Figure 1 In the embodiments depicted, the relationship determination system 100 includes a span set preparation subsystem 102 and a relationship extraction subsystem 104.

[0052] These subsystems can be implemented as one or more computer systems. Figure 1 The systems, subsystems, and other components described herein may be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, or using hardware or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 1 The relationship determination system 100 described herein is merely an example and is not intended to unduly limit the scope of the embodiments. Many variations, alternatives, and modifications are possible. For example, in some embodiments, the relationship determination system 100 may have more than Figure 1 The more or fewer subsystems or components shown may be combined into two or more subsystems, or may have different configurations or arrangements of subsystems. Figure 1 The relationship determination system 100 and subsystems described herein can be used with one or more computer systems (such as...) Figure 12 This is achieved through the computer system described in the text.

[0053] like Figure 1 As shown, the relation determination system 100 also includes a storage subsystem 110, which can store various data constructs and procedures used by the relation determination system 100. For example, the storage subsystem 110 can store an entity type set 112 and a relation type set 114.

[0054] The relation determination system 100 is configured to receive a sequence of words 101, such as a block of text, and output target span pairs and the predicted relation type between the two target spans contained in each pair.

[0055] In this paper, the span corresponding to an entity type (e.g., included in entity type set 112) is referred to as the target span. A target span can share a relation with another span (e.g., another target span). Two target spans that can share a relation are referred to as a target span pair or target pair. Each target span pair includes a subject span and an object span; that is, the target spans in each pair are linked by a subject-object relation, where the subject span and object span of the pair are associated with corresponding entity types that may be different from each other. A target span that is a subject span in one pair can be regarded as an object span in another target pair in the batch processing described below.

[0056] Furthermore, the span between the target spans of a target span pair is called the intermediate context span. The intermediate context span includes one or more intermediate context words that appear between the subject span and the object span contained in the target span pair.

[0057] Continue to refer to Figure 1 The span set preparation subsystem 102 receives text as input, such as a document, a portion of text, or a block of text. The span set preparation subsystem 102 can identify spans in the text.

[0058] In an embodiment, the span set preparation subsystem 102 can receive a sequence of input words. D = { w 1 , w 2 ，…，w n The text block is a span set preparation subsystem 102 that can use the input word sequence. D Construct span set S ={ s 1 , s 2 ,…，s N The span set includes spans within the text of the document (up to a reasonable length). For example, the span set can include all spans within a sentence, within a block of text, within the document, etc. The length of each span can be from one to five consecutive words. However, this is not intended to be restrictive. In some implementations, the span length can be greater than five words.

[0059] Given the example input text, “The patient is advised Telmisartan for hypertension. His BP reading is moderate.”, the span set preparation subsystem 102 can construct the span set. SExamples of spans generated for the above sentences could include:

[0060] etc.

[0061] Span set preparation subsystem 102 can provide span sets S As input to Named Entity Recognition (NER) task 122. Although in Figure 1 NER task 122 is shown as a component of relation extraction subsystem 104, but this is not intended to be limiting. In some implementations, NER task 122 may be a component separate from relation extraction subsystem 104.

[0062] As mentioned above, storage subsystem 110 can store entity type set 112. For example, entity type set 112 includes a collection of predefined entity types or classes. E For example, people, drugs, diseases, etc. The entity type set 112 can be provided as input to the NER task 122.

[0063] NER task 122 is acceptable, for each span. Predict entity type (in E It is a predefined collection of entity types, including empty entity types. )or (The latter means) s i This does not represent the fact of an entity; for example, it is a non-entity such as the word "is". As mentioned above, in the embodiment, each entity of the corresponding entity type within the text-predicted entity type set 112 corresponds to a target span, for example, a span that can share a relationship with another span (e.g., another target span).

[0064] NER quest 122 can then output enhanced entities. For example, NER task 122 can output annotated text corresponding to the input text, with inserted tags to indicate entities and their types. In this paper, the augmented entities output by NER task 122 can be referred to as annotated entities. In the annotated text, the augmented entities have corresponding tags, and intermediate context spans are arranged between the annotated entities.

[0065] For example, input text D The text is: "The patient is advised of hypertension. His blood pressure reading is moderate." NER Task 122 can use the input text... DThe entity is mapped to the entity type (or empty entity) of entity type set 112, as shown below:

[0066] As shown above, "the patient" is mapped to a person, "telmisartan" is mapped to a drug, "hypertension" is mapped to a disease, "BP reading" is mapped to a clinical examination, "moderate" is mapped to severity, and "advised" is mapped to ( ). (Other spans in the text are mapped to empty entities.)

[0067] Below is an example of a target span with mapped entity types:

[0068] Accordingly, the output of NER task 122 can be annotated text with entity values ​​augmented by entity types. Intermediate context spans are arranged between augmented entities (e.g., target spans).

[0069] The relation extraction subsystem 104 may further include a lexicon 123. The lexicon 123 may receive annotated text from the NER task 122 as input. The lexicon 123 may perform some processing on the annotated text and output lexicons, where each lexicon corresponds to a word for each span (e.g., a target span or an intermediate context span). The lexicon 123 may also output lexicons corresponding to words contained in the entity type description.

[0070] The relation extraction subsystem 104 may also include a pre-trained language model 127. As an example, the pre-trained language model 127 may be a BERT model, such as a multilayer bidirectional transformer encoder. In embodiments, the BERT model may be Bio-BERT, Med-Bert, Clinical BERT, etc.

[0071] The pre-trained language model 127 can receive the lexical units output by the lexer 123 as input. The pre-trained language model 127 can perform some processing on the lexical units and output the lexical embeddings corresponding to the lexical units.

[0072] The relation extraction subsystem 104 may also include a modified RE task 130. See below for reference. Figure 5A Describe the modified architecture and detailed operation of RE task 130.

[0073] In this embodiment, the modified RE task 130 can receive lexical embeddings output by the pre-trained language model 127 as input. The modified RE task 130 can also receive a set of relation types 114 stored in the storage subsystem 110 as input. For example, the set of relation types 114 includes a set of predetermined relation types or classes. R .

[0074] The modified RE task 130 can target text D For each target span pair identified (e.g., labeled entities) (e.g., {[the patient, person]; [Telmisartan, med]}), based on the contextual information of the specific target span pair and the contextual information of the intermediate contextual spans appearing in the word sequence between the specific target span pairs, the system predicts the relation type included in the relation types contained in the relation type set 114, or predicts no relation (e.g., an empty relation). For example, the contextual information of a specific target span pair may be provided by entity type information.

[0075] The modified processing of RE task 130 can be described as follows: For input text D All target span pairs appearing in the text, such as the labeled entity. and Predict relation type in a single pass ,in R It contains empty relation (That is, the relation type can be no relation—for example, A predefined set of relations. The output for all annotated pairs within this text block is given as a set of triples: It appears in the text block.

[0076] above, Y e Indicates the span of the tagged entity (e.g., target span). ( s i , e k ) is the subject span of the target span pair (e.g., the first target span). ( s j , e l ) is the object span of the target span pair (e.g., the second target span). s i It is the entity value of the main span. e kIt is the entity type with the main span. s j It is the entity value of the object span. e l It is an entity type with a span of objects. y r This indicates a relationship mapping between span pairs. r It is a given relationship type between a given subject span and an object span pair. R Indicates a null relation The set of relation types is 114. S This represents a span set.

[0077] Accordingly, the modified RE task 130 can output a set of relations for target pairs of text blocks. .

[0078] For example, based on the input text D The example above, "The patient is advised Telmisartan for hypertension. His BP reading is moderate.", could output the following triplet:

[0079] In the example above, each triple has a main entity. s i and the corresponding main entity type e k and object entity s j and the corresponding main entity type e l That is, the target span is linked by subject-object relations. For example, in the pair {Thepatient, Telmisartan}, patient is the subject and Telmisartan is the object. Furthermore, each triple includes relations such as med-takenby, disease-sufferedby, and severity-clinexam.

[0080] III. Relationship Extraction Techniques

[0081] Figure 2 A simplified diagram of an example of a related domain relationship extraction system 200 is shown.

[0082] The relation extraction system 200 includes NER task 222 and RE task 224.

[0083] Input sentence 210 can be fed into NER task 222. NER task 222 processes the sentence and outputs labeled entities 212, where each entity (e.g., entity value) is augmented with its corresponding entity type. Each labeled entity (e.g., augmented entity) represents a target span, such as a span that can share a relationship with another span.

[0084] The labeled entity 212 is tokenized and passed to the BERT model 227.

[0085] The output of BERT model 227 is a token embedding associated with each labeled entity, which is then passed to a fully connected layer 216. The fully connected layer 216 maps each token embedding to a scalar value passed to a softmax layer 218. The softmax layer 218 can construct a softmax function using one or more scalar values. The softmax function provides the probability of a given relation between a given pair of target spans. If, for a given pair of spans, the probability of a particular relation type is higher than the probability of any other relation type in the set of relation types, then that particular relation type is assigned as the predicted relation between the given pair of spans.

[0086] For example, the input sentence 210 is “The patient is advised Telmisartan for hypertension”, and the target span pair is “the patient” and “telmisartan”. This pair is passed through the BERT model 227 to obtain lexical embeddings. The lexical embeddings are passed to a fully connected layer 216 to obtain scalar values ​​for, for example, all possible relation types used in a predefined set of relation types. A softmax layer 218 can construct a softmax function using the scalar values ​​and provide a softmax probability distribution that identifies the probability that patient and telmisartan are joined by each relation type. For example, if the probability value of the relation type “med-takenby” is greater than the probability of other relation types in the predefined set of relation types (e.g., Disease-Sufferedby, Severity-Clinexam, etc.), then it can be determined that patient and telmisartan are joined by the relation type “med-takenby”.

[0087] A. Relation extraction using R-BERT

[0088] Figure 3 A diagram showing an example of a domain relation extraction system 300 including the R-BERT model 302 is provided.

[0089] exist Figure 3 In the text, the annotated sentence 310 indicates the output of the NER task. For example, Figure 2 The input sentence 210 shown is labeled. For example, each target span pair includes a first target span (e.g., subject span) and a second target span (e.g., object span). The subject span of the labeled sentence 310, such as "The patient," is determined by the label at the beginning of the subject span. <s:e1> and the mark at the end of the main span< / s:e1> Indicator. The object span of the tagged sentence 310, such as "telmisartan", is indicated by the marker at the beginning of the object span. <s:e2> and the mark at the end of the object span< / s:e2> Instruction. Between the subject span and the object span, there exists an intermediate contextual span, "is advised".

[0090] The annotated sentence 310 is passed to the lexicon 312. The lexicon 312 is configured to convert each word across the span into a lexicon. As shown in the figure, the string... Indicator span s1 ,string Indicates the intermediate context span, string Indicator object span s2 and string Indicates the span corresponding to "forhypertension".

[0091] like Figure 3 As shown, the special word character " "Inserted at the beginning and end of the main span (corresponding to the markers respectively)" <s:e1> and< / s:e1> ), and special word motifs' 'Inserted at the beginning and end of the object's span (which correspond to the markers respectively) <s:e2> and< / s:e2> These special morphemes capture and span s1 and s2 Corresponding entity type e 1 and e 2 The embedding of [something]. Similarly, the special morpheme "CLS" indicates the beginning of a sentence.

[0092] The lexical span output by lexer 312 can be passed to BERT model 327. BERT model 327 can be used for models with two target spans and entities. e1 and e2 Sentences s Positional information of the two target spans is captured based on special morphemes inserted at the beginning and end of the first target span (e.g., the subject span) and the beginning and end of the second target span (e.g., the object span).

[0093] The output of BERT model 327 is a lexical embedding 314 associated with the target span of the lexical sequence received from lexer 312. For example, the embedding that appears in the first target span... s The lexical units in 1 are represented as ,in h i Indicates the first word element, and h j This indicates the last word in the first target span. Similarly, it appears in the second target span. s The lexical units in 2 are represented as Lexical units appearing in the intermediate context are represented as .

[0094] The lexical embeddings of the first and second target spans (e.g., subject span and object span) are averaged over all representative lexical units in that span to obtain vector representations of the subject span and object span, respectively.315

[0095] Similarly, the CLS lexical is embedded as h 0.

[0096] The vector representations 315 of the subject span and object span are passed to the fully connected layer with the activation function (Figure 316). However, although the BERT model 327 generates the lexical units of the intermediate context, no further computation is performed on these lexical units, and information about the intermediate context is not passed to the fully connected layer with the activation function (Figure 316).

[0097] When the vector representations 315 of the subject span and the object span are passed to a fully connected layer with an activation function (reference numeral 316), the activation function and the fully connected layer are applied to obtain:

[0098] Simplification is achieved by setting the matrix as follows: W1 = W2 = W and b 1= b 2= b .

[0099] Activation functions and fully connected layers are also applied to the vector representation of CLS:

[0100] The vector representation is concatenated and passed to the fully connected layer 216 feedforward network (FFN) and the linear layer to obtain: For connecting target spans 1 and s 2 Specific Relationship r , where w3 is a vector.

[0101] Probability of generating a softmax layer of 218:

[0102] Where R is the set of all relation types.

[0103] The problem with the above techniques is that repeated computation is required for each target span pair, for example, when it is expected that multiple pairs of relations will be predicted in a single pass (batch prediction). For example, although the span corresponding to the entity "hypertension" exists in the input sentence 210, the relation extraction system 300 cannot process more than one target span pair in the same computation.

[0104] B. Using R-BERT for relation extraction in batch processing

[0105] Figure 4 It shows that it can be based on Figure 3 A diagram of the relation extraction system 300, including an example of the R-BERT model 402 for batch computation.

[0106] exist Figure 4 In the text, the annotated sentence 410 indicates the output of the NER task, similar to... Figure 3 The example shown. The first target span of the annotated sentence 410, such as "The patient", is marked by the tag at the beginning of the first target span. <s:e1> and the mark at the end of the first target span< / s:e1> Indicator. The second target span of the tagged sentence 410, such as "telmisartan", is indicated by the marker at the beginning of the second target span. <s:e2> Mark at the end of the second target span< / s:e2> Indicator. The third target span of the tagged sentence 410, such as "hypertension", is indicated by the marker at the beginning of the third target span. <s:e3> and the mark at the end of the third target span< / s:e3> Instructions. Between the target span, there are two intermediate context spans: "is advised" and "for".

[0107] The annotated sentence 410 is passed to the lexicon 412. The lexicon 412 is configured to convert each word across the span into a lexicon. As shown in the figure, the string... Corresponding to the first target span s1 ,string Corresponding to the first intermediate context span "is advised", the string Corresponding to the second target span s2 ,string Corresponding to the second intermediate context span "for", and the string Corresponding to the third target span s3 .

[0108] Similar to Figure 3 Special word character " "Inserted at the beginning and end of the first target span (which correspond to the markers respectively)" <s:e1> and< / s:e1> ), and special word motifs' 'Inserted at the beginning and end of the second target span (which correspond to the markers respectively) <s:e2> and< / s:e2> Additionally, the special word " "Inserted at the beginning and end of the third target span (which correspond to the markers respectively)" <s:e3> and< / s:e3> ).

[0109] The lexicalized spans output by lexer 412 can be passed to BERT model 327. BERT model 327 can capture entity types based on special lexical segments inserted at the beginning and end of the first, second, and third target spans. e1 , e2 and e3 Location information for the first, second, and third target spans.

[0110] The output of BERT model 327 is a lexical embedding of 414. For example, appearing in the first target span s The lexical units in 1 are represented as ,in h i Indicates the first word element, and h j This indicates the last word in the first target span. It appears in the second target span. s The lexical units in 2 are represented as It appears in the third target span. s The lexical units in 3 are represented as Lexical units appearing in the first intermediate context span are represented as And the lexical units appearing in the second intermediate context span are represented as .

[0111] Then, the lexical embeddings are averaged over all representative lexical units in the target span to obtain a vector representation of the target span (e.g., subject span and object span) and CLS 415.

[0112] The triples of vector representation 415, such as {h0, h1, h2} and {h0, h1, h3}, are passed to fully connected layers with activation functions (see figures 316, 416) to apply the activation functions to the vector representations, similar to the reference above. Figure 3The content described. However, this is not intended to be restrictive. For example, other triples, such as vector representations of words derived for a target span of an input text block, can also be fed into a fully connected layer with an activation function, such as {h0, h2, h3}.

[0113] Although Figure 4 The figure illustrates two fully connected layers with activation functions (reference numerals 316 and 416), but this is not intended to be restrictive. For example, a single fully connected layer with an activation function can be used.

[0114] Vector representations are concatenated. For example, each concatenation may include a vector representation of the target span pair (e.g., subject span and object span) and a vector representation of the CLS:

[0115] The concatenation is passed to fully connected layers 216 and 418, and then to softmax layers 218 and 424 to obtain the probability of a given relationship between a given target span pair, as shown in the reference above. Figures 1 to 3 As described.

[0116] Although Figure 4 Two connected layers 216 and 418 and two softmax layers 218 and 424 are shown, but this is not intended to be limiting. For example, a fully connected layer and / or a softmax layer could be used.

[0117] In the techniques described above, the relation type can be calculated for each target span pair appearing in the text. However, this technique has problems.

[0118] For example, CLS lexical embeddings should be distinct for scenarios predicting more than one relation. For instance, when considering cross-sentence relations, one relation can be formed using the first sentence, and another relation can span two sentences. For example, consider the input text “The patient is advised Telmisartan. He is having hypertension for the last two months.” Examples of relations in this text could be: Relation 1: Drug_advised_for_patient(The patient(PERSON), Telmisartan(Drug)). Relation 2: Drug_for_disease(Hypertension(disease), Telmisartan(Drug)). Ideally, CLS lexical embeddings should compute self-attention from other lexical embeddings using the following text segments: Relationship 1: CLS: The patient is advised Telmisartan. Relationship 2: CLS: The patient is advised Telmisartan. He is havinghypertension for the last two months. In the techniques described above, placing different CLS terms in two different locations can be very difficult in order to predict relations in batches. For example, it may be difficult to set the context length for each CLS term. Additionally, the number of CLS terms may increase as the number of relations in the input text increases.

[0119] Furthermore, in cases involving cross-sentence relation prediction, it is desirable to have representations of lexical units that appear between the subject entity span and the object entity span, which would serve as context. However, the existing R-BERT framework lacks provisions for calculating intermediate context, and therefore, the accuracy of the calculation may be affected.

[0120] C. Relation extraction using modified R-BERT

[0121] Figure 5A A diagram illustrates an example of a relation extraction system 500 including a modified R-BERT model 502 according to at least one embodiment. For example, the relation extraction system 500 may correspond to a relation extraction subsystem 104.

[0122] By using a novel modified R-BERT model 502, the relation extraction system 500 is able to extract the relation types of all target pairs present in at least one sentence in a single pass, as described in detail below. However, refer to Figure 5A For the sake of simplicity, the following describes the processing of a target pair.

[0123] When a text block (e.g., input sentence 210) is input to NER task 122, NER task 122 processes the sentence and outputs the annotated sentence 510.

[0124] The first target span of the tagged sentence 510, such as "The patient," is marked by the tag at the beginning of the first target span. <s:e1> and the mark at the end of the first target span< / s:e1> Instructions. In the example, the first target span can be the body span. The second target span, tagged sentence 510, such as "telmisartan," is indicated by the tag at the beginning of the second target span. <s:e2> Mark at the end of the second target span< / s:e2> Instructions. The second target span can be an object span. Between the first and second target spans, there exists an intermediate context span, "is advised".

[0125] The annotated sentence 510 is passed to lexer 123. Similar to what is described above, lexer 123 is configured to convert words in each span into lexical units. Figure 5A In the middle, string Corresponding to the first target span s1 ,string Corresponding to the intermediate context span "is advised", and the string Corresponding to the second target span s2 As mentioned above, although the relation extraction system 500 can handle all target spans existing in a text block, due to limitations in... Figure 5A For the sake of simplicity in describing the embodiments, the processing is described below with respect to a first target span, a second target span, and a first intermediate context span arranged between the first target span (e.g., subject span) and the second target span (e.g., object span). Accordingly, no references are made. Figure 5A Describe the processing corresponding to the third target span and the second intermediate context span.

[0126] In the embodiment, with Figure 3 and Figure 4 In contrast, no special lexical "CLS" is generated. Instead, the lexer 123 generates an intermediate lexical representation 512 of the entity type of the target span pair in the same latent space used to generate the embeddings of the span lexical.

[0127]

[0128] The intermediate lexical representation 512 of the entity type of the lexical span and target span pair output by the lexer 123 can be passed to the pre-trained language model 127. The pre-trained language model 127 can capture the first entity based on special lexical representations inserted at the beginning and end of the first and second target spans. e1 Second Entity e2 Location information.

[0129] The output of the pre-trained language model 127 is a lexical embedding associated with words in the first target span, the first intermediate context span, the second target span, and entity types for the first and second target spans. For example... Figure 5A As shown, the lexical units of the first intermediate context span are represented as Based on this, an intermediate context representation 527 is generated. The generation of the intermediate context representation 527 is described in detail below.

[0130] The lexical embeddings from the pre-trained language model 127 are passed to the soft attention layer 520, where the entity type of the entity spans the first target is determined.e1 The corresponding entity-specific information and the information appearing in the first target span s The lexical embeddings corresponding to the lexical in 1 Integration, such as in Figure 5A The expression in the middle As shown. Entity type of the entity spanning the second target. e2 The corresponding entity-specific information and the information appearing in the second target span s The lexical embeddings corresponding to the lexical in 2 Integration, such as in Figure 5A The expression in the middle As shown.

[0131] The output of the soft attention layer 520 is entity-centric span embedding representations 522 and 523 corresponding to the first target span and the second target span:

[0132] The following describes the generation of entity-centric span embedding representations 522 and 523.

[0133] Generation of intermediate context representation

[0134] As mentioned above, the output of the pre-trained language model 127 can target the first target span. s 1 (e.g., main span) is represented as And for the second target span s 2 (e.g., object span) is represented as .

[0135] The entire text can be represented using lexical embeddings as follows:

[0136] in Corresponding to the main body span s 1 and object span s The intermediate context span between the lexical units of 2 m Embedding of each word element.

[0137] The intermediate context representation 527 can be generated as the average embedding representation of the lexical representation of the intermediate context:

[0138] Where p is the number of intermediate context words that appear between the subject span and object span of the target span pair.

[0139] In this paper, the intermediate context representation 527 can also be referred to as intermediate context computation.

[0140] Generation of entity-centric span embedding representations

[0141] To generate an entity-centric span embedding representation of the target span 522, 523, entity-specific information (e.g., entity type) is integrated into the lexical embedding of the target span.

[0142] To compute the entity-centric span embedding representation corresponding to each target span, three components are considered according to the embodiment: (1) Entity-independent spanning lexical embeddings, (2) Entity embedding aligned with span, and (3) Entity representation independent of span.

[0143] (1) Entity-independent span lexical embedding

[0144] For entity-independent span lexical embeddings, span s i The lexical embedding of each lexical in the model is maintained as, for example, by the output of a pre-trained language model 127.

[0145] (2) Entity embedding aligned with span

[0146] For entity embeddings aligned with the span, a fixed-length representation of the entity is used based on soft alignment with the span lexical units. Alignment is computed via neural attention, for which attention scores are calculated on the dot product between nonlinear mappings of lexical embeddings (e.g., soft attention).

[0147] Suppose the lexical representation of the span is as follows .

[0148] Entity type e The lexical can be represented as an embedded lexical sequence For example, the sequence can be obtained using a pre-trained language model 127 by using the intermediate lexical representations 512 of the entity types of the target span.

[0149] Then, the lexical units of the target span are calculated using the following equation. With the lexical q appearing in the entity type j Attention scores between: For each :

[0150] Importance score of normalization Used to weight the representations of entity type lexics, and to obtain for each lexicographical element in the target span n A single attention vector for each lexical unit to obtain entity embeddings aligned with the span:

[0151] (3) Entity representation independent of span

[0152] For span-independent entity representations, representations of entity types that do not depend on the target span are included. Similar to the case of span-aligned entity embeddings, attention scores are calculated using dot products, except that entity type lexics are compared with general learned embeddings rather than any specific lexics within the target span.

[0153] Additionally, contextual information generated by a feedforward neural network (FFN) is incorporated before aggregating the output using an attention mechanism. This aims to generate coarse-grained summaries of entities that depend on word order.

[0154] Assuming entity type lexical The FFN network on the input produces the embedding. .

[0155] Apply a linear layer to each output term to obtain: .

[0156] In one embodiment, the FFN and linear layers used to compute span-independent entity representations can be incorporated into the soft attention layer 520. However, this is not intended to be limiting. For example, in some implementations, the FFN and linear layers used to compute span-independent entity representations may differ from the soft attention layer 520.

[0157] Then:

[0158] Finally, the entity independent of the span is represented as .

[0159] and h i (It is generated by the pre-trained language model 127) The corresponding complete entity-centric span embedding representation is given as

[0160] The entity-centric span embeddings 522 and 523 are averaged across all representative lexical units in the target span to obtain the entity-centric representation of the target span:

[0161] (Assuming the target span is from) i arrive j (Word element distribution) (Assuming the target span is from) k arrivel (Word element distribution) A fully connected layer with an activation function (see figure 526) applies the activation function to the intermediate context representation 527 of the target span and the entity-centric span embedding representations 524 and 525 to obtain:

[0162] Cascade representation is generated as .

[0163] Then, Set as And it is passed to the feedforward network (FFN) of the fully connected layer 530 and the linear layer 528 to achieve the desired connection span. s 1 and s 2 Specific Relationship r ,get .

[0164] Although Figure 5A The diagram illustrates three fully connected layers with activation functions (Figure 526), ​​which correspond to... and However, this is not intended to be limiting. In some implementations, different numbers of fully connected layers with activation functions can be used, for example, 1, 2, ..., n.

[0165] As mentioned above, although Figure 3 and Figure 4 The BERT model 327 of R-BERT generates lexical units of intermediate context, but does not perform further computation on these lexical units, and the information about the intermediate context is not passed to the fully connected layer with activation function and the fully connected layer at the output of the RE task.

[0166] The softmax layer 532 generates a probability distribution for each relation of each target pair in the input text:

[0167] Where R is the set of all relation types.

[0168] Accordingly, for an input text block, after calculating the probability distribution of the relation connecting each target span pair, if the probability of the relation is the maximum among all relations including empty relations, then the given relation of the target span pair is predicted based on the probability distribution.

[0169] However, the above description is not intended to be limiting. In some implementations, it may be determined that certain target span pairs can only share one possible relation type from the set 114 of relation types.

[0170] The probability that target span pairs share only one possible relation is calculated as follows: ,in It is the sigmoid function. If the probability exceeds a threshold (e.g., 0.55), then the candidate relation type is determined to be a relation connecting the subject span and object span of the target span pair under consideration. The value of the sigmoid function is between 0 and 1, and the threshold can be adjusted.

[0171] In one embodiment, the sigmoid layer 534 can be incorporated into the softmax layer 532, such as Figure 5A As shown in the diagram. However, this is not intended to be limiting. In some implementations, the sigmoid layer 534 may be separate from the softmax layer 532.

[0172] If the relationship prediction problem is modeled as a multi-class classification problem, then the softmax values ​​are used as the probability distribution of the relationships. If the relationship prediction problem is modeled as a binary classification problem (e.g., where only a relationship can exist between two entities), then the sigmoid function is used to compute the probability distribution of the relationships. In such scenarios, a sigmoid layer of 534 can be used instead of a softmax layer of 532.

[0173] Training phase and reference above Figure 5A The described technique is similar. The loss function is defined as follows: It is maximized. A loss function is computed for each relation of each target pair in the input text.

[0174] Unlike related art techniques, as described above, in this embodiment, entity-type-specific information is embedded, for example via entity embedding aligned with the span. and span-independent entity representation It is passed to the fully connected layer with the activation function (Figure 526). Similarly, the intermediate context is computed. h mid The results are passed to the fully connected layer with the activation function (see figure 526). Then, as described above, the results of these calculations are passed to the fully connected layer 530 and the softmax layer 532.

[0175] In an embodiment, Figure 1 The modified RE task 130 may be at least a part of the modified R-BERT model 502. For example, the modified RE task 130 may include a soft attention layer 520, a fully connected layer with an activation function (reference numeral 526), ​​a fully connected layer 530, and a softmax layer 532.

[0176] In this embodiment, NER task 122, pre-trained language model 127, and modified RE task 130 may be incorporated into the pre-trained ML model. However, this is not intended to be limiting. For example, in some implementations, NER task 122 and pre-trained language model 127 may be separate models networked with modified RE task 130.

[0177] D. Using the modified R-BERT for batch processing relation extraction

[0178] Figure 5B Use according to at least one embodiment Figure 5A A simplified block diagram of the modified R-BERT used in the batch processing relation extraction system 500. The processing performed by the modified R-BERT is referenced above. Figure 5A The above has already been described, so some detailed descriptions will not be repeated here.

[0179] exist Figure 5B In the text, the annotated sentence 550 indicates the output of NER task 122, similar to... Figures 3 to 5A The example shown. The first target span of sentence 550, such as "The patient," is marked by the marker at the beginning of the first target span. <s:e1> and the mark at the end of the first target span< / s:e1> Indicator. The second target span of the tagged sentence 550, such as "telmisartan", is indicated by the marker at the beginning of the second target span. <s:e2> Mark at the end of the second target span< / s:e2> Indicator. The third target span of the tagged sentence 550, such as "hypertension", is indicated by the marker at the beginning of the third target span. <s:e3> and the mark at the end of the third target span< / s:e3> Instructions. Between the target span, there are two intermediate context spans: "is advised" and "for".

[0180] The annotated sentence 550 is passed to lexicon 123. Lexicon 123 is configured to convert each word in the span into a lexicon. As shown in the figure, the string... Corresponding to the first target span s1 ,string Corresponding to the first intermediate context span "is advised", the string Corresponding to the second target span s2 ,string Corresponding to the second intermediate context span "for", and the string Corresponding to the third target span s3 .

[0181] Similar to Figure 4 Special word character " "Inserted at the beginning and end of the first target span, corresponding to the mark" <s:e1> and< / s:e1>Special word character' 'Inserted at the beginning and end of the second target span, corresponding to the mark' <s:e2> and< / s:e2> And special word character " "Inserted at the beginning and end of the third target span, corresponding to the mark." <s:e3> and< / s:e3> .

[0182] As described above, in the embodiments, with Figure 3 and Figure 4 In contrast, no special lexical "CLS" is generated. Instead, the lexer 123 generates an intermediate lexical representation 512 of the entity type of the target span in the same latent space used to generate the embeddings of the span lexical.

[0183]

[0184] The intermediate lexical representations 512 of the entity types for the lexical span and target span output by the lexer 123 can be passed to the pre-trained language model 127. The pre-trained language model 127 can capture entity types based on special lexical representations inserted at the beginning and end of the first, second, and third target spans. e1 , e2 and e3 The location information of the corresponding first, second, and third entities.

[0185] The output of the pre-trained language model 127 is a lexical embedding associated with words in the first target span, the first intermediate context span, the second target span, the second intermediate context span, the third target span, and the entity types in the first, second, and third target spans.

[0186] The lexical embeddings from the pre-trained language model 127 are passed to the soft attention layer 520, where the entity type of the entity spans the first target is determined. e1 The corresponding entity-specific information and the information appearing in the first target span s The lexical embeddings corresponding to the lexical in 1 Integration, such as in Figure 5B The expression in the middle As shown. Entity type of the entity spanning the second target. e2 The corresponding entity-specific information and the information appearing in the second target span s The lexical embeddings corresponding to the lexical in 2 Integration, such as in Figure 5B The expression in the middle As shown. Entity type of the entity spanning the third target. e3 The corresponding entity-specific information and the information appearing in the second target span s Lexical embeddings corresponding to lexical elements in 3 Integration, such as in Figure 5B The expression in the middle As shown.

[0187] The output of the soft attention layer 520 is entity-centric span embedding representations 522, 523, and 554 generated corresponding to the first target span, the second target span, and the third target span:

[0188] The above reference Figure 5A Techniques for generating entity-centric span embedding representations 522, 523, and 554 are described.

[0189] The entity-centric span embedding representations 522, 523, and 554 are averaged across all representative lexical units in the target span to obtain the entity-centric representation of the target span: (Assuming the target span is from) i arrive j (Word element distribution) (Assuming the target span is from) k arrive l (Word element distribution) (Assuming the target span is from) m arrive n (Word element distribution) Similar to the reference above Figure 5A The content described, intermediate contexts 527 and 556 are generated based on the first intermediate context span and the second intermediate context span, respectively.

[0190] The fully connected layers with activation functions (see figures 526, 558) apply the activation functions to the intermediate context representations 527, 556 of the target span and the entity-centric span embedding representations 524, 525, 554 to obtain:

[0191] Cascade representation is generated as .

[0192] The cascaded representation is passed to fully connected layers 530 and 560 and softmax layers 532 and 562 to obtain the target span of the connections, respectively. s 1 and s 2. and target span s 2 and s The probability of a specific relationship of 3. These operations have been described in detail above and will not be repeated.

[0193] As described above, the relation extraction system 500 can perform operations on target span pairs. and The processing of [the data / processes] is described above. However, the above description is not intended to be limiting. That is, because the relation determination system 100 is able to determine all relations of the input text block in a single pass, the relation extraction system 500 can also process the target span pairs. Execute the processing. In this case, it may be considered to include it in the first target span. s 1 and the third target span s The intermediate context words in the intermediate context span between 3. Similarly, the relation extraction system 500 can perform processing on all target span pairs present in the text block into which it is input.

[0194] As described above, target spans form pairs, where one target span in the pair is the subject span, and the other target span in the pair is the object span. A target span that is the subject span in one pair can be regarded as the object span in another target pair. However, target spans within the same pair cannot be flipped.

[0195] IV. Shared Layer Architecture

[0196] Figure 6 This is a simplified block diagram of a shared layer architecture for joint NER and RE tasks according to various embodiments.

[0197] For example, ML model 600 can be generated by jointly training the NER and RE models based on a modified R-BERT. In the shared layers, the first few layers can be used by the NER model, and the remaining layers can be shared by the RE model. The output embeddings from the NER can be used as input to the RE model. A few layers in between can be shared by both the NER and RE models, meaning that the parameters of these layers will be updated during training for both tasks.

[0198] In a shared model architecture used for multiple tasks including NER and RE, NER can produce contextualized outputs for the input text fragments that must be fed to RE. In the example above, for R-BERT, entity tags are provided before and after the target span. However, for the contextualized representation of the input text for RE, it may be difficult to insert entity tags in appropriate places, but information can be embedded using the positional embeddings of the start and end lemmas of the span in question. This would require decoupling the use of entity tags from the input text representation used for RE. Moreover, explicitly representing entities (in terms of constituent lemma embeddings) will add more relevant information to RE predictions.

[0199] V. Use of the Relationship Determination System

[0200] like Figure 13 As shown, the relationship determination system 100 can be provided as part of a distributed computing environment, wherein the relationship determination system 100 is connected to one or more user computers 1326 via a communication network 1328. An example of a distributed computing environment is... Figure 12 It is described in the middle and in detail below.

[0201] like Figure 14 As shown, the relationship determination system 100 may be part of a CSP infrastructure 1429 provided by a CSP for providing one or more cloud services. For example, the one or more cloud services may include ABC cloud service 1431 to XYZ cloud service 1434 connected to the computers of one or more users 1435 via a communication network 1437. For example, the relationship determination system 100 may be part of ABC cloud service 1431.

[0202] Examples of cloud infrastructure architectures provided by CSPs are in Figures 8 to 11 It is described in the middle and in detail below.

[0203] VI. Methods

[0204] Figure 7 This is a simplified block diagram of a process 700 performed by a relation determination system 100 according to various embodiments. Process 700 may be performed by a span set preparation subsystem 102 and a relation extraction subsystem 104.

[0205] Figure 7 The process 700 described herein can be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding subsystem, or using hardware or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 7 The methods presented and described below are intended to be illustrative and non-limiting. Although Figure 7 Various processing operations that occur in a specific sequence or order are described, but this is not intended to be limiting. In some alternative embodiments, processing 700 may be performed in a different order, or some operations may be performed at least partially in parallel.

[0206] refer to Figure 7At operation 702, the relation determination system 100 can access text containing a sequence of words, wherein multiple spans are identified within the text, each of the multiple spans comprising one or more words in the word sequence, and the multiple spans comprising one or more target span pairs and one or more intermediate context spans, each of the one or more target span pairs being associated with an entity type, each pair comprising a subject span and an object span, and each of the one or more intermediate context spans appearing between the subject span and object span contained in each of the one or more target span pairs.

[0207] At operation 704, the relation determination system 100 can use a pre-trained ML model to generate embedding representations of lexical units associated with one or more words appearing in each target span, embedding representations of lexical units associated with entity types in each target span, and embedding representations of lexical units associated with one or more intermediate context words appearing in one or more intermediate context spans.

[0208] At operation 706, the relation determination system 100 can use a pre-trained ML model to generate an entity-centric span embedding representation for each target span based on the embedding representation of the lexical associated with each target span and the embedding representation of the lexical associated with the entity type of that target span.

[0209] In various embodiments, the entity-centric span embedding representation is generated by integrating entity-type-specific information with the embedding representation of lemmas associated with each target span using entity-independent span lemma embeddings, span-aligned entity embeddings, and span-independent entity representations. The entity-independent span lemma embeddings maintain the embedding of each lemma in the target span from the embedding representation of lemmas, the span-aligned entity embeddings contain a fixed-length representation of the entity type based on soft alignment with the lemmas of the target span, and the span-independent entity representations contain a representation of the entity type of the lemmas that does not depend on the target span.

[0210] At operation 708, the relation determination system 100 can use a pre-trained ML model to generate an intermediate context embedding representation for each intermediate context span in one or more intermediate context spans based on the embedding representation of the lexical associated with each intermediate context span.

[0211] In various embodiments, the method further includes applying activation functions by a fully connected layer of a pre-trained ML model to entity-centric span embedding representations of the subject span and object span, and to intermediate context embedding representations of intermediate context spans appearing between the subject span and object span for each target span pair, to obtain activated entity-centric span embedding representations of the subject span and object span for each target span pair and activated intermediate context embedding representations.

[0212] In various embodiments, the method further includes, for each target span pair, generating a concatenation of activated entity-centric span embedding representations and activated intermediate context embedding representations of the subject span and object span associated with each target span pair by a fully connected layer.

[0213] At operation 710, the relation determination system 100 can use a pre-trained ML model to generate a probability distribution of each relation in a relation set containing one or more target span pairs based on entity-centric span embedding representations of the subject span and object span contained in each pair, and intermediate context embedding representations of the intermediate context spans appearing between the subject span and object span in that pair. This relation set contains empty relations.

[0214] The probability distribution for each relation of each target span pair can be generated based on a concatenation of entity-centric span embedding representations and activated intermediate context embedding representations of the subject span and object span associated with each target span pair.

[0215] In various embodiments, the probability distribution generated for each relation of each target span pair takes into account entity-type-specific information integrated into the entity-centric span embedding representation of the activation of the subject span and object span contained in each target span pair.

[0216] At operation 712, the relation determination system 100 can use a pre-trained ML model to predict a given relation of target span pairs based on a probability distribution.

[0217] Example cloud implementation

[0218] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, and clustering software, etc.). Therefore, because these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0219] In some instances, IaaS customers can access resources and services over a wide area network (WAN) such as the internet, and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0220] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0221] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the handling of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for handling the OS, middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0222] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0223] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some instances, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0224] In some examples, the infrastructure can have many interconnected elements. For example, one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network, may exist. In some examples, one or more inbound / outbound traffic group rules may also be provided to define how inbound / outbound traffic to the network and one or more virtual machines (VMs) will be configured. Other infrastructure elements, such as load balancers, databases, etc., may also be provided. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are desired.

[0225] In some instances, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some instances, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0226] Figure 8This is a block diagram 800 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 802 may be communicatively coupled to a secure host lease 804, which may include a virtual cloud network (VCN) 806 and a secure host subnet 808. In some examples, service operator 802 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. Client computing devices can be workstation computers running a variety of commercially available UNIX® or UNIX-like operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, client computing devices can be any other electronic device, such as thin client computers, internet-enabled gaming systems (e.g., Microsoft Xbox game consoles with or without Kinect® gesture input devices), and / or personal messaging devices capable of communicating over a network that can access VCN 806 and / or the internet.

[0227] VCN 806 may include a local peering gateway (LPG) 810, which may be communicatively coupled to a secure shell (SSH) VCN 812 via an LPG 810 contained in an SSH VCN 812. SSH VCN 812 may include an SSH subnet 814, and SSH VCN 812 may be communicatively coupled to a control plane VCN 816 via an LPG 810 contained in a control plane VCN 816. Furthermore, SSH VCN 812 may be communicatively coupled to a data plane VCN 818 via an LPG 810. Control plane VCN 816 and data plane VCN 818 may be contained within a service lease 819 that may be owned and / or operated by an IaaS provider.

[0228] The control plane VCN 816 may include a control plane demilitarized zone (DMZ) layer 820 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 820 may include one or more load balancer (LB) subnets 822, a control plane application layer 824 that may include one or more application subnets 826, and a control plane data layer 828 that may include one or more database (DB) subnets 830 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 822 contained in the control plane DMZ layer 820 may be communicatively coupled to one or more application subnets 826 contained in the control plane application layer 824 and an Internet gateway 834 that may be contained in the control plane VCN 816. The application subnets 826 may be communicatively coupled to one or more DB subnets 830 contained in the control plane data layer 828, as well as a service gateway 836 and a Network Address Translation (NAT) gateway 838. The control plane VCN 816 may include the service gateway 836 and the NAT gateway 838.

[0229] The control plane VCN 816 may include a data plane mirror application layer 840, which may include one or more application subnets 826. The one or more application subnets 826 included in the data plane mirror application layer 840 may include a virtual network interface controller (VNIC) 842 capable of executing a compute instance 844. The compute instance 844 may communicatively couple the one or more application subnets 826 of the data plane mirror application layer 840 to the one or more application subnets 826 that may be included in the data plane application layer 846.

[0230] Data plane VCN 818 may include data plane application layer 846, data plane DMZ layer 848, and data plane data layer 850. Data plane DMZ layer 848 may include one or more LB subnets 822 communicatively coupled to one or more application subnets 826 of data plane application layer 846 and Internet gateway 834 of data plane VCN 818. One or more application subnets 826 may be communicatively coupled to service gateway 836 and NAT gateway 838 of data plane VCN 818. Data plane data layer 850 may also include one or more DB subnets 830 communicatively coupled to one or more application subnets 826 of data plane application layer 846.

[0231] The Internet gateway 834 of the control plane VCN 816 and data plane VCN 818 can be communicatively coupled to the metadata management service 852, which in turn can be communicatively coupled to the public Internet 854. The public Internet 854 can be communicatively coupled to the NAT gateway 838 of the control plane VCN 816 and data plane VCN 818. The service gateway 836 of the control plane VCN 816 and data plane VCN 818 can be communicatively coupled to the cloud service 856.

[0232] In some examples, the service gateway 836 of the control plane VCN 816 or data plane VCN 818 can make application programming interface (API) calls to the cloud service 856 without traversing the public internet 854. API calls from the service gateway 836 to the cloud service 856 can be unidirectional: the service gateway 836 can make API calls to the cloud service 856, and the cloud service 856 can send requested data to the service gateway 836. However, the cloud service 856 may not initiate API calls to the service gateway 836.

[0233] In some examples, secure host lease 804 can be directly connected to service lease 819, which would otherwise be isolated. Secure host subnet 808 can communicate with SSH subnet 814 via LPG 810, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 808 to SSH subnet 814 allows secure host subnet 808 to access other entities within service lease 819.

[0234] Control plane VCN 816 allows users of service lease 819 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 816 can be deployed or otherwise used in data plane VCN 818. In some examples, control plane VCN 816 can be isolated from data plane VCN 818, and the data plane mirror application layer 840 of control plane VCN 816 can communicate with the data plane application layer 846 of data plane VCN 818 via VNIC 842, which can be included in both the data plane mirror application layer 840 and the data plane application layer 846.

[0235] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 854, which can transmit requests to the metadata management service 852. The metadata management service 852 can transmit the request to the control plane VCN 816 via internet gateway 834. The request can be received by one or more LB subnets 822 contained in the control plane DMZ layer 820. The LB subnets 822 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 826 contained in the control plane application layer 824. If the request is validated and requires a call to the public internet 854, the call to the public internet 854 can be transmitted to a NAT gateway 838 that can make calls to the public internet 854. The request may expect the stored metadata to be stored in one or more DB subnets 830.

[0236] In some examples, the data plane mirroring application layer 840 can facilitate direct communication between the control plane VCN 816 and the data plane VCN 818. For example, it might be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 818. Through VNIC 842, the control plane VCN 816 can communicate directly with the resources contained in the data plane VCN 818, and thus can perform configuration changes, updates, or other appropriate modifications.

[0237] In some embodiments, the control plane VCN 816 and data plane VCN 818 may be included in a service lease 819. In this case, the system's users or customers may not own or operate the control plane VCN 816 or data plane VCN 818. Alternatively, the IaaS provider may own or operate both the control plane VCN 816 and data plane VCN 818, and both planes may be included in the service lease 819. This embodiment can enable the isolation of networks that might prevent users or customers from interacting with resources from other users or customers. Furthermore, this embodiment can allow users or customers of the system to privately store databases without relying on the public Internet 854, which may not have the desired level of threat prevention for storage.

[0238] In other embodiments, one or more LB subnets 822 included in the control plane VCN 816 may be configured to receive signals from the service gateway 836. In this embodiment, the control plane VCN 816 and the data plane VCN 818 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 854. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 819, which can be isolated from the public internet 854.

[0239] Figure 9 This is a block diagram 900 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 902 (e.g., Figure 8 The service provider (802) can communicatively couple to the secure host lease (904) (e.g., Figure 8 The secure hosting lease 804), the secure hosting lease 904 may include a virtual cloud network (VCN) 906 (e.g., Figure 8 VCN806) and Secure Host Subnet 908 (e.g., Figure 8 The secure host subnet 808). VCN 906 may include a local peering gateway (LPG) 910 (e.g., Figure 8 The LPG 810), which can be communicatively coupled to the Secure Shell (SSH) VCN 912 (e.g., via the LPG 810 contained in the SSH VCN 912) Figure 8 SSH VCN 812). SSH VCN 912 can include SSH subnet 914 (e.g., Figure 8 The SSH subnet 814), and the SSH VCN 912 can be communicatively coupled to the control plane VCN 916 via the LPG 910 contained in the control plane VCN 916 (e.g., Figure 8 Control plane VCN 816). Control plane VCN 916 may be included in service lease 919 (e.g., Figure 8 In the service lease 819), and the data plane VCN 918 (e.g., Figure 8 The data plane VCN 818 can be included in a customer lease 921 that can be owned or operated by the system's users or customers.

[0240] The control plane VCN 916 may include one or more LB subnets 922 (e.g., Figure 8 The control plane DMZ layer 920 of (one or more) LB subnets 822) (e.g., Figure 8The control plane DMZ layer 820), may contain one or more application subnets 926 (e.g., Figure 8 The control plane application layer 924 of (one or more) application subnets 826 (e.g., Figure 8 The control plane application layer 824), and may contain one or more database (DB) subnets 930 (e.g., similar to...). Figure 8 The control plane data layer 928 of (one or more) DB subnets 830 (e.g., Figure 8 The control plane data layer 828). One or more LB subnets 922 contained in the control plane DMZ layer 920 can be communicatively coupled to one or more application subnets 926 contained in the control plane application layer 924 and an Internet gateway 934 that can be contained in the control plane VCN 916 (e.g., Figure 8 Internet gateway 834), and application subnet(s) 926 can communicatively couple to DB subnet(s) 930 contained in control plane data layer 928 and service gateway 936 (e.g., Figure 8 Service gateway 836) and Network Address Translation (NAT) gateway 938 (e.g., Figure 8 (NAT gateway 838). The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0241] The control plane VCN 916 may include a data plane mirror application layer 940 that may contain one or more application subnets 926 (e.g., Figure 8 The data plane mirror application layer 840). One or more application subnets 926 contained in the data plane mirror application layer 940 may include computational instances 944 (e.g., similar to...). Figure 8 The virtual network interface controller (VNIC) 942 (e.g., the VNIC of 842) of the computing instance 844. The computing instance 944 may facilitate the mirroring of the application layer 940 with one or more application subnets 926 and may be included in the application layer 946 of the data plane (e.g., Figure 8 Communication between one or more application subnets 926 in the data plane application layer 846 via VNIC 942 contained in the data plane mirror application layer 940 and VNIC 942 contained in the data plane application layer 946.

[0242] The Internet gateway 934, included in the control plane VCN 916, can be communicatively coupled to the metadata management service 952 (e.g., Figure 8 Metadata management service 852), which can communicatively couple to the public Internet 954 (e.g., Figure 8The public internet 954 can communicatively couple to a NAT gateway 938 contained in the control plane VCN 916. The service gateway 936 contained in the control plane VCN 916 can communicatively couple to a cloud service 956 (e.g., ...). Figure 8 Cloud services (856).

[0243] In some examples, data plane VCN 918 may be included in customer lease 921. In this case, the IaaS provider may provide control plane VCN 916 for each customer, and the IaaS provider may set up a unique compute instance 944 for each customer, included in service lease 919. Each compute instance 944 may allow communication between control plane VCN 916 included in service lease 919 and data plane VCN 918 included in customer lease 921. Compute instance 944 may allow resources provisioned in control plane VCN 916 included in service lease 919 to be deployed or otherwise used in data plane VCN 918 included in customer lease 921.

[0244] In other examples, an IaaS provider's customer may have a database residing in customer lease 921. In this example, control plane VCN 916 may include data plane mirror application layer 940, which may include one or more application subnets 926. Data plane mirror application layer 940 may reside in data plane VCN 918, but may not reside in data plane VCN 918. That is, data plane mirror application layer 940 may have access to customer lease 921, but may not reside in data plane VCN 918 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 940 may be configured to invoke data plane VCN 918, but may not be configured to invoke any entity contained in control plane VCN 916. Customers may expect to deploy or otherwise use resources provided in the control plane VCN 916 in the data plane VCN 918, and the data plane mirroring application layer 940 can facilitate the customer's desired deployment or other use of resources.

[0245] In some embodiments, an IaaS provider's customer can apply filters to data plane VCN 918. In this embodiment, the customer can determine what data plane VCN 918 can access, and the customer can restrict access from data plane VCN 918 to the public Internet 954. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 918's access to any external networks or databases. Applying filters and controls to data plane VCN 918 contained in customer lease 921 helps isolate data plane VCN 918 from other customers and the public Internet 954.

[0246] In some embodiments, cloud service 956 may be invoked by service gateway 936 to access services that may not exist on public internet 954, control plane VCN 916, or data plane VCN 918. The connection between cloud service 956 and control plane VCN 916 or data plane VCN 918 may not be real-time or continuous. Cloud service 956 may reside on different networks owned or operated by an IaaS provider. Cloud service 956 may be configured to receive calls from service gateway 936 and may be configured not to receive calls from public internet 954. Some cloud services 956 may be isolated from other cloud services 956, and control plane VCN 916 may be isolated from cloud services 956 that may not be in the same region as control plane VCN 916. For example, control plane VCN 916 may be located in "Region 1," and cloud service "Deployment 8" may be located in both "Region 1" and "Region 2." If the service gateway 936, contained in the control plane VCN 916 located in region 1, makes a call to deployment 8, then that call can be transmitted to deployment 8 in region 1. In this example, the control plane VCN 916 or deployment 8 in region 1 may be uncoupled from or otherwise communicate with deployment 8 in region 2.

[0247] Figure 10 This is a block diagram 1000 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1002 (e.g., Figure 8 The service provider 802) can communicatively couple to the secure host lease 1004 (e.g., Figure 8 Secure hosting lease 804), the secure hosting lease 1004 may include a virtual cloud network (VCN) 1006 (e.g., Figure 8 VCN 806) and Secure Host Subnet 1008 (e.g., Figure 8 The secure host subnet 808). VCN 1006 may include LPG1010 (e.g., Figure 8The LPG 810), which can be communicatively coupled to the SSH VCN 1012 via the LPG 1010 included in the SSH VCN 1012 (e.g., Figure 8 SSH VCN 812). SSH VCN 1012 can include SSH subnet 1014 (e.g., Figure 8 SSH subnet 814), and SSH VCN 1012 can be communicatively coupled to control plane VCN 1016 via LPG 1010 included in control plane VCN 1016 (e.g., Figure 8 The control plane VCN 816) and coupled to the data plane VCN 1018 via the LPG 1010 contained in the data plane VCN 1018 (e.g., Figure 8 Data plane 818). Control plane VCN 1016 and data plane VCN 1018 can be included in service lease 1019 (e.g., Figure 8 In the service rental (819).

[0248] The control plane VCN 1016 may include one or more load balancer (LB) subnets 1022 (e.g., Figure 8 The control plane DMZ layer 1020 of (one or more) LB subnets 822) (e.g., Figure 8 The control plane DMZ layer 820 may contain one or more application subnets 1026 (e.g., similar to...). Figure 8 The control plane application layer 1024 of (one or more) application subnets 826 (e.g., Figure 8 The control plane application layer 824), and the control plane data layer 1028 (e.g., may contain one or more DB subnets 1030) Figure 8 The control plane data layer 828). One or more LB subnets 1022 contained in the control plane DMZ layer 1020 can be communicatively coupled to one or more application subnets 1026 contained in the control plane application layer 1024 and an Internet gateway 1034 that can be contained in the control plane VCN 1016 (e.g., Figure 8 Internet gateway 834), and application subnet(s) 1026 can communicatively couple to DB subnet(s) 1030 contained in control plane data layer 1028 and service gateway 1036 (e.g., Figure 8 The service gateway) and Network Address Translation (NAT) gateway 1038 (e.g., Figure 8 (NAT gateway 838). The control plane VCN 1016 may include the service gateway 1036 and the NAT gateway 1038.

[0249] Data plane VCN 1018 may include data plane application layer 1046 (e.g., Figure 8 Data plane application layer 846), data plane DMZ layer 1048 (e.g., Figure 8 Data plane DMZ layer 848), and data plane data layer 1050 (e.g., Figure 8 The data plane data layer 850). The data plane DMZ layer 1048 may include one or more trusted application subnets 1060 and one or more untrusted application subnets 1062 that can be communicatively coupled to the data plane application layer 1046, and one or more LB subnets 1022 of the Internet gateway 1034 contained in the data plane VCN 1018. The one or more trusted application subnets 1060 may be communicatively coupled to the service gateway 1036 contained in the data plane VCN 1018, the NAT gateway 1038 contained in the data plane VCN 1018, and one or more DB subnets 1030 contained in the data plane data layer 1050. The one or more untrusted application subnets 1062 may be communicatively coupled to the service gateway 1036 contained in the data plane VCN 1018 and the one or more DB subnets 1030 contained in the data plane data layer 1050. The data plane data layer 1050 may include one or more DB subnets 1030 that can be communicatively coupled to the service gateway 1036 contained in the data plane VCN 1018.

[0250] One or more untrusted application subnets 1062 may include one or more primary VNICs 1064(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1066(1)-(N). Each tenant VM 1066(1)-(N) may be communicatively coupled to a corresponding application subnet 1067(1)-(N) that may be contained in a corresponding container egress VCN 1068(1)-(N), which may be contained in a corresponding customer lease 1070(1)-(N). A corresponding secondary VNIC 1072(1)-(N) may facilitate communication between one or more untrusted application subnets 1062 contained in data plane VCN 1018 and application subnets contained in container egress VCN 1068(1)-(N). Each container exit VCN 1068(1)-(N) may include a NAT gateway 1038, which can communicatively couple to the public Internet 1054 (e.g., Figure 8 The public internet (854).

[0251] The Internet gateway 1034, contained in the control plane VCN 1016 and the data plane VCN 1018, can be communicatively coupled to the metadata management service 1052 (e.g., Figure 8 Metadata management service 1052 can be communicatively coupled to the public internet 1054. The public internet 1054 can be communicatively coupled to a NAT gateway 1038 contained in a control plane VCN 1016 and a data plane VCN 1018. Service gateway 1036 contained in control plane VCN 1016 and data plane VCN 1018 can be communicatively coupled to a cloud service 1056.

[0252] In some embodiments, the data plane VCN 1018 may be integrated with the customer lease 1070. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer to the IaaS provider.

[0253] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1046. The code running this functionality may execute in VMs 1066(1)-(N) and may not be configured to run anywhere else on the data plane VCN 1018. Each VM 1066(1)-(N) may be connected to a customer lease 1070. A corresponding container 1071(1)-(N) contained in VMs 1066(1)-(N) may be configured to run the code. In this case, dual isolation may exist (e.g., container 1071(1)-(N) runs the code, where container 1071(1)-(N) may be contained in at least one VM 1066(1)-(N) contained in untrusted application subnet 1062), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1071(1)-(N) may be communicatively coupled to customer lease 1070 and may be configured to transmit or receive data from customer lease 1070. Containers 1071(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 1018. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of containers 1071(1)-(N).

[0254] In some embodiments, one or more trusted application subnets 1060 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1060 may be communicatively coupled to one or more database subnets 1030 and configured to perform CRUD operations in one or more database subnets 1030. One or more untrusted application subnets 1062 may be communicatively coupled to one or more database subnets 1030, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1030. Containers 1071 (1)-(N) that may be contained in each customer's VM 1066 (1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1030.

[0255] In other embodiments, the control plane VCN 1016 and the data plane VCN 1018 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1016 and the data plane VCN 1018. However, communication may occur indirectly through at least one method. The LPG 1010 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1016 and the data plane VCN 1018. In another example, either the control plane VCN 1016 or the data plane VCN 1018 may invoke the cloud service 1056 via the service gateway 1036. For example, an invocation of the cloud service 1056 from the control plane VCN 1016 may include a request for a service that can communicate with the data plane VCN 1018.

[0256] Figure 11 This is a block diagram 1100 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1102 (e.g., Figure 8 The service provider 802) can communicatively couple to the secure host lease 1104 (e.g., Figure 8 Secure hosting lease 804), the secure hosting lease 1104 may include a virtual cloud network (VCN) 1106 (e.g., Figure 8 VCN 806) and Secure Host Subnet 1108 (e.g., Figure 8 The secure host subnet 808). VCN 1106 may include LPG1110 (e.g., Figure 8 The LPG 810), the LPG 1110 can be accessed via SSH VCN 1112 (e.g., LPG 810), Figure 8LPG 1110 in SSH VCN 1112 is communicatively coupled to SSH VCN 1112. SSH VCN 1112 may include SSH subnet 1114 (e.g., Figure 8 SSH subnet 814), and SSH VCN 1112 can be communicatively coupled to control plane VCN 1116 via LPG 1110 contained in control plane VCN 1116 (e.g., Figure 8 The control plane VCN 816) and coupled to the data plane VCN 1118 via the LPG 1110 contained in the data plane VCN 1118 (e.g., Figure 8 Data plane 818). Control plane VCN 1116 and data plane VCN 1118 can be included in service lease 1119 (e.g., Figure 8 In the service rental (819).

[0257] The control plane VCN 1116 may include one or more LB subnets 1122 (e.g., Figure 8 The control plane DMZ layer 1120 of (one or more) LB subnets 822) (e.g., Figure 8 The control plane DMZ layer 820), may contain one or more application subnets 1126 (e.g., Figure 8 The control plane application layer 1124 of (one or more) application subnets 826 (e.g., Figure 8 The control plane application layer 824) may contain one or more DB subnets 1130 (e.g., Figure 10 The control plane data layer 1128 of (one or more) DB subnets 1030 (e.g., Figure 8 The control plane data layer 828). One or more LB subnets 1122 contained in the control plane DMZ layer 1120 can be communicatively coupled to one or more application subnets 1126 contained in the control plane application layer 1124 and an Internet gateway 1134 that can be contained in the control plane VCN 1116 (e.g., Figure 8 Internet gateway 834), and application subnet(s) 1126 can communicatively couple to DB subnet(s) 1130 contained in control plane data layer 1128 and service gateway 1136 (e.g., Figure 8 The service gateway) and Network Address Translation (NAT) gateway 1138 (e.g., Figure 8 (NAT gateway 838). The control plane VCN 1116 may include the service gateway 1136 and the NAT gateway 1138.

[0258] Data plane VCN 1118 may include data plane application layer 1146 (e.g., Figure 8 Data plane application layer 846), data plane DMZ layer 1148 (e.g., Figure 8 Data plane DMZ layer 848), and data plane data layer 1150 (e.g., Figure 8 The data plane data layer 850). The data plane DMZ layer 1148 may include one or more trusted application subnets 1160 that can be communicatively coupled to the data plane application layer 1146 (e.g., Figure 10 (one or more) trusted application subnets 1060 and (one or more) untrusted application subnets 1162 (e.g., Figure 10 The data plane includes one or more untrusted application subnets 1062 and one or more LB subnets 1122 of Internet gateway 1134 contained in data plane VCN 1118. One or more trusted application subnets 1160 may communicatively couple to service gateway 1136, NAT gateway 1138, and DB subnets 1130 contained in data plane VCN 1118. One or more untrusted application subnets 1162 may communicatively couple to service gateway 1136 and DB subnets 1130 contained in data plane VCN 1118 and data plane data layer 1150, respectively. Data plane data layer 1150 may include one or more DB subnets 1130 that may communicatively couple to service gateway 1136 contained in data plane VCN 1118.

[0259] One or more untrusted application subnets 1162 may include a primary VNIC 1164(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1166(1)-(N) residing within one or more untrusted application subnets 1162. Each tenant VM 1166(1)-(N) may run code in a corresponding container 1167(1)-(N) and is communicatively coupled to an application subnet 1126 that may be contained in a data plane application layer 1146 contained in a container egress VCN 1168. A corresponding secondary VNIC 1172(1)-(N) may facilitate communication between one or more untrusted application subnets 1162 contained in a data plane VCN 1118 and the application subnet contained in a container egress VCN 1168. The container egress VCN may include a public internet 1154 (e.g., Figure 8 The public internet (854) uses NAT gateway 1138.

[0260] Internet gateway 1134, contained in control plane VCN 1116 and data plane VCN 1118, can be communicatively coupled to metadata management service 1152 (e.g., Figure 8 Metadata management service 1152 can be communicatively coupled to the public internet 1154. The public internet 1154 can be communicatively coupled to a NAT gateway 1138 contained in a control plane VCN 1116 and a data plane VCN 1118. A service gateway 1136 contained in a control plane VCN 1116 and a data plane VCN 1118 can be communicatively coupled to a cloud service 1156.

[0261] In some examples, Figure 11 The architecture shown in block diagram 1100 can be considered as... Figure 10 This is an exception to the pattern shown in the architecture of block diagram 1000, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1167(1)-(N) contained in each customer's virtual machine 1166(1)-(N). Container 1167(1)-(N) can be configured to invoke the corresponding auxiliary VNIC 1172(1)-(N) contained in one or more application subnets 1126 of the data plane application layer 1146, which may be contained in the container egress VCN 1168. The auxiliary VNIC 1172(1)-(N) can transmit the invocation to NAT gateway 1138, which can transmit the invocation to the public internet 1154. In this example, containers 1167(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1116 and from other entities contained in the data plane VCN 1118. Containers 1167(1)-(N) can also be isolated from resources from other clients.

[0262] In other examples, a client may use container 1167(1)-(N) to invoke cloud service 1156. In this example, the client may run code within container 1167(1)-(N) requesting a service from cloud service 1156. Container 1167(1)-(N) may transmit the request to auxiliary VNIC 1172(1)-(N), which may transmit the request to a NAT gateway, which may then transmit the request to the public internet 1154. The public internet 1154 may then transmit the request via internet gateway 1134 to one or more LB subnets 1122 contained in control plane VCN 1116. In response to determining that the request is valid, one or more LB subnets may transmit the request to one or more application subnets 1126, which may then transmit the request to cloud service 1156 via service gateway 1136.

[0263] It should be recognized that the IaaS architectures 800, 900, 1000, and 1100 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0264] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0265] Example computer system

[0266] Figure 12 An example computer system 1200 in which various embodiments can be implemented is illustrated. Computer system 1200 can be used to implement any of the computer systems described above. As shown, computer system 1200 includes a processing unit 1204 that communicates with a plurality of peripheral subsystems via a bus subsystem 1202. These peripheral subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 includes a tangible computer-readable storage medium 1222 and system memory 1210.

[0267] Bus subsystem 1202 provides a mechanism for allowing various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0268] A processing unit 1204, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1200. One or more processors may be included in the processing unit 1204. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1204 may be implemented as one or more independent processing units 1232 and / or 1234, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1204 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0269] In various embodiments, processing unit 1204 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1204 and / or storage subsystem 1218. With appropriate programming, processor(s) 1204 can provide the various functions described above. Computer system 1200 may additionally include processing acceleration unit 1206, which may include digital signal processor (DSP), dedicated processor, etc.

[0270] I / O subsystem 1208 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input to an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0271] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0272] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1200 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0273] Computer system 1200 may include a storage subsystem 1218 that provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which provide the aforementioned functionality when executed by one or more cores or processors of processing unit 1204. Storage subsystem 1218 may also provide a repository for storing data used according to this disclosure.

[0274] As in Figure 12 As illustrated in the example, storage subsystem 1218 may include various components, including system memory 1210, computer-readable storage medium 1222, and computer-readable storage medium reader 1220. System memory 1210 may store program instructions that can be loaded and executed by processing unit 1204. System memory 1210 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1210, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0275] System memory 1210 may also store operating system 1216. Examples of operating system 1216 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS and Palm® OS. In some embodiments where computer system 1200 executes one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 1210 and executed by one or more processors or cores of processing unit 1204.

[0276] Depending on the type of computer system 1200, system memory 1210 can have different configurations. For example, system memory 1210 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, system memory 1210 may include a basic input / output system (BIOS), which contains basic routines that facilitate, for example, the transfer of information between elements within computer system 1200 during startup.

[0277] Computer-readable storage medium 1222 may represent a remote, local, fixed and / or removable storage device, as well as a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1204 of the computer system 1200) for use by the computer system 1200.

[0278] Computer-readable storage medium 1222 may include any suitable medium known or used in the art, including storage and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0279] For example, computer-readable storage medium 1222 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a disk drive that reads or writes to a removable non-volatile magnetic disk, and an optical disc drive that reads or writes to a removable non-volatile optical disc (such as a CD-ROM, DVD, and Blu-ray® disc or other optical media). Computer-readable storage medium 1222 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and the like. Computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program services and other data for computer system 1200.

[0280] Machine-readable instructions executable by one or more processors or cores of processing unit 1204 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0281] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 serves as an interface for receiving data from other systems and sending data from computer system 1200 to other systems. For example, the communication subsystem 1224 enables computer system 1200 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1224 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, 5G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1224 may provide a wired network connection (e.g., Ethernet).

[0282] In some embodiments, the communication subsystem 1224 may also represent one or more users who can use the computer system 1200 to receive input communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc.

[0283] For example, the communication subsystem 1224 can be configured to receive data feeds 1226 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0284] Furthermore, the communication subsystem 1224 can also be configured to receive data in the form of a continuous data stream, which may include event streams 1228 and / or event updates 1230 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0285] The communication subsystem 1224 can also be configured to output structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1200.

[0286] The computer system 1200 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing system.

[0287] Due to the ever-evolving nature of computers and networks, the description of the computer system 1200 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0288] According to at least one embodiment, a computer program product is provided. The computer program product includes a computer program or instructions, characterized in that, when the computer program or instructions are executed by a processor, they perform the steps of one or more of the disclosed methods.

[0289] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can freely operate within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the series of transactions and steps described. The various features and aspects of the above embodiments can be used individually or in combination.

[0290] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes can communicate using various technologies, including but not limited to conventional inter-process communication technologies, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0291] Accordingly, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0292] In the context of describing the disclosed embodiments (especially in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as covering both singular and plural, unless otherwise indicated herein or obviously contradicted by the context. Unless otherwise stated, the terms “including,” “having,” “comprising,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or obviously contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0293] Disjunctive language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, each individually.

[0294] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0295] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0296] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

[0297] As used herein, when an action is “based on” something, it means that the action is based at least partially on at least a portion of that thing. As used herein, the terms “substantially,” “about,” and “approximately” are defined as largely but not necessarily entirely of the specified content (and include the fully specified content), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “substantially,” “about,” or “approximately” may be replaced with “within [percentage] of the specified content,” as a non-limiting example where the percentage can be between 0 and 10%.

[0298] While illustrative examples of this application have been described in detail herein, it is to be understood that the inventive concept can be implemented and used in a variety of other ways, and the appended claims are intended to be construed as including such variations, except as limited by the prior art.

Claims

1. A computer-implemented method, comprising: Access text that includes word sequences, where: Multiple spans are identified within the text. Each of the plurality of spans includes one or more words in the word sequence, and The multiple spans include: One or more target span pairs, each of the one or more target span pairs being associated with an entity type and including a subject span and an object span, and One or more intermediate context spans, each of the one or more intermediate context spans appearing between the subject span and the object span contained in the target pair; Embedded representations of lexical units associated with one or more words appearing in the one or more target span pairs, embedded representations of lexical units associated with entity types of the one or more target span pairs, and embedded representations of lexical units associated with one or more intermediate context words appearing in the one or more intermediate context spans are generated by a pre-trained machine learning (ML) model. The pre-trained ML model generates an entity-centric span embedding representation for each target span based on the embedding representation of the lexical units associated with each target span and the embedding representation of one or more lexical units associated with the entity type of that target span; The pre-trained ML model generates an intermediate context embedding representation for each intermediate context span based on the embedding representation of the lexical units associated with each intermediate context span; The pre-trained ML model generates a probability distribution for each relation in a relation set for each target pair, based on entity-centric span embeddings of the subject span and object span, and intermediate context embeddings for intermediate context spans occurring between the subject and object spans in that target pair; the relation set includes empty relations. The pre-trained ML model predicts a given relationship between target span pairs based on the probability distribution.

2. The computer-implemented method as described in claim 1, wherein: Entity-centric span embedding representations are generated by integrating entity-type-specific information with the embedding representations of lemmas associated with each target span through entity-independent span lemma embeddings, span-aligned entity embeddings, and span-independent entity representations.

3. The computer-implemented method as described in claim 2, further comprising: The fully connected layers of the pre-trained ML model apply activation functions to the entity-centric span embedding representation of the subject span, the entity-centric span embedding representation of the object span, and the intermediate context embedding representation of the intermediate context span that appears between the subject span and the object span of each target pair, to obtain the activated entity-centric span embedding representation of the subject span, the activated entity-centric span embedding representation of the object span, and the activated intermediate context embedding representation of each target pair.

4. The computer-implemented method as described in claim 3, further comprising: For each target pair, the fully connected layer generates a concatenation of the active entity-centric span embedding representation of the subject span, the active entity-centric span embedding representation of the object span, and the active intermediate context embedding representation associated with each target pair.

5. The computer-implemented method of claim 4, wherein a probability distribution for each relation of each target pair is generated based on the cascade.

6. The computer-implemented method of claim 5, wherein generating the probability distribution for each relation of each target pair takes into account entity-specific information integrated into the entity-centric span embedding representation of the subject span and the entity-centric span embedding representation of the object span contained in each target pair.

7. The computer-implemented method of claim 2, wherein the entity-independent span lexical embedding maintains an embedding representation of lexical units associated with one or more words within the target span, the span-aligned entity embedding includes a fixed-length representation of entity types based on soft alignment with lexical units of the target span, and the span-independent entity representation includes a representation of entity types that do not depend on lexical units of the target span.

8. A system comprising: One or more processors; as well as One or more computer-readable media storing instructions, which, when executed by the one or more processors, cause the system to perform a method comprising: Access text that includes word sequences, where: Multiple spans are identified within the text, each of which includes one or more words from the word sequence. The multiple spans include: One or more target span pairs, each of the one or more target span pairs being associated with an entity type and including a subject span and an object span, and One or more intermediate context spans, each of the one or more intermediate context spans appearing between the subject span and the object span contained in the target pair; Embedded representations of lexical units associated with one or more words appearing in the one or more target span pairs, embedded representations of lexical units associated with entity types of the one or more target span pairs, and embedded representations of lexical units associated with one or more intermediate context words appearing in the one or more intermediate context spans are generated by a pre-trained machine learning (ML) model. The pre-trained ML model generates an entity-centric span embedding representation for each target span based on the embedding representation of the lexical units associated with each target span and the embedding representation of one or more lexical units associated with the entity type of that target span; The pre-trained ML model generates an intermediate context embedding representation for each intermediate context span based on the embedding representation of the lexical units associated with each intermediate context span; The pre-trained ML model generates a probability distribution for each relation in a relation set for each target pair, based on entity-centric span embeddings of the subject span and object span, and intermediate context embeddings for intermediate context spans occurring between the subject and object spans in that target pair; the relation set includes empty relations. The pre-trained ML model predicts a given relationship between target span pairs based on the probability distribution.

9. The system of claim 8, wherein: Entity-centric span embedding representations are generated by integrating entity-type-specific information with the embedding representations of lemmas associated with each target span through entity-independent span lemma embeddings, span-aligned entity embeddings, and span-independent entity representations.

10. The system of claim 9, wherein the method further comprises: The fully connected layers of the pre-trained ML model apply activation functions to the entity-centric span embedding representation of the subject span, the entity-centric span embedding representation of the object span, and the intermediate context embedding representation of the intermediate context span that appears between the subject span and the object span of each target pair, to obtain the activated entity-centric span embedding representation of the subject span, the activated entity-centric span embedding representation of the object span, and the activated intermediate context embedding representation of each target pair.

11. The system of claim 10, wherein the method further comprises: For each target pair, the fully connected layer generates a concatenation of the active entity-centric span embedding representation of the subject span, the active entity-centric span embedding representation of the object span, and the active intermediate context embedding representation associated with each target pair.

12. The system of claim 11, wherein a probability distribution for each relation of each target pair is generated based on the cascade.

13. The system of claim 12, wherein the probability distribution generated for each relation of each target pair takes into account entity-type specific information integrated into the entity-centric span embedding representation of the subject span and the entity-centric span embedding representation of the object span contained in each target pair.

14. The system of claim 9, wherein the entity-independent span lexical embedding maintains an embedding representation of lexical units associated with one or more words within the target span, the span-aligned entity embedding includes a fixed-length representation of entity types based on soft alignment with lexical units of the target span, and the span-independent entity representation includes a representation of entity types that do not depend on lexical units of the target span.

15. One or more non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: Access text that includes word sequences, where: Multiple spans are identified within the text. Each of the plurality of spans includes one or more words in the word sequence, and The multiple spans include: One or more target span pairs, each of the one or more target span pairs being associated with an entity type and including a subject span and an object span, and One or more intermediate context spans, each of the one or more intermediate context spans appearing between the subject span and the object span contained in the target pair; Embedded representations of lexical units associated with one or more words appearing in the one or more target span pairs, embedded representations of lexical units associated with entity types of the one or more target span pairs, and embedded representations of lexical units associated with one or more intermediate context words appearing in the one or more intermediate context spans are generated by a pre-trained machine learning (ML) model. The pre-trained ML model generates an entity-centric span embedding representation for each target span based on the embedding representation of the lexical units associated with each target span and the embedding representation of one or more lexical units associated with the entity type of that target span; The pre-trained ML model generates an intermediate context embedding representation for each intermediate context span based on the embedding representation of the lexical units associated with each intermediate context span; The pre-trained ML model generates a probability distribution for each relation in a relation set for each target pair, based on entity-centric span embeddings of the subject span and object span, and intermediate context embeddings for intermediate context spans occurring between the subject and object spans in that target pair; the relation set includes empty relations. The pre-trained ML model predicts a given relationship between target span pairs based on the probability distribution.

16. One or more non-transitory computer-readable media as claimed in claim 15, wherein the entity-centric span embedding representation is generated by integrating entity-type-specific information with the embedding representation of lemmas associated with each target span using entity-independent span lexical embeddings, span-aligned entity embeddings, and span-independent entity representations, and The entity-independent span lexical embeddings maintain the embedding representations of lexical units associated with one or more words within the target span, the span-aligned entity embeddings include fixed-length representations of entity types based on soft alignment with lexical units of the target span, and the span-independent entity representations include representations of entity types that do not depend on lexical units of the target span.

17. The one or more non-transitory computer-readable media as claimed in claim 16, wherein the method further comprises: The fully connected layers of the pre-trained ML model apply activation functions to the entity-centric span embedding representation of the subject span, the entity-centric span embedding representation of the object span, and the intermediate context embedding representation of the intermediate context span that appears between the subject span and the object span of each target pair, to obtain the activated entity-centric span embedding representation of the subject span, the activated entity-centric span embedding representation of the object span, and the activated intermediate context embedding representation of each target pair.

18. The one or more non-transitory computer-readable media of claim 17, wherein the method further comprises: For each target pair, the fully connected layer generates a concatenation of the active entity-centric span embedding representation of the subject span, the active entity-centric span embedding representation of the object span, and the active intermediate context embedding representation associated with each target pair.

19. One or more non-transitory computer-readable media as claimed in claim 18, wherein a probability distribution for each relation of each target pair is generated based on the cascading.

20. One or more non-transitory computer-readable media as claimed in claim 19, wherein the probability distribution generated for each relation of each target pair takes into account entity-specific information integrated into the active entity-centric span embedding representation of the subject span and the active entity-centric span embedding representation of the object span contained in each target pair.

21. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, the computer-implemented method as described in any one of claims 1-7 is performed.

Images