Text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs

Abstract

The application relates to a text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs, which comprises a lightweight entity pair label mechanism, span representation based on global semantics and local adhesion, multimodal token pair association representation, global semantic consistency constraint for entity pairs, and triple decoding with bidirectional mapping of boundaries. A general representation of a span is formed based on global semantics and local adhesion, and multimodal token pair association representation and relation token pair global semantic consistency constraint are modeled in two stages before and after embedding the relation. In addition, the relation representation embedded with the text semantics is combined with a lightweight bidirectional mapping decoding mechanism to complete one-step extraction of the relation triple. Compared with the prior art, the adhesion characteristics between span tokens are considered, the generation probability of irrelevant noise entity pairs is reduced, the resources occupied by the model during training are less, and the performance is better.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology for short texts, and in particular to a method for extracting text relation triples based on multimodal association representation and global semantic consistency of entity pairs. Background Technology

[0002] With the widespread application of social networks in daily life and work, online information mediated by mobile internet permeates our surroundings. This information contains fragmented and noisy knowledge about social risk events, making their perception and regulation particularly difficult. Therefore, extracting relevant knowledge from massive amounts of complex data and representing it in a structured form is a crucial component of risk event representation and a key step in constructing a knowledge graph of risk events. Furthermore, the relationships between elements in risk events are complex and diverse, and the elements are highly dispersed.

[0003] Entity relation extraction for short texts is a research hotspot in natural language processing. The task requires extracting entity pairs representing relationships from short texts and classifying the relationships between these pairs. Traditional entity relation extraction treats entity recognition and relation classification as two independent models: first, it identifies the span of entities and classifies their types; then, it predicts the relationships between potentially related entity pairs. Due to the dependency between subtasks, model errors accumulate, significantly impacting performance. Furthermore, deep learning-based joint entity relation extraction methods interact with the entity recognition and relation classification subtasks to share information and jointly implement both tasks. This joint extraction method represents a significant advancement over pipeline-based extraction methods. Currently, it can be broadly categorized into sequence-label-based methods, table-filling-based methods, and generative model-based methods. The core idea of ​​sequence-label-based methods is to train the model using a sequence of labels to identify the starting position of entities, determine the entity span, and then combine the identified entity spans pairwise to determine whether a relationship exists between entity pairs or what type of relationship exists. Table-filling-based methods primarily determine the relationship between token pairs corresponding to different positions, accurately fill the table, and then apply a decoding mechanism to parse the predicted triples from the table. Generative models, on the other hand, mainly establish sequence-to-sequence generative models to generate entity pairs or triples from text sequences.

[0004] In existing solutions for relation triple extraction, whether based on sequence labeling or table filling, most methods measure entities from their starting position, neglecting the influence of relationships between multiple tokens representing the same entity. That is, in a specific context, multiple tokens representing the same entity are sticky and inseparable. This stickiness plays a crucial role in recognizing semantic entities within a specific context and also benefits the recognition of subsequent relationships within that context. Furthermore, most existing relation triple extraction methods match all possible start and end entities and relationships one by one to find suitable triples, but this method consumes a large amount of memory resources, significantly impacting resource usage and model training speed.

[0005] Therefore, how to provide a general representation of single-token entities, multi-token entities, and various nested entity types in a specific context, how to simultaneously model multimodal information within and between spans, how to reduce the generation of noisy data for non-triple semantic entities, and how to achieve one-step extraction of relation triples with a lightweight model architecture to save resource consumption are urgent problems to be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for extracting relation triples based on multimodal association representation and global semantic consistency of entity pairs.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A method for extracting text relation triples based on multimodal association representation and global semantic consistency of entity pairs, comprising the following steps:

[0009] For a given text sequence, a global semantic representation is obtained, while the sticky features of adjacent spans are used as local semantic representations; the global semantic representation and the local semantic representation are fused to form a sequence semantic representation of the token.

[0010] Before relation embedding, token pairs within a span and token pairs between spans are treated as two different modal spaces to obtain a multimodal token pair association representation;

[0011] After relation embedding, the token pairs of embedded relations are subjected to global semantic consistency constraints oriented towards entity pairs, and lightweight relation triple encoding based on bidirectional boundary mapping is performed considering the boundary conditions of span constraints.

[0012] Decoding nested relation triples based on a boundary bidirectional mapping mechanism yields relation triples for a given text sequence.

[0013] As a preferred technical solution, the method uses a pre-trained language model BERT to encode the sequence to generate context-related word vector representations as a text-level global semantic representation of the token sequence.

[0014] As a preferred technical solution, the viscous characteristics based on adjacent spans are described as follows:

[0015] Previous adjacent viscosity characteristics:

[0016]

[0017] Adjacent viscosity characteristics:

[0018]

[0019] Among them, W before b before and W after b after These are the trainable parameters of adjacent tokens before and after fusion, x i This represents the embedding representation corresponding to the i-th token in the sequence.

[0020] As a preferred technical solution, the sequence semantic representation that integrates global semantics and local stickiness is expressed as follows:

[0021]

[0022] Where, x prei and x posti W represents the formed preceding adjacent viscosity feature and the following adjacent viscosity feature, respectively. fuse and b fuse To integrate trainable parameters that combine global semantics and local stickiness, For the concatenation operation, ReLU is the activation function.

[0023] As a preferred technical solution, for the multimodal token pair association representation, token pairs within a span and token pairs between spans are treated as two different modal spaces:

[0024] The token pair association representation of the token pair association pattern within the span is as follows:

[0025]

[0026] The token pair association representation of the span-based token pair association pattern is as follows:

[0027]

[0028] Among them, W correlation and bcorrelation Here are the training parameters for the association process, and + represents the sigmoid activation function; x fusei x is the fused sequence semantic representation corresponding to the i-th token; interij and x exterij These represent the intra-span association representation of the token pair and the inter-span association representation of the token pair corresponding to the i-th token and the j-th token, respectively.

[0029] As a preferred technical solution, the global semantic consistency constraint for entity pairs specifically includes:

[0030] g ij =σ(W g T ij +b g )

[0031] Among them, g ij W represents the global semantic consistency score of entity pairs after embedding relations. g and b g T represents globally constrained trainable parameters. ij The entity pair representation of the i-th token and the j-th token after embedding the relation:

[0032]

[0033] Where dropout is the random probability connection loss function; x fusei and x fusej These are the semantic representations of the i-th and j-th tokens after fusing global semantics and local stickiness, respectively; W r and b r , respectively, are trainable parameters for the entity pair representation process of embedded relations; seq is the full-text semantic representation of the sequence, and is the max pooling of the sequence tokens.

[0034] As a preferred technical solution, the lightweight relation triple encoding based on boundary bidirectional mapping adopts a lightweight relation triple labeling strategy: a unique deterministic strategy for head-to-head and tail-to-tail entity pairs is adopted to establish directed secondary labels for relation entity pairs in short text data; and considering the boundary conditions of span constraints, the head-to-head and tail-to-tail associations between span pairs and the head-to-tail associations within the same span are used as sufficient and necessary conditions for the decoding of constraint entity pairs, and a binary classification task with different association patterns is adopted to achieve lightweight encoding of relation triples.

[0035] As a preferred technical solution, the lightweight relation triple encoding based on boundary bidirectional mapping includes:

[0036] The cross-entity head-token encoding process for relational constraints is represented as follows:

[0037] score ss =σ(W rt (W ss T ij +b ss )+b rt )

[0038] The cross-entity tail-tail token pair encoding process for relational constraints is represented as follows:

[0039] score ee =σ(W rt (W ee T ij +b ee )+b rt )

[0040] The head-to-tail association encoding process for entities with the same span is represented as follows:

[0041] score se =σ(W rt (W se T ij +b se )+b rt )

[0042] Among them, W ss b ss W ee b ee W se and b se W represents the trainable parameters for token pairs associated between head-to-head, tail-to-tail, and head-to-tail pairs within the same span, respectively. rt and b rt For trainable parameters of relational constraints, score ss score ee and score se These are the score pairs of tokens that constrain relationships between spans (head-to-head, tail-to-tail, and head-to-tail within the same span).

[0043] As a preferred technical solution, the decoding based on bidirectional boundary mapping uses cross-entity token pairs with the same span head-to-tail constraint to determine the relationship triples. It searches for the nearest neighbor tail-to-tail association position from the head-to-head association position based on nearest neighbor matching, supplemented by the same span head-to-tail association constraint; it extracts entity pairs and subordinate relationships in tuple form, wherein:

[0044] The decoding process employs a bidirectional mapping mechanism that maps from head-to-head to tail-to-tail and from tail-to-tail to head-to-head. The head-to-head to tail-to-tail mapping process constitutes a unidirectional decoding channel, while the tail-to-tail to head-to-head mapping process constitutes another decoding channel. In this process, the start and end tokens of the span are associated for verification to ensure that the constructed decoding channel belongs to the normal decoding order within the span.

[0045] As a preferred technical solution, the method establishes a multi-task joint optimization based on binary classification cross-entropy loss, which is based on the loss of the association representation subtask based on multimodal tokens before relation embedding, the loss of the global semantic consistency constraint subtask based on relation tokens after relation embedding, and the loss of the relation triple encoding subtask.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) This invention adopts a local and global tokens representation mode, taking into account the stickiness between span tokens, and uses a local and global tokens representation mode, and integrates local stickiness and text semantics to jointly construct a semantic representation of the token sequence.

[0048] 2) This invention considers multimodal associations within and between spans, forming different representation modes for token pairs in two stages: before and after embedding the relation. Before embedding the relation mode, a multimodal token pair association representation is established, based on whether the token pairs are associated and under what modality the association belongs, thus forming a multimodal token pair association representation paradigm. After embedding the relation mode, a global semantic consistency constraint is established for relation token pairs to constrain the formation of relation entity pairs, thereby reducing the probability of generating irrelevant noise entity pairs.

[0049] 3) This invention adopts a boundary bidirectional mapping decoding mechanism to achieve universal decoding of nested entities and nested entity pairs. Compared with existing similar methods, the model occupies less resources and takes less time during training, and has better performance. Attached Figure Description

[0050] Figure 1 This is a framework diagram of the relation triplet extraction method for short text based on multimodal association representation and global semantic consistency of entity pairs, as described in this invention.

[0051] Figure 2 Example diagram of lightweight relation triple decoding for bidirectional boundary mapping. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0053] Example 1

[0054] This invention proposes a method for extracting relation triples from short texts based on multimodal association representation and global semantic consistency of entity pairs. It achieves relation triple extraction for various nesting types through three levels of subtasks: whether token pairs are associated, whether relation token pairs are valid, and what kind of relation the relation token pairs belong to. Specifically, in the text representation stage: considering the indivisibility between multiple tokens within a span, this invention integrates global semantics based on the pre-trained language model BERT with adjacent local features to form a preliminary representation of the span. In the multimodal token pair association stage, considering the relational irrelevance of the span itself and the relational dependency of span pairs, before embedding the relation, this invention treats token pairs with the same span and token pairs with different spans as different modal spaces. After embedding the relation, it applies global semantic constraints on entity pairs of different spans to the relation token pairs. In the lightweight relation triple decoding stage, a boundary bidirectional mapping decoding mechanism is used to achieve lightweight decoding of multi-nested triples. Finally, a multi-task joint loss is used for joint training and learning.

[0055] Figure 1 This paper demonstrates the workflow of a method for extracting relation triples from short texts based on multimodal association representation and global semantic consistency of entity pairs. The specific steps of this method are described below:

[0056] The first step: the sequence representation stage that integrates global semantics and local sticky features. For a given text sequence S = (s1, s2, s3, ..., snp.) containing n tokens... n This invention uses the embedding representation of the sequence based on the pre-trained language model BERT as the global sequence representation, and the sticky features of adjacent spans as the local sequence representation. Specifically, the features of the boundary are represented as special vectors of the same dimension. The global semantics based on the pre-trained language model BERT and the local features of adjacent spans are fused to form the sequence semantic representation of the token.

[0057] Taking advantage of the pre-trained corpus possessed by pre-trained language models, this invention first uses the large-scale pre-trained language model BERT to encode the text sequence S to generate context-dependent word vector representations, which serve as a text-level global semantic representation of the token sequence. The BERT encoding process can be represented as follows:

[0058] X = (x1, x2, x3, ..., x n ) = Bert(s1,s2,s3,...,s n (1)

[0059] Where Bert() represents the pre-trained language model Bert encoder, X represents the last hidden layer after Bert encoding, and x i Let S represent the embedding representation corresponding to the i-th token in S.

[0060] Regarding the representation of local stickiness features, considering that for multiple tokens within a span, their collective representation in the text is the only effective representation of that span, this invention models the local stickiness of the span from adjacent tokens and integrates global sequence semantics to jointly construct a preliminary representation of the span. The local stickiness representation process can be described as follows:

[0061] Characterization of viscosity between adjacent nodes:

[0062]

[0063] Characterization of viscosity in adjacent components:

[0064]

[0065] Semantic representation that integrates global semantics and local stickiness:

[0066]

[0067] Among them, W before b before and W after b after These are the trainable parameters of adjacent tokens before and after fusion, x prei and x posti W represents the formed preceding adjacent viscosity feature and the following adjacent viscosity feature, respectively. fuse and b fuse To integrate trainable parameters that combine global semantics and local cohesion, x fusei Let i be the fused semantic representation corresponding to the i-th token. For the concatenation operation, ReLU is the activation function.

[0068] The second step is to represent the association of multimodal token pairs. Considering the irrelevance of the relationship between the span itself and the dependency of the relationship between the span pairs, we will model the association of token pairs in two stages before and after embedding the relationship.

[0069] This invention first establishes a sequence subtask for token pair association representation before establishing embedding relationships. For tokens before embedding relationships, token pairs within a span and token pairs between spans are treated as two different modal spaces. A subtask—multimodal token pair association representation—is established based on whether the token pairs are associated, belonging to intra-span association or inter-span association. The process is as follows:

[0070] The association pattern of tokens within a span is represented as follows:

[0071]

[0072] The association pattern of token pairs across spans is represented as follows:

[0073]

[0074] Among them, W correlation and b correlation Here are the training parameters for the association process, σ is the sigmoid activation function, and x is the x-axis. interij and x exterij These represent the intra-span association representation of the token pair and the inter-span association representation of the token pair corresponding to the i-th token and the j-th token, respectively.

[0075] The third step: Regarding global semantic consistency constraints for span pairs, this invention views textual semantics as a fused representation of multiple relation triples contained within it. The relationship between entity pairs depends on their respective textual context. Therefore, this invention embeds the relation between entity pairs within the full-text context and considers the constraints of the embedded entity pairs under global semantics to reduce the probability of generating irrelevant entity pairs, thereby improving the encoding performance of effective relation triples. The process can be represented as follows:

[0076] This invention uses max-pooling of sequence tokens as a full-text semantic representation:

[0077] seq = maxpool(X) (7) The token pair of the embedding relation is represented as:

[0078]

[0079] The subtask of constraining global semantic consistency for entity pairs can be represented as:

[0080] g ij =σ(W g T ij +b g (9)

[0081] Among them, maxpool() is the element-wise max pooling operation, W rand b r Here, represents the trainable parameters for the entity pair representation process of embedding relations, `dropout` is the random probabilistic connection loss function, `seq` is the full-text semantic representation of the sequence, and `x` is the trainable parameters. fusei and x fusej T represents the semantic representation of the i-th and j-th tokens after fusing global semantics and local stickiness, respectively. ij W represents the entity pair representation of the i-th token and the j-th token after embedding the relation. g and b g For globally constrained trainable parameters, g ij This represents the global semantic consistency score of entity pairs after embedding the relation.

[0082] The fourth step: Lightweight relation triple encoding based on bidirectional boundary mapping. In the decoding stage, bidirectional head-to-head and tail-to-tail boundary two-level mapping (head represents the starting token of the span, and tail represents the ending token of the span) is used to solve the problem of nested entities and nested entity pairs.

[0083] This invention considers the boundary conditions of span constraints, taking head-to-head and tail-to-tail associations between span pairs, as well as head-to-tail associations within the same span, as necessary and sufficient conditions for decoding constraint entity pairs. It employs a binary classification task with different association modes to achieve lightweight encoding of relation triples. Specifically, this invention unifies the promoting effect of span recognition on relation extraction with the unifying representation of relation extraction on span recognition. For relation extraction, considering head and tail entities that may contain multiple tokens in relation triples, this invention uses boundary mapping to locate the head-to-head and tail-to-tail correspondences of head and tail entities; corresponding to span recognition, this invention locates the head-to-tail correspondence of entities. In this way, this invention integrates information within the same span while also incorporating associations between different spans, synchronously modeling associations within the same span and across different spans. This multi-modal association, on the one hand, synchronously models the mutual promoting effect of span and relation extraction, and on the other hand, ensures the accuracy of decoding relation entity pairs under the same span association constraint during subsequent decoding, effectively filtering out non-span association noise. The modeling process for the correlation between different spans and the correlation within the same span is as follows:

[0084] The cross-entity head-token encoding process for relational constraints can be represented as:

[0085] score ss =σ(W rt (W ss T ij +b ss )+b rt (10)

[0086] The cross-entity pair tail-tail token pair encoding process for relational constraints can be represented as:

[0087] score ee =σ(W rt (W ee T ij +b ee )+b rt (11)

[0088] The head-to-tail association encoding process for entities with the same span can be represented as:

[0089] score se =σ(W rt (W se T ij +b se )+b rt (1)

[0090] Among them, W ss b ss W ee b ee W se and b se W represents the trainable parameters for token pairs associated between head-to-head, tail-to-tail, and head-to-tail pairs within the same span, respectively. rt and b rt For trainable parameters of relational constraints, score ss score ee and score se These are the score pairs of tokens that constrain relationships between spans (head-to-head, tail-to-tail, and head-to-tail within the same span).

[0091] Step 5: Relational Triple Decoding: The decoding process employs a bidirectional boundary mapping mechanism, mapping from head-to-head to tail-to-tail and from tail-to-tail to head-to-head, supplemented by same-span constraints to extract entity pairs and associated relations (entity, relation, entity) in tuple form. This invention uses head-to-tail constraints on cross-entity token pairs with the same span to determine the composition of relational triples. Based on nearest neighbor matching, it searches for the nearest tail-to-tail association position from the head-to-head association position, while simultaneously using same-span head-to-tail association constraints to filter noisy span association predictions, effectively improving extraction accuracy. In this process, this invention uses bidirectional head-to-head and tail-to-tail relationships with different spans to solve the problem of entity pair nesting. This lightweight decoding strategy can be applied to relational triple extraction tasks with different nesting types, such as nested spans and nested entity pairs. Figure 2An example of bidirectional boundary mapping decoding with span constraints is given, where SS represents the association between the start token of the head entity and the end token of the tail entity, EE represents the association between the end token of the head entity and the end token of the tail entity, and SE represents the association between the start token and the end token of the span. The specific process can be described as follows: the SS-EE process forms a unidirectional decoding channel, and the EE-SS process forms another decoding channel. SE verification is used during this process to ensure that the constructed decoding channel follows a normal decoding order within the span, effectively filtering out noisy decoding that is not associated with the span, and effectively addressing different types of nesting challenges.

[0092] Finally, based on the loss loss 2 for the association representation subtask using multimodal tokens before relation embedding, the loss loss 3 for the global semantic consistency subtask using relation tokens after relation embedding, and the loss loss 1 for the relation triple encoding subtask, a joint optimization loss for multi-task learning based on binary classification cross-entropy loss is established:

[0093] L=loss1+λ1·loss2+λ2·loss3

[0094] In this embodiment, the hyperparameters are set to λ1 = λ2 = 0.01.

[0095] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for extracting text relation triples based on multimodal association representation and global semantic consistency of entity pairs, characterized in that the steps include... include: For a given text sequence, a global semantic representation is obtained, while the sticky features of adjacent spans are used as local semantic representations. The global semantic representation and the local semantic representation are fused to form a sequence semantic representation of the token. Specifically, the text sequence is encoded using a pre-trained language model BERT to generate context-related word vector representations, which serve as the text-level global token sequence semantic representation. The viscous characteristics based on adjacent spans are described as follows: Previous adjacent viscosity characteristics: Adjacent viscosity characteristics: in, , and , These are the trainable parameters of adjacent tokens before and after fusion. Indicates the first in the sequence i The embedded representation corresponding to each token; The sequence semantic representation that integrates global semantics and local stickiness is represented as follows: in, and These represent the formed preceding adjacent viscosity features and the subsequent adjacent viscosity features, respectively. and To integrate trainable parameters that combine global semantics and local stickiness, For splicing operations, For activation functions; Before relational embedding, token pairs within a span and token pairs between spans are treated as two different modal spaces to obtain a multimodal token pair association representation. After relation embedding, the token pairs of embedded relations are subjected to global semantic consistency constraints oriented towards entity pairs, and lightweight relation triple encoding based on bidirectional boundary mapping is performed considering the boundary conditions of span constraints. Decoding nested relation triples based on a boundary bidirectional mapping mechanism yields relation triples for a given text sequence.

2. The text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs according to claim 1, characterized in that, For the aforementioned multimodal token pair association representation, token pairs within a span and token pairs between spans are treated as two different modal spaces: The token pair association representation of the token pair association pattern within the span is as follows: The token pair association representation of the span-based token pair association pattern is as follows: in, and For training parameters of the association process, It is the sigmoid activation function; For the first i The fused sequence semantic representation corresponding to each token; and They represent the first i The token and the first j The association representation of token pairs within a span and the association representation of token pairs between spans for each token.

3. The text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs according to claim 1, characterized in that, The specific global semantic consistency constraint for entity pairs is as follows: in, The global semantic consistency score of entity pairs after embedding relations; and Trainable parameters are globally constrained; For the first embedded relation i The token and the first j Entity pairs represented by each token: in, For random probability connection loss function; and The first i, j The semantic representation corresponding to each token after integrating global semantics and local stickiness; and These are the trainable parameters for the entity pair representation process of the embedded relationship; For the full-text semantic representation of the sequence, and for the max pooling of the sequence tokens.

4. The text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs according to claim 1, characterized in that, The lightweight relation triple encoding based on boundary bidirectional mapping adopts a lightweight relation triple labeling strategy: it uses a unique deterministic strategy for head-to-head and tail-to-tail entity pairs to establish directed secondary labels for relation entity pairs in short text data; and considering the boundary conditions of span constraints, it uses head-to-head and tail-to-tail associations between span pairs and head-to-tail associations within the same span as sufficient and necessary conditions for decoding the entity pairs, and uses binary classification tasks with different association patterns to achieve lightweight encoding of relation triples.

5. The text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs according to claim 4, characterized in that, In the lightweight relation triple encoding based on boundary bidirectional mapping: The cross-entity head-token encoding process for relational constraints is represented as follows: The cross-entity tail-tail token pair encoding process for relational constraints is represented as follows: The head-to-tail association encoding process for entities with the same span is represented as follows: in, , , , , and These are the trainable parameters for token pairs associated between head-to-head, tail-to-tail, and head-to-tail pairs within the same span, respectively. and These are the trainable parameters for relational constraints. , and These are the score pairs of tokens that constrain relationships between spans (head-to-head, tail-to-tail, and head-to-tail within the same span).

6. The text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs according to claim 5, characterized in that, The decoding based on bidirectional boundary mapping uses cross-entity token pairs with the same span head-to-tail constraint to determine the relationship triples. It searches for the nearest neighbor tail-to-tail association position from the head-to-head association position based on nearest neighbor matching, supplemented by the same span head-to-tail association constraint. This extracts entity pairs and their associated relationships in tuple form, where: The decoding process employs a bidirectional mapping mechanism that maps from head-to-head to tail-to-tail and from tail-to-tail to head-to-head. The head-to-head to tail-to-tail mapping process constitutes a unidirectional decoding channel, while the tail-to-tail to head-to-head mapping process constitutes another decoding channel. In this process, the start and end tokens of the span are associated for verification to ensure that the constructed decoding channel belongs to the normal decoding order within the span.

7. The text relation triple extraction method based on multimodal association representation and global semantic consistency of entity pairs according to claim 1, characterized in that, The method is based on the loss of the association representation subtask using multimodal tokens before relation embedding, the loss of the global semantic consistency constraint subtask using relation tokens after relation embedding, and the loss of the relation triple encoding subtask. It establishes a multi-task joint optimization based on binary classification cross-entropy loss.

Images