Text relation triple extraction method based on multi-modal association representation and entity-to-global semantic consistency

Abstract

The invention relates to a text relation triple extraction method based on multi-modal association representation and entity pair global semantic consistency. The method comprises a lightweight entity pair label mechanism, span representation based on global semantics and local viscosity, multi-modal token pair association representation, entity pair-oriented global semantic consistency constraint, and triple decoding bidirectionally mapped with boundaries. Universal representation of span formed based on global semantics and local viscosity is adopted, and multi-modal token pair association representation and relation token pair global semantics consistency constraint are modeled in the front stage and the back stage of the embedding relation respectively. In addition, one-step extraction of a relation triple is completed by adopting relation representation of embedded text semantics and combining a lightweight bidirectional mapping decoding mechanism. Compared with the prior art, the method has the advantages that the viscosity characteristic between span tokens is considered, the generation probability of irrelevant noise entity pairs is reduced, meanwhile, the model occupies fewer resources during training, and the performance is better.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing for short texts, and in particular to a method for extracting text relationship triples based on multimodal association representation and global semantic consistency of entity pairs. Background Art

[0002] With the popularization of social networks in daily production and life, network information mediated by mobile Internet is all around us. It contains knowledge of social risk events with high degree of fragmentation and huge noise information, making it particularly difficult to perceive and supervise. Therefore, how to extract the knowledge of interest from a large amount of complex data and represent it in a structured form is an important part of characterizing risk events, and it is also a key step in building a knowledge graph of events in the risk field. In addition, the relationship between elements in risk events is complex and diverse, and the elements are highly dispersed.

[0003] Entity relationship extraction for short texts is one of the research hotspots in the field of natural language processing. Its task requires extracting entity pairs that represent relationships from short texts and classifying the relationships between entity pairs. Traditional entity relationship extraction treats entity recognition and relationship classification as two independent models, that is, first performing entity span recognition and type classification, and then classifying and predicting the relationship for entity pairs that may have relationships. Due to the dependency between subtasks, the model errors of this method are accumulated, which greatly affects the performance of the task. In addition, the joint entity relationship extraction method based on deep learning interacts the two subtasks of entity recognition and relationship classification to achieve information sharing between subtasks, thereby jointly implementing the two subtasks. This joint extraction method has made great progress compared to the pipeline extraction method. It can currently be divided into sequence label-based methods, table filling-based methods, generative model-based methods, etc. The core idea of ​​the sequence label-based method is to promote model training in the form of label sequences, so as to find the starting position of the entity, determine the entity span, and then combine the identified entity spans two by two to determine whether there is a relationship between the entity pairs or what type of relationship exists; the table filling-based method mainly determines the relationship between the token pairs corresponding to different positions, accurately fills the table, and then applies the decoding mechanism to parse the predicted triples from the table; the generative model mainly generates entity pairs or triples from the text sequence by establishing a sequence-to-sequence generative model.

[0004] In the existing solutions for relation triple extraction tasks, whether based on sequence labels or table filling, most of them measure entities from the starting position, ignoring the impact of the relationship between multiple tokens of the same entity on the entity itself, that is, in a specific context, multiple tokens representing the same entity are sticky and inseparable. This sticky relationship plays an important role in the recognition of semantic entities in a specific context, and also plays a beneficial role in the recognition of subsequent relations in the context. In addition, most existing relation triple extraction methods use all possible head and tail entities and relations to match one by one to find a suitable triple composition, but this method consumes a lot of memory resources, which greatly affects resource usage and model training speed.

[0005] Therefore, how to universally represent single-token entities, multi-token entities, and multiple nested entity types in a specific context, synchronously model multimodal information within and between spans, and reduce the generation of noise data for non-triplet semantic entities, and how to achieve one-step extraction of relation triplets with a lightweight model architecture to save resource usage are issues that need to be addressed urgently. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a relationship triple extraction method based on multimodal association representation and global semantic consistency of entity pairs.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A method for extracting text relation triples based on multimodal association representation and global semantic consistency of entity pairs, comprising the following steps:

[0009] For a given text sequence, obtain a global semantic representation, and use the sticky features of the adjacent spans as a local semantic representation; fuse the global semantic representation with the local semantic representation to form a token sequence semantic representation;

[0010] Before relation embedding, token pairs within a span and token pairs between spans are treated as two different modal spaces to obtain multimodal token pair association representations.

[0011] After the relationship is embedded, the token pairs of the embedded relationship are constrained for global semantic consistency towards entity pairs, and lightweight relationship triple encoding based on boundary bidirectional mapping is performed considering the boundary conditions of span constraints;

[0012] The nested relation triples are decoded based on the boundary bidirectional mapping mechanism to obtain the relation triples of the given text sequence.

[0013] As a preferred technical solution, the method uses a pre-trained language model Bert to encode the sequence to generate context-related word vector representations as a global token sequence semantic representation at the text level.

[0014] As a preferred technical solution, the viscosity characteristics based on the adjacent spans are described as follows:

[0015] Front adjacent viscosity characteristics:

[0016]

[0017] Post-adjacent viscosity characteristics:

[0018]

[0019] Among them, W before , b before and W after , b after are the trainable parameters of adjacent tokens before and after fusion, x i Represents the embedding representation corresponding to the i-th token in the sequence.

[0020] As a preferred technical solution, the sequence semantic representation integrating global semantics and local viscosity is expressed as:

[0021]

[0022] Among them, x prei and x posti are the front adjacent viscosity characteristics and the rear adjacent viscosity characteristics formed, respectively, W fuse and b fuse To fuse the trainable parameters of global semantics and local viscosity, is the concatenation operation, and Relu is the activation function.

[0023] As a preferred technical solution, for the multimodal token pair association representation, the token pairs within the span and the token pairs between spans are treated as two different modal spaces:

[0024] The token pair association representation of the token pair association pattern within the span is expressed as:

[0025]

[0026] The token pair association representation of the token pair association pattern between spans is expressed as:

[0027]

[0028] Among them, W correlation and bcorrelation is the training parameter of the association process, + is the sigmoid activation function; x fusei is the fused sequence semantic representation corresponding to the i-th token; x interij and x exterij They represent the associated representation of the token pair within the span and the associated representation of the token pair between spans corresponding to the i-th token and the j-th token respectively.

[0029] As a preferred technical solution, the global semantic consistency constraint for entity pairs is specifically:

[0030] g ij =σ(W g T ij +b g )

[0031] Among them, g ij is the global semantic consistency score of the entity pair after embedding the relation; W g and b g is a globally constrained trainable parameter; T ij The entity pair representation of the i-th token and the j-th token after the embedding relationship:

[0032]

[0033] Among them, dropout is a random probability connection loss function; x fusei and x fusej are the semantic representations of the ith and jth tokens respectively, which are the fusion of global semantics and local viscosity; W r and b r They are respectively the trainable parameters of the entity pair representation process of the embedded relationship; seq is the semantic representation of the full text of the sequence, and is the maximum pooling of the sequence token.

[0034] As an optimal technical solution, the lightweight relation triple encoding based on boundary bidirectional mapping adopts a lightweight relation triple label strategy: a head-head and tail-tail entity pair unique deterministic strategy is adopted to establish directed secondary labels for relation entity pairs in short text data; and considering the boundary conditions of span constraints, the head-head and tail-tail associations between span pairs and the head-tail associations with the same span are used as sufficient and necessary conditions for decoding constrained entity pairs, and a binary classification task with different association patterns is used to achieve lightweight encoding of relation triples.

[0035] As a preferred technical solution, in the lightweight relation triple encoding based on boundary bidirectional mapping:

[0036] The cross-entity head-to-head token pair encoding process of relation constraints is expressed as:

[0037] score ss =σ(W rt (W ss T ij +b ss )+b rt )

[0038] The cross-entity tail-tail token pair encoding process of relation constraints is expressed as:

[0039] score ee =σ(W rt (W ee T ij +b ee )+b rt )

[0040] The head-tail association encoding process of entities with the same span is expressed as:

[0041] score se =σ(W rt (W se T ij +b se )+b rt )

[0042] Among them, W ss , b ss , W ee , b ee , W se and b se are the trainable parameters of the token pairs associated head-to-head, tail-to-tail, and head-to-tail within the same span, respectively, W rt and b rt is a trainable parameter of the relational constraint, score ss 、score ee and score se They are the token pair scores for the head-to-head and tail-to-tail relationships between spans and the head-to-tail relationships within the same span.

[0043] As a preferred technical solution, the decoding based on boundary bidirectional mapping adopts the association of cross-entity token pairs with the same span head-tail constraint to determine the relationship triple composition, searches from the head-head association position to the tail-tail association position of the nearest neighbor based on nearest neighbor matching, and is supplemented by the same span head-tail association constraint; extracts entity pairs and subordinate relationships associated in the form of tuples, where:

[0044] The decoding process adopts a boundary bidirectional mapping mechanism from head-to-head mapping to tail-to-tail mapping and from tail-to-tail mapping to head-to-head mapping. The process from head-to-head mapping to tail-to-tail mapping constitutes a unidirectional decoding channel, and the process from tail-to-tail mapping to head-to-head mapping constitutes another decoding channel. In this process, the association verification of the start token and the end token of the span is supplemented to ensure that the constructed decoding channel belongs to the normal decoding order between spans.

[0045] As an optimal technical solution, the method establishes a multi-task learning joint optimization based on binary cross entropy loss based on the association representation subtask loss based on multimodal tokens before relationship embedding, the global semantic consistency constraint subtask loss based on relationship tokens after relationship embedding, and the relationship triplet encoding subtask loss.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) The present invention adopts local and global tokens representation modes, takes the stickiness characteristics between span tokens into consideration, adopts local and global tokens representation modes, and integrates local stickiness with text semantics to jointly construct token sequence semantic representation.

[0048] 2) The present invention considers the multimodal association within and between spans, and forms different representation modes of token pairs in the two stages before and after embedding the relationship. Before embedding the relationship mode, a multimodal token pair association representation is established, and the association representation paradigm of multimodal token pairs is formed based on whether the token pairs are associated and the mode of association. After embedding the relationship mode, a global semantic consistency constraint of the relationship token pair is established to constrain the formation of relationship entity pairs, thereby reducing the probability of generating irrelevant noise entity pairs.

[0049] 3) The present invention adopts a boundary bidirectional mapping decoding mechanism to achieve universal decoding covering nested entities and nested entity pairs. Compared with existing similar methods, the model occupies fewer resources during training, takes less time, and has better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The framework diagram of the method for extracting relation triples based on multimodal association representation and global semantic consistency of entity pairs for short texts of the present invention is as follows

[0051] Figure 2 An example diagram of lightweight relation triple decoding for boundary bidirectional mapping. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0053] Example 1

[0054] The present invention proposes a relation triple extraction method for short texts based on multimodal association representation and global semantic consistency of entity pairs. Relation triples of various nested types are extracted from three hierarchical subtasks: whether the token pair is associated, whether the relation token pair is valid, and what kind of relationship the relation token pair belongs to. Specifically, in the text representation stage: considering the inseparability between multiple tokens within a span, the present invention integrates the global semantics based on the pre-trained language model BERT and the local features of the previous and next neighbors to form a preliminary representation of the span. In the multimodal token pair association stage, considering the relational independence of the span itself and the relational dependence of the span pair, before embedding the relationship, the present invention treats the token pairs of the same span and the token pairs of different spans as different modal spaces. After embedding the relationship, the relationship token pairs are subjected to global semantic constraints of entity pairs of different spans. In the lightweight relation triple decoding stage, the boundary bidirectional mapping decoding mechanism is used to achieve lightweight decoding of multi-nested type triples. Finally, a multi-task joint loss is used for joint training and learning.

[0055] Figure 1 The process of extracting relation triples based on multimodal association representation and global semantic consistency of entity pairs for short texts is shown. The steps of this method are described in detail:

[0056] The first step: the sequence representation stage that integrates global semantics and local sticky features. For a given short text sequence S containing n tokens, S = (s1, s2, s3, ..., s n ), using the embedded representation of the sequence based on the pre-trained language model Bert as the global level sequence representation, and the sticky features of the adjacent spans as the local level sequence representation, wherein the present invention represents the boundary features as special same-dimensional vectors. The global semantics based on the pre-trained language model BERT and the adjacent local features are integrated to form the sequence semantic representation of the token.

[0057] Due to the pre-trained corpus advantage of the pre-trained language model, the present invention first uses the large-scale pre-trained language model Bert to encode the text sequence S to generate context-related word vector representations, which are used as global token sequence semantic representations at the text level. The Bert encoding process can be expressed as:

[0058] X=(x1,x2,x3,...,x n )=Bert(s1,s2,s3,...,s n ) (1)

[0059] Among them, Bert() represents the pre-trained language model Bert encoder, X represents the last hidden layer after Bert encoding, and x i Represents the embedding representation corresponding to the i-th token in S.

[0060] In terms of local viscosity feature representation, considering that for multiple tokens in a span, their entirety in the text is the effective representation of a span, the present invention models the local viscosity of the span from the adjacent preceding and following tokens, and integrates the global sequence semantics to jointly construct the initial representation of the span. The local viscosity representation process can be described as:

[0061] Front adjacent viscosity characterization:

[0062]

[0063] Post-adjacent viscosity characterization:

[0064]

[0065] Semantic representation that combines global semantics with local stickiness:

[0066]

[0067] Among them, W before , b before and W after , b after are the trainable parameters of adjacent tokens before and after fusion, x prei and x posti are the front adjacent viscosity characteristics and the rear adjacent viscosity characteristics formed, respectively, W fuse and b fuse is a trainable parameter that integrates global semantics and local viscosity, x fusei is the fused semantic representation corresponding to the i-th token, is the concatenation operation, and Relu is the activation function.

[0068] The second step: multimodal token pair association representation, taking into account the relational independence of the span itself and the relational dependence of the span pair, respectively model the two-stage token pair association representation before and after the embedding relationship.

[0069] The present invention first establishes a sequence subtask before the embedding relationship for the token pair association representation. For the tokens before the embedding relationship, the token pairs within the span and the token pairs between spans are treated as two different modal spaces. From the perspective of whether the token pairs are associated, whether they are associated within the span or between spans, a subtask is established - multimodal token pair association representation. The process is as follows:

[0070] The token pair association pattern within a span is expressed as:

[0071]

[0072] The token pair association pattern between spans is expressed as:

[0073]

[0074] Among them, W correlation and b correlation is the training parameter of the association process, σ is the sigmoid activation function, x interij and x exterij They represent the associated representation of the token pair within the span and the associated representation of the token pair between spans corresponding to the i-th token and the j-th token respectively.

[0075] The third step: facing the global semantic consistency constraint of span pairs, the present invention regards text semantics as the fusion representation of multiple relation triples contained therein, and the relationship between entity pairs depends on the text context in which the two are located. Therefore, the present invention embeds the entity pairs in the context of the full text, and considers the constraints of the entity pairs after the embedded relationship under the global semantics to reduce the generation probability of irrelevant entity pairs, thereby improving the encoding performance of valid relation triples. The process can be expressed as follows:

[0076] The present invention uses the maximum pooling of sequence tokens as the semantic representation of the full text:

[0077] seq = maxpool(X) (7) The token pair of the embedding relationship is expressed as:

[0078]

[0079] The global semantic consistency constraint subtask for entity pairs can be expressed as:

[0080] g ij =σ(W g T ij +b g ) (9)

[0081] Among them, maxpool() is the element-level maximum pooling operation, W rand b r They are the trainable parameters of the entity pair representation process of the embedded relationship, dropout is the random probability connection loss function, seq is the semantic representation of the full text of the sequence, and x fusei and x fusej are the semantic representations of the ith and jth tokens after integrating global semantics and local viscosity, T ij is the entity pair representation of the i-th token and the j-th token after embedding the relationship, W g and b g is a globally constrained trainable parameter, g ij is the global semantic consistency score of the entity pair after embedding the relation.

[0082] Step 4: Lightweight relation triple encoding based on boundary bidirectional mapping. In the decoding stage, bidirectional head-head and tail-tail boundary double-level mapping (head represents the start token of the span, and tail represents the end token of the span) is used to solve the problem of nested entities and nested entity pairs.

[0083] The present invention considers the boundary conditions of span constraints, takes the head-head and tail-tail associations between span pairs and the head-tail associations of the same span as the necessary and sufficient conditions for the decoding of constrained entity pairs, and uses binary classification tasks with different association modes to realize the lightweight encoding of relation triples. That is, the present invention uniformly characterizes the promotion effect of span recognition tasks on relation extraction tasks and the relationship extraction on span recognition tasks. For relationship extraction, considering the head entity and tail entity in the relationship triple that may contain multiple tokens, the present invention uses boundary mapping to locate the head-head correspondence and tail-tail correspondence of the head entity and the tail entity; the corresponding span recognition present invention locates the head-tail correspondence of the entity. In this way, the present invention not only incorporates the same span information but also integrates the association relationship between different spans, and synchronously models the association between the same span and different spans. On the one hand, this association of different modes synchronously models the mutual promotion effect between span and relation extraction, and on the other hand, it also ensures the accuracy of decoding of relation entity pairs under the same span association constraints in the subsequent decoding process, and effectively filters out non-span association noise. The modeling process of the association between different spans and the association with the same span is as follows:

[0084] The cross-entity head-to-head token pair encoding process of relational constraints can be expressed as:

[0085] score ss =σ(W rt (W ss T ij +b ss )+b rt ) (10)

[0086] The cross-entity tail-tail token pair encoding process of relational constraints can be expressed as:

[0087] score ee =σ(W rt (W ee T ij +b ee )+b rt ) (11)

[0088] The head-tail association encoding process of entities with the same span can be expressed as:

[0089] score se =σ(W rt (W se T ij +b se )+b rt ) (1)

[0090] Among them, W ss , b ss , W ee , b ee , W se and b se are the trainable parameters of the token pairs associated head-to-head, tail-to-tail, and head-to-tail within the same span, respectively, W rt and b rt is a trainable parameter of the relational constraint, score ss 、score ee and score se They are the token pair scores for the head-to-head and tail-to-tail relationships between spans and the head-to-tail relationships within the same span.

[0091] Step 5: Decoding of relational triples: The decoding process adopts a boundary bidirectional mapping mechanism from head-head mapping to tail-tail mapping and tail-tail mapping to head-head mapping, and is supplemented by the same span constraint to extract entity pairs and subordinate relationships (entity, relationship, entity) associated in the form of tuples. The present invention uses the head-tail constraint of the same span token to associate cross-entity token pairs to determine the composition of relational triples, searches from the head-head association position to the tail-tail association position of the nearest neighbor based on the nearest neighbor matching, and is supplemented by the same span head-tail association constraint to screen the noise span association prediction, effectively improving the extraction accuracy. In this process, the present invention uses the bidirectionality of head-head-tail-tail and tail-tail-head-head of different spans to solve the nesting problem of entity pairs. This lightweight decoding strategy can be used for relational triple extraction tasks of different nesting types such as nested spans and nested entity pairs. Figure 2An example of span-constrained boundary bidirectional mapping decoding is given, where SS represents the association between the head entity start token and the tail entity token, EE represents the association between the head entity end token and the tail entity end token, and SE represents the association between the start token and the end token of the span. The specific process can be described as a one-way decoding channel formed from the SS-EE process, and another decoding channel formed from the EE-SS process, and SE verification is used in this process to ensure that the constructed decoding channel belongs to the decoding order between normal spans, effectively screen out non-span-associated noise decoding, and effectively deal with different types of nested problems.

[0092] Finally, based on the loss2 of the association representation subtask based on multimodal tokens before relation embedding, the loss3 of the global semantic consistency subtask based on relation tokens after relation embedding, and the loss1 of the relation triple encoding subtask, a multi-task learning joint optimization loss based on binary cross entropy loss is established:

[0093] L=loss1+λ1·loss2+λ2·loss3

[0094] In this embodiment, the hyper-parameter is set to λ1=λ2=0.01.

[0095] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for extracting text relation triples based on multimodal association representation and entity pair global semantic consistency, characterized in that the steps include: For a given text sequence, obtain a global semantic representation, and use the sticky features of the adjacent spans as a local semantic representation; fuse the global semantic representation with the local semantic representation to form a token sequence semantic representation; Before relation embedding, token pairs within a span and token pairs between spans are treated as two different modal spaces to obtain multimodal token pair association representations. After the relationship is embedded, the token pairs of the embedded relationship are constrained for global semantic consistency towards entity pairs, and lightweight relationship triple encoding based on boundary bidirectional mapping is performed considering the boundary conditions of span constraints; The nested relation triples are decoded based on the boundary bidirectional mapping mechanism to obtain the relation triples of the given text sequence.

2. According to claim 1, a method for extracting text relationship triples based on multimodal association representation and entity pair global semantic consistency is characterized in that: The method uses a pre-trained language model Bert to encode the sequence to generate context-related word vector representations as a global token sequence semantic representation at the text level.

3. According to claim 2, a method for extracting text relationship triples based on multimodal association representation and entity pair global semantic consistency is characterized in that: The viscosity characteristics based on the adjacent spans are described as follows: Front adjacent viscosity characteristics: Post-adjacent viscosity characteristics: Among them, W before , b before and W after , b after are the trainable parameters of adjacent tokens before and after fusion, x i Represents the embedding representation corresponding to the i-th token in the sequence.

4. According to claim 3, a method for extracting text relationship triples based on multimodal association representation and entity pair global semantic consistency is characterized in that: The sequence semantic representation that integrates global semantics and local viscosity is expressed as: Among them, x prei and x posti are the front adjacent viscosity characteristics and the rear adjacent viscosity characteristics formed, W fuse and b fuse To fuse the trainable parameters of global semantics and local viscosity, is the concatenation operation, and Relu is the activation function.

5. According to claim 1, a method for extracting text relationship triples based on multimodal association representation and entity pair global semantic consistency is characterized in that: For the multimodal token pair association representation, the token pairs within a span and the token pairs between spans are treated as two different modal spaces: The token pair association representation of the token pair association pattern within the span is expressed as: The token pair association representation of the token pair association pattern between spans is expressed as: Among them, W correlation and b correlation is the training parameter of the association process, σ is the sigmoid activation function; x fusei is the fused sequence semantic representation corresponding to the i-th token; x interij and x exterij They represent the associated representation of the token pair within the span and the associated representation of the token pair between spans corresponding to the i-th token and the j-th token respectively.

6. The method for extracting text relation triples based on multimodal association representation and entity pair global semantic consistency according to claim 1, characterized in that: The global semantic consistency constraints for entity pairs are specifically: g ij =σ(W g T ij +b g ) Among them, g ij is the global semantic consistency score of the entity pair after embedding the relation; W g and b g is a globally constrained trainable parameter; T ij The entity pair representation of the i-th token and the j-th token after the embedding relationship: Among them, dropout is a random probability connection loss function; x fusei and x fusej are the semantic representations of the ith and jth tokens respectively, which are the fusion of global semantics and local viscosity; W r and b r They are respectively the trainable parameters of the entity pair representation process of the embedded relationship; seq is the semantic representation of the full text of the sequence, and is the maximum pooling of the sequence token.

7. The method for extracting text relation triples based on multimodal association representation and entity pair global semantic consistency according to claim 1, characterized in that: The lightweight relation triple encoding based on boundary bidirectional mapping adopts a lightweight relation triple label strategy: a head-head and tail-tail entity pair unique deterministic strategy is adopted to establish directed secondary labels for relation entity pairs in short text data; and considering the boundary conditions of span constraints, the head-head and tail-tail associations between span pairs and the head-tail associations with the same span are used as sufficient and necessary conditions for decoding constrained entity pairs, and a binary classification task with different association patterns is adopted to realize lightweight encoding of relation triples.

8. The method for extracting text relation triples based on multimodal association representation and entity pair global semantic consistency according to claim 7, characterized in that: In the lightweight relation triple encoding based on boundary bidirectional mapping: The cross-entity head-to-head token pair encoding process of relation constraints is expressed as: score ss =σ(W rt (W ss T ij +b ss )+b rt ) The cross-entity tail-tail token pair encoding process of relation constraints is expressed as: score ee =σ(W rt (W ee T ij +b ee )+b rt ) The head-tail association encoding process of entities with the same span is expressed as: score se =σ(W rt (W se T ij +b se )+b rt ) Among them, W ss , b ss , W ee , b ee , W se and b se are the trainable parameters of the token pairs associated head-to-head, tail-to-tail, and head-to-tail within the same span, respectively, W rt and b rt is a trainable parameter of the relational constraint, score ss 、score ee and score se They are the token pair scores for the head-to-head and tail-to-tail relationships between spans and the head-to-tail relationships within the same span.

9. The method for extracting text relation triples based on multimodal association representation and entity pair global semantic consistency according to claim 8, characterized in that: The decoding based on boundary bidirectional mapping adopts the association of cross-entity token pairs with the same span head-tail constraint to determine the relationship triple composition, searches from the head-head association position to the tail-tail association position of the nearest neighbor based on the nearest neighbor matching, and is supplemented by the same span head-tail association constraint; extracts the entity pairs and subordinate relations associated in the form of tuples, where: The decoding process adopts a boundary bidirectional mapping mechanism from head-to-head mapping to tail-to-tail mapping and from tail-to-tail mapping to head-to-head mapping. The process from head-to-head mapping to tail-to-tail mapping constitutes a unidirectional decoding channel, and the process from tail-to-tail mapping to head-to-head mapping constitutes another decoding channel. In this process, the association verification of the start token and the end token of the span is supplemented to ensure that the constructed decoding channel belongs to the normal decoding order between spans.

10. The method for extracting text relation triples based on multimodal association representation and entity pair global semantic consistency according to claim 1, characterized in that: The method is based on the association representation subtask loss based on multimodal tokens before relation embedding, the global semantic consistency constraint subtask loss based on relation tokens after relation embedding, and the relation triple encoding subtask loss, and establishes a multi-task learning joint optimization based on binary cross entropy loss.

Images