A heterogeneous modality fusion semantic structure learning method for relation extraction

Abstract

The application relates to the technical field of natural language processing and artificial intelligence, in particular to a heterogeneous modal fusion semantic structure learning method for relation extraction, which comprises the following steps: performing entity feature construction on input text; performing heterogeneous modal representation learning based on an entity feature sequence, outputting continuous semantic modal representation and discrete structure modal representation; constructing a dynamic semantic association relationship based on the continuous semantic modal representation, and performing continuous semantic structure propagation based on the dynamic semantic association relationship; performing discrete structure constraint learning based on the discrete structure modal representation; and performing dynamic collaborative fusion on the continuous semantic structure representation and the discrete structure representation, adaptively adjusting the modal contribution degree of the continuous semantic structure representation and the discrete structure representation according to input semantic information and performing relation classification. The application is used for realizing collaborative learning of continuous semantics and discrete structure semantics, and improving target entity relation recognition capability in a complex semantic scene.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and artificial intelligence, specifically to a heterogeneous modality fusion semantic structure learning method oriented towards relation extraction. Background Technology

[0002] Relation extraction (RE) is an important task in the field of natural language processing. Its goal is to identify semantic relationships between target entity pairs from unstructured text and convert them into structured knowledge representations. It is widely used in knowledge graph construction, intelligent question answering, medical text analysis, and information retrieval. Existing relation extraction methods mainly include methods based on continuous semantic modeling, methods based on graph structure modeling, and methods based on structural feature fusion.

[0003] Methods based on convolutional neural networks, recurrent neural networks, Transformers, and pre-trained language models primarily learn contextual semantic representations of text through continuous vector spaces, which can effectively model global semantic dependencies. However, they lack the ability to explicitly express local structural patterns and discrete structural semantics. Methods based on graph neural networks and graph convolutional networks learn structural propagation through syntactic dependency graphs, enhancing the ability to model dependencies between entities. However, their propagation process typically relies on continuous weight updates, which can easily lead to noise propagation and over-smoothing of features, resulting in semantic confusion in scenarios with complex syntax, long-distance dependencies, and overlapping entities. Existing feature fusion methods mostly employ feature concatenation, attention weighting, and static weight fusion, which are essentially still homogeneous continuous feature aggregations, making it difficult to effectively handle the heterogeneous differences between continuous semantics and discrete structural semantics.

[0004] Therefore, existing technologies mainly suffer from the following problems: continuous semantic modeling lacks explicit structural semantic expression capabilities; existing fusion mechanisms lack heterogeneous modality collaborative modeling capabilities; there is a lack of fine-grained semantic modeling mechanisms oriented towards target entities; and the models lack robustness and interpretability in complex semantic scenarios. The main reason for this is that existing methods rely excessively on modeling a single continuous semantic space, lack a unified collaborative learning framework between continuous semantics and discrete structural semantics, and cannot dynamically adjust the importance of different modalities according to different text semantics.

[0005] To address the aforementioned issues, this patent proposes a heterogeneous modality fusion semantic structure learning method oriented towards relation extraction, which enables collaborative learning of continuous semantics and discrete structural semantics, thereby improving the ability to identify target entity relationships in complex semantic scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a heterogeneous modality fusion semantic structure learning method for relation extraction in order to solve the above-mentioned technical problems. This method is used to achieve collaborative learning of continuous semantics and discrete structural semantics, thereby improving the ability to identify target entity relations in complex semantic scenarios.

[0007] The objective of this invention can be achieved through the following technical solutions: A heterogeneous modality fusion semantic structure learning method oriented towards relation extraction includes: S1. Construct entity features from the input text to obtain an entity feature sequence, which includes entity label features, relative position features, and entity attribute features. S2. Learn heterogeneous modal representations based on entity feature sequences, and output continuous semantic modal representations and discrete structural modal representations; S3. Construct dynamic semantic associations based on continuous semantic modal representations, and propagate continuous semantic structures based on dynamic semantic associations to generate continuous semantic structure representations; S4. Based on the discrete structure modal representation, perform discrete structure constraint learning to generate discrete structure representation; S5. Dynamically and collaboratively fuse the continuous semantic structure representation and the discrete structure representation, adaptively adjust the modal contribution of the continuous semantic structure representation and the discrete structure representation according to the input semantic information, and output the fused semantic representation. S6. Classify relationships based on fused semantic representation and output the relationship categories between target entities.

[0008] Furthermore, step S1 includes: Add entity boundary markers to the target entities in the input text to obtain entity marker features; Calculate the distance between each word in the input text and the starting position of the target entity to obtain the relative position features; Extract the entity type, entity subtype, entity core word, entity order information, and boundary part-of-speech information of the target entity to obtain entity attribute features; Entity feature sequences are constructed based on entity label features, relative position features, and entity attribute features.

[0009] Furthermore, step S1 also includes: Pooling is performed on the features within the entity span corresponding to the target entity to obtain the entity span representation; By concatenating and mapping the span representations of two entities, we obtain the entity pair representation; The entity pair representation is broadcast to each lexical position in the entity feature sequence to obtain an entity feature sequence containing a global index of entity pairs.

[0010] Furthermore, S2 includes: A pre-trained language model is used to perform context encoding on the text sequence in the entity feature sequence to obtain a continuous semantic modality representation; A randomly initialized and trainable embedding lookup table is used to embed and map discrete structural features in entity feature sequences to obtain discrete structural modal representations. By concatenating the continuous semantic modal representation and the discrete structural modal representation, an initial heterogeneous modal representation is obtained.

[0011] Furthermore, step S3 includes: Perform a first linear mapping on the continuous semantic modal representation to obtain the query feature matrix; A second linear mapping is performed on the continuous semantic modal representation to obtain the key feature matrix; The semantic association strength between terms is calculated based on the query feature matrix and the key feature matrix using a scaled dot product attention mechanism. in, Represents the query feature matrix. Represents the key feature matrix, Representing feature dimension, This represents the normalization function; the output is the dynamic semantic association matrix. .

[0012] Furthermore, step S3 also includes: A self-connect matrix is ​​added to the dynamic semantic association matrix to obtain an enhanced semantic association matrix; a degree matrix is ​​constructed based on the enhanced semantic association matrix. Based on the graph convolution mechanism, the enhanced semantic association matrix, degree matrix, and continuous modality learning parameter matrix are propagated through a graph structure to obtain a continuous semantic structure representation: in, For continuous feature dimensions, Let L be the discrete feature dimension, and L represent the length of the input sequence. This represents the identity matrix, used to introduce self-joins. It is a degree matrix; This is the parameter matrix for continuous modal learning; ( It is a non-linear activation function. This is the input feature matrix of the previous layer.

[0013] Furthermore, step S4 includes: Construct a trainable weight matrix; The trainable weight matrix is ​​mapped to a predefined finite set of discrete states, which is restricted to the set of discrete weights. The discrete weight matrix is ​​obtained. , is represented as: in, This is a quantization threshold used to control the sparsity of the weights; This is the scaling factor; Discrete structure transformation based on discrete weight matrix for discrete structural modal representation: in, This is the discretized weight matrix; As a bias term, a discrete structure representation is obtained.

[0014] Furthermore, step S5 includes: Calculation of discrete modal weight coefficients based on continuous semantic structure representation and discrete structure representation: in, This is used to indicate the degree of dependence of the current lexical unit on the discrete mode. It is a linear mapping function; The discrete structural representation is weighted based on discrete modal weight coefficients to obtain a weighted discrete structural representation; the continuous semantic structural representation and the weighted discrete structural representation are then fused. in, This represents element-wise multiplication, and the output structure is a fused representation. ; Calculate the fusion gate vector based on structural fusion representation. : Gated residual fusion is performed on the input representation corresponding to the entity feature sequence based on the fusion gate vector: Output fused semantic representation .

[0015] Furthermore, step S5 also includes: The fused semantic representation is subjected to nonlinear feature fusion processing to obtain the output representation of the current layer: in, For feedforward network parameters, This is a bias term.

[0016] Furthermore, step S6 includes: For the process The output representation after dynamic layer-by-layer collaborative fusion is subjected to max pooling to obtain global discriminative features: in, Indicates the first Layer output representation; Semantic encoding is performed based on global discriminative features to obtain encoded features: in, This represents the parameter matrix of the fully connected layer. Indicates the bias parameter; Relationship classification is performed based on encoded features to obtain the predicted relationship probability: in, Represents the classification parameter matrix, This represents the classification bias parameter.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention constructs a joint learning framework for continuous semantic modal and discrete structural modal, and combines it with a dynamic collaborative fusion mechanism to achieve collaborative modeling of continuous semantic structure and discrete structural semantic, thereby improving the relational semantic modeling capability and fusion stability in complex semantic scenarios.

[0018] (2) This invention improves the expressive power of target entity related semantic features and the relationship recognition ability in complex multi-entity scenarios by introducing entity boundary marker features, entity relative position features, entity attribute features and entity pair global index in the input stage through entity feature sequence construction mechanism.

[0019] (3) This invention improves the semantic structure modeling capability in complex syntactic structures and long-distance dependency scenarios by using a dynamic semantic association matrix construction mechanism and a continuous semantic structure propagation mechanism to dynamically generate semantic association relationships between word units based on input semantics.

[0020] (4) This invention uses a discrete structure constraint learning mechanism and an end-to-end joint optimization mechanism to jointly optimize the continuous modality learning parameters, discrete modality learning parameters, dynamic collaborative fusion parameters and classification parameters, thereby improving the local structural feature expression ability, robustness, generalization ability and adaptability of the relation extraction model in different language environments. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the model structure for implementing the heterogeneous modality fusion semantic structure learning method for relation extraction according to the present invention. Detailed Implementation

[0022] The present embodiment will now be described in detail with reference to the accompanying drawings and specific examples. This embodiment is implemented based on the technical solution described below, and provides detailed implementation methods and specific operating procedures. However, the scope of protection of this embodiment is not limited to the following embodiments.

[0023] like Figure 1 The method for learning heterogeneous modal fusion semantic structure oriented by relation extraction, as shown, includes: S1. Construct entity features from the input text to obtain an entity feature sequence, which includes entity label features, relative position features, and entity attribute features; specifically, step S1 includes: Let the input sentence be [ , , …, ],in: Sentence The i-th word, N represents the sentence length. Its target entity pairs are... , …, [ , …, To enhance the model's ability to focus on the relationships between target entities: generate entity tag features. : Explicitly label the target entity using entity boundary markers: <e1> ,<\e1>, <e2>The comma (<e2>) is used to highlight the location of the target entity and guide the model to focus on the entity's boundaries and its local contextual semantics. For example: The company <e1>Apple<\e1>acquired <e2>Beats<\e2>.

[0024] The relative position feature P is obtained by calculating the distance of each word in the input text relative to the starting position of the target entity. in, Indicates the position index of the current word in the sentence. Representing entities The index of the starting position. Representing entities The index of the starting position. This positional feature is used to enhance the ability to model the locality of relationships; Extract the entity type, entity subtype, entity headword, entity order information, and boundary part-of-speech information of the target entity to obtain entity attribute features; where the entity type is represented as: , For example, PERSON, LOCATION, ORGANIZATION. Entity subtypes are represented as: , This is used to provide more fine-grained semantic information. Entity center words are represented as: , This is used to represent the syntactic core of entities. Entity pair structural features are represented as follows: Introducing entity order information: This is used to represent the subject-object positional relationship between entities. Boundary part-of-speech features are represented as follows: extracting the part-of-speech information of the left and right adjacent words of an entity. , , , , used to enhance the expressive power of local grammatical structures.

[0025] Based on entity label features, relative position features, and entity attribute features, a feature sequence is constructed for each input instance, ultimately forming an entity center feature sequence: Pooling is performed on the features within the entity span corresponding to the target entity to obtain the entity span representation: By concatenating and mapping the span representations of two entities, we obtain the entity pair representation: Broadcasting entity pair representations to each lexical position in the entity feature sequence yields an entity feature sequence containing a global index of entity pairs: in, Represents the first target entity The entity representation vector, Indicates the second target entity The entity representation vector; Representing entities Within the lexical range of a sentence, Representing entities The range of words in a sentence; Represents multimodal input features. This indicates that the entity is broadcasting to the vector.

[0026] Step S1 addresses the problem of target relationship focus being diluted by contextual noise in traditional models by reconstructing entity features, making the model pay more attention to semantic associations between target entities.

[0027] S2. Based on entity feature sequences, perform heterogeneous modality representation learning, outputting continuous semantic modality representation and discrete structural modality representation. Specifically: Using a pre-trained language model PLM to process entity feature sequences Context encoding is performed on the text sequence to obtain continuous semantic modality representation. ;in, is a continuous feature dimension. This representation is used to model long-distance contextual semantic dependencies.

[0028] Discrete structural features in entity feature sequences are embedded and mapped using a randomly initialized and trainable embedding lookup table to obtain discrete structural modal representations. ;in, This refers to discrete feature dimensions. Discrete features include: relative position, entity type, entity subtype, entity center word, entity order, and boundary part-of-speech tagging. By concatenating the continuous semantic modal representation and the discrete structural modal representation, an initial heterogeneous modal representation is obtained. This enables the global injection of target entity relationship focus and unified encoding of multi-source heterogeneous semantic features, avoiding interference from other entities with entity relationship information in complex sentences, and providing a shared input foundation for subsequent continuous and discrete modal learning.

[0029] S3. Construct dynamic semantic associations based on continuous semantic modal representations, and propagate continuous semantic structures based on these dynamic semantic associations to generate continuous semantic structure representations. To model long-distance semantic dependencies between target entities in a sentence, a graph-based continuous learning (GCL) branch is constructed in the continuous modal space to learn global semantic associations between lexical units in the input text. Specifically: Let the input features obtained in step 2 be represented as: in: Indicates the length of the input sequence; Indicates the current fusion layer number; , and These represent the continuous modal dimension, the discrete modal dimension, and the entity pair representation dimension, respectively.

[0030] Instead of using a fixed syntactic dependency tree structure, the semantic adjacency matrix is ​​dynamically constructed based on the input representation to enhance the model's adaptive modeling ability for task-related semantic relationships. Perform a first linear mapping on the continuous semantic modal representation to obtain the query feature matrix. A second linear mapping is performed on the continuous semantic modal representation to obtain the key feature matrix. ; in, and It is a trainable linear mapping function; The semantic association strength between terms is calculated based on the query feature matrix and the key feature matrix using a scaled dot product attention mechanism. in, Represents the query feature matrix. Represents the key feature matrix, Representing feature dimension, This represents the normalization function; the output is the dynamic semantic association matrix. Compared to fixed syntax tree methods, this embodiment achieves adaptive learning of semantic relationships through a dynamic adjacency matrix, which can automatically adjust the entity relationship modeling path according to different sentence content.

[0031] A self-connect matrix is ​​added to the dynamic semantic association matrix to obtain an enhanced semantic association matrix; a degree matrix is ​​then constructed based on the enhanced semantic association matrix. Based on the graph convolution mechanism, the enhanced semantic association matrix, degree matrix, and continuous modality learning parameter matrix are propagated through a graph structure to obtain a continuous semantic structure representation: in, For continuous feature dimensions, Let L be the discrete feature dimension, and L represent the length of the input sequence. This represents the identity matrix, used to introduce self-joins. It is a degree matrix; This is the parameter matrix for continuous modal learning; ( It is a non-linear activation function. This is the input feature matrix for the previous layer. By learning long-distance lexical dependencies in sentences through dynamic graph convolution, we can effectively capture cross-entity contextual semantic connections, thereby enhancing the modeling ability of relation trigger words, modifiers, and implicit dependency structures, while preserving fine-grained semantics of the context.

[0032] S4. Based on discrete structural modal representation, perform discrete structural constraint learning to generate discrete structural representations; construct a trainable weight matrix; to enhance the model's ability to express local structural information, sparse semantic features, and high-contrast relational patterns, this embodiment constructs a Discrete Convolution Learning (DCL) branch, which learns relational representations with structural interpretability through a discrete constraint mechanism; specifically: The trainable weight matrix is ​​mapped to a predefined finite set of discrete states, which is restricted to the set of discrete weights. The discrete weight matrix is ​​obtained. , is represented as: in, This is a quantization threshold used to control the sparsity of the weights; This is the scaling factor; Discrete structure transformation based on discrete weight matrix for discrete structural modal representation: in, This is the discretized weight matrix; As a bias term, a discrete structural representation is obtained. By learning sparse, high-contrast local semantic patterns through discrete constraints, the model pays more attention to key structural cues in the entity neighborhood, reduces feature smoothing in continuous representations, and improves the stability and interpretability of relation discrimination.

[0033] S5. Dynamically and collaboratively fuse the continuous semantic structure representation and the discrete structure representation, adaptively adjusting the modal contribution levels of the continuous and discrete semantic structure representations based on the input semantic information, and outputting the fused semantic representation; specifically, calculating the discrete modal weight coefficients based on the continuous and discrete semantic structure representations: in, This is used to indicate the degree of dependence of the current lexical unit on the discrete mode. It is a linear mapping function; The discrete structural representation is weighted based on discrete modal weight coefficients to obtain a weighted discrete structural representation; the continuous semantic structural representation and the weighted discrete structural representation are then fused. in, This represents element-wise multiplication, and the output structure is a fused representation. ; Calculate the fusion gate vector based on structural fusion representation. : Gated residual fusion is performed on the input representation corresponding to the entity feature sequence based on the fusion gate vector: Output fused semantic representation .

[0034] To further enhance feature representation capabilities, a nonlinear feature fusion process is applied to the fused semantic representation to obtain the output representation of the current layer: in, For feedforward network parameters, This is the bias term. Nonlinear transformations further enhance feature abstraction capabilities, improve the model's ability to represent complex relationship patterns, and alleviate gradient degradation problems during deep model training.

[0035] S6. Classify relationships based on fused semantic representation and output the relationship categories between target entities; For the process The output representation after dynamic layer-by-layer collaborative fusion is subjected to max pooling to obtain global discriminative features: in, Indicates the first Layer output representation; Semantic encoding is performed based on global discriminative features to obtain encoded features: in, This represents the parameter matrix of the fully connected layer. Indicates the bias parameter; Relationship classification is performed based on encoded features to obtain the predicted relationship probability: in, Represents the classification parameter matrix, This represents the classification bias parameter. Through high-order semantic features obtained from multimodal fusion, the target entity can accurately determine the relationship category.

[0036] During the model training phase, this embodiment employs supervised learning to jointly optimize the model parameters.

[0037] Let the real relationship label be The predicted probability is: The objective optimization function is then defined as the cross-entropy loss function: in: Indicates the total number of relation categories; For the actual label indicator variable; Predict probabilities for the model.

[0038] The following parameters are jointly updated using the gradient backpropagation algorithm: continuous modality learning parameters, discrete modality learning parameters, bimodal fusion parameters, and classification parameters. End-to-end joint optimization improves the overall relation extraction performance of the model and enhances its generalization ability to complex entity relation scenarios. Compared with existing relation extraction methods based on single continuous semantic modeling or single structural feature learning, this embodiment proposes a heterogeneous modality fusion semantic structure learning method (DMFN) for relation extraction. By constructing a bimodal collaborative learning framework of "continuous semantic dependency + discrete structural pattern," it achieves refined modeling of entity relations and exhibits good performance and stability in complex semantic structures, cross-linguistic data distributions, and multi-relation scenarios.

[0039] This embodiment was validated on multiple public relation extraction datasets, including the English dataset (ACE 2005English, SemEval-2010 Task 8), the Chinese dataset (ACE 2005 Chinese), and the Korean dataset (Ko-RE). A unified training configuration was used, including: a maximum sentence length of 150, the Adam optimizer, a batch size of 32, and 100 training epochs. Experimental results show that this embodiment has good effectiveness and generalization ability under different languages, different relation sizes, and complex entity structure scenarios. Its specific advantages and beneficial effects are as follows: (1) Dual-modal collaborative learning significantly improves relation extraction accuracy Traditional relation extraction methods typically rely on a single continuous context representation, which can easily overlook discrete structural features around entities; or they may only utilize regular structural information, making it difficult to fully learn long-distance semantic dependencies, resulting in insufficient entity relation discrimination capabilities.

[0040] This embodiment constructs a joint learning mechanism of continuous modality (global semantic dependency) and discrete modality (local structural pattern) to achieve collaborative modeling of heterogeneous semantic structures under a unified framework, enabling the model to simultaneously capture "1. long-distance contextual semantic dependency between entities", "2. local structural patterns in the neighborhood of entities", and "3. discriminative semantic features of target entity relationships".

[0041] Experimental results show that this embodiment achieves superior performance on multiple public datasets, for example: 1) The F1 score on the ACE EN dataset reaches over 91%; 2) The F1 score on the SemEval dataset reaches over 95%; 3) The F1 score on the ACE CN dataset reaches over 93%; 4) The F1 score on the Ko-RE dataset reaches approximately 77%.

[0042] Compared with traditional CNN, GCN and methods based on single representation learning, this embodiment shows higher accuracy in complex entity relationships and overlapping semantic scenarios, verifying the effectiveness of the dual-modal collaborative learning framework.

[0043] (2) Semantic encoding of entity features effectively alleviates the problem of focus dilution of target relations. Existing relation extraction models typically encode the original sentence uniformly, which easily leads to ambiguity in relation focus in complex multi-entity sentences, meaning the model struggles to accurately focus on the semantic relationships between target entity pairs. This embodiment addresses this by introducing an entity feature encoding mechanism at the input stage, incorporating "1. Entity boundary marker information," "2. Entity attribute features," "3. Entity relative position features," and "4. Global index representation of entity pairs." This allows the model to explicitly focus on target entity pairs from the initial encoding stage, increasing the weight of contextual information relevant to relation discrimination. Experimental results demonstrate that this mechanism significantly reduces semantic interference in complex sentences and improves the model's relation recognition accuracy in multi-entity scenarios.

[0044] (3) Dynamic graph continuous structure learning enhances long-distance semantic dependency modeling ability Existing relation extraction methods based on Graph Convolutional Networks (GCNs) typically rely on external parsers to generate fixed dependency trees. Their performance is easily affected by syntactic analysis errors and they struggle to adapt to different task requirements. This embodiment utilizes a dynamic graph structure induction mechanism to dynamically construct a semantic adjacency matrix between lexical units based on self-attention. This enables the model to: 1. Automatically learn implicit structural connections relevant to relation classification tasks; 2. Dynamically adjust information propagation paths between entities; and 3. Effectively model long-distance dependencies. Compared to fixed syntactic structure methods, this embodiment exhibits stronger task adaptability and structural expressiveness, making it particularly suitable for complex long-text relation extraction scenarios.

[0045] (4) Discrete structure learning effectively enhances the ability to express local discriminative patterns. While continuous representation learning can capture global semantic information, it is prone to weakening local discriminative structural information due to feature smoothing. To address this issue, this embodiment designs a discrete structure learning branch, which learns high-contrast semantic patterns through finite discrete state weight constraints, achieving "1. Local structural semantic enhancement", "2. Sparse discriminative feature extraction", "3. Non-critical noise information suppression", and "4. Explicit modeling of relation structure boundaries". Experimental results show that the model performance significantly decreases after removing the discrete modality branch, indicating that discrete structure learning makes a significant contribution to relation extraction.

[0046] (5) Adaptive gating fusion mechanism improves model robustness and stability Existing multi-feature fusion methods typically employ direct concatenation or fixed weighting, making it difficult to dynamically adjust the information contribution ratio based on different input semantics. This embodiment proposes an adaptive bimodal gated fusion mechanism, which achieves dynamic coordination of information between continuous and discrete modalities through "modal dynamic weight estimation" and "gated residual fusion," enabling the model to automatically adjust modality dependency strength according to different sentence complexities. Experimental results show that this mechanism significantly enhances model training stability and improves gradient propagation capability during deep structure learning. Ablation experiments further demonstrate that removing this fusion module significantly degrades model performance, validating the necessity and effectiveness of this technical solution.

[0047] (6) Possesses good cross-language generalization ability Because this embodiment uses a language-independent heterogeneous semantic structure learning framework, it can be applied to multiple language environments such as English, Chinese, and Korean.

[0048] Experimental results show that this embodiment performs stably on different language datasets, indicating that the method can effectively adapt to different grammatical structures and language distributions, and has good cross-language transfer capabilities and versatility.

[0049] (7) Compared with larger language models, it has higher efficiency and deployment stability. Compared to current relation extraction methods based on large language models, this embodiment has the following advantages: smaller parameter size; lower training and inference resource requirements; stable output results, independent of generative inference; and greater suitability for structured relation extraction tasks. Experimental results show that this embodiment achieves or outperforms the performance of some large language model fine-tuning schemes on multiple public datasets, while also exhibiting better engineering deployability.

[0050] (8) It has good feasibility for engineering implementation. This embodiment can complete training and deployment in a standard GPU environment, and has the following advantages: controllable training cost; strong model reproducibility; easy integration with named entity recognition, knowledge graph construction and other systems; and can be widely used in fields such as intelligent question answering, knowledge extraction, intelligence analysis, medical text analysis and Oracle knowledge structure construction.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0053] The present embodiment has been further described above with reference to specific embodiments. However, it should be understood that the specific descriptions herein should not be construed as limiting the substance and scope of the present embodiment. Various modifications made to the above embodiments by those skilled in the art after reading this specification are all within the scope of protection of the present embodiment. < / e1>

Claims

1. A heterogeneous modality fusion semantic structure learning method oriented towards relation extraction, characterized in that, include: S1. Construct entity features from the input text to obtain an entity feature sequence, which includes entity marker features, relative position features, and entity attribute features; S2. Based on the entity feature sequence, perform heterogeneous modal representation learning, and output continuous semantic modal representation and discrete structural modal representation; S3. Construct dynamic semantic associations based on the continuous semantic modal representations, and propagate continuous semantic structures based on the dynamic semantic associations to generate continuous semantic structure representations; S4. Based on the discrete structural modal representation, perform discrete structural constraint learning to generate a discrete semantic structural representation; S5. Dynamically and collaboratively fuse the continuous semantic structure representation and the discrete structure representation. Through modal dynamic weight allocation and gated residual fusion methods, adaptively adjust the modal contribution degree of the continuous semantic structure representation and the discrete semantic structure representation according to the input semantic information, and output the fused semantic representation. S6. Based on the fused semantic representation, perform relation classification and output the relation categories between the target entities.

2. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 1, characterized in that, Step S1 includes: Add entity boundary markers to the target entities in the input text to obtain entity marker features; Calculate the distance of each word element in the input text relative to the starting position of the target entity to obtain the relative position features; Extract the entity type, entity subtype, entity core word, entity order information, and boundary part-of-speech information of the target entity to obtain entity attribute features; The entity feature sequence is constructed based on the entity label features, the relative position features, and the entity attribute features.

3. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 2, characterized in that, Step S1 further includes: The features within the entity span corresponding to the target entity are pooled to obtain the entity span representation; The two entity span representations are concatenated and mapped to obtain entity pair representations; The entity pair representation is broadcast to each lexical position in the entity feature sequence to obtain an entity feature sequence containing a global index of entity pairs.

4. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 1, characterized in that, S2 includes: A pre-trained language model is used to perform context encoding on the text sequence in the entity feature sequence to obtain the continuous semantic modality representation; A randomly initialized and trainable embedding lookup table is used to embed and map discrete structural features in the entity feature sequence to obtain the discrete structural modal representation. The continuous semantic modal representation and the discrete structural modal representation are concatenated to obtain the initial heterogeneous modal representation.

5. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 1, characterized in that, Step S3 includes: A first linear mapping is performed on the continuous semantic modal representation to obtain the query feature matrix; A second linear mapping is performed on the continuous semantic modal representation to obtain the key feature matrix; The semantic association strength between terms is calculated based on the query feature matrix and the key feature matrix using a scaled dot product attention mechanism. in, Represents the query feature matrix. Represents the key feature matrix, Representing feature dimension, This represents the normalization function; the output is the dynamic semantic association matrix. .

6. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 5, characterized in that, Step S3 further includes: A self-connect matrix is ​​added to the dynamic semantic association matrix to obtain an enhanced semantic association matrix; a degree matrix is ​​then constructed based on the enhanced semantic association matrix. Based on the graph convolution mechanism, the enhanced semantic association matrix, the degree matrix, and the continuous modality learning parameter matrix are propagated through a graph structure to obtain the continuous semantic structure representation: in, For continuous feature dimensions, Let L be the discrete feature dimension, and L represent the length of the input sequence. This represents the identity matrix, used to introduce self-joins. It is a degree matrix; This is the parameter matrix for continuous modal learning; ( It is a non-linear activation function. This is the input feature matrix of the previous layer.

7. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 1, characterized in that, Step S4 includes: Construct a trainable weight matrix; The trainable weight matrix is ​​mapped to a preset finite set of discrete states, wherein the finite set of discrete states is limited to a set of discrete weights. To obtain discrete weights , represented as: in, This is a quantization threshold used to control the sparsity of the weights; This is the scaling factor; Based on the discrete weight matrix, perform a discrete structure transformation on the discrete structural mode representation: in, This is the discretized weight matrix; The discrete structure representation is obtained by using the bias term.

8. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 1, characterized in that, Step S5 includes: Calculate the discrete modality weight coefficients based on the continuous semantic structure representation and the discrete structure representation: in, This is used to indicate the degree of dependence of the current lexical unit on the discrete mode. It is a linear mapping function; The discrete structure representation is weighted based on the discrete modality weight coefficients to obtain a weighted discrete structure representation; the continuous semantic structure representation and the weighted discrete structure representation are then fused. in, This represents element-wise multiplication, and the output structure is a fused representation. ; Calculate the fusion gate vector based on the structural fusion representation. : Based on the fusion gate vector, gated residual fusion is performed on the input representation corresponding to the structural fusion representation and the entity feature sequence: Output the fused semantic representation .

9. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 8, characterized in that, Step S5 further includes: The fused semantic representation is subjected to nonlinear feature fusion processing to obtain the current layer output representation: in, For feedforward network parameters, This is a bias term.

10. The heterogeneous modality fusion semantic structure learning method for relation extraction according to claim 1, characterized in that, Step S6 includes: For the process The output representation after dynamic layer-by-layer collaborative fusion is subjected to max pooling to obtain global discriminative features: in, Indicates the first Layer output representation; Based on the global discriminative features, semantic encoding is performed to obtain the encoded features: in, This represents the parameter matrix of the fully connected layer. Indicates the bias parameter; Based on the encoded features, relationship classification is performed to obtain the relationship prediction probability: in, Represents the classification parameter matrix, This represents the classification bias parameter.

Images