A method and system for joint extraction of medical text relationships by combining BERT models and large-scale models.

Abstract

This invention relates to a method and system for jointly extracting relationships from medical text by combining a BERT model and a large-scale model, belonging to the field of medical information processing technology. It addresses the limitations of existing BERT-type models in extracting complex relationships from medical text. The method includes the following steps: acquiring the medical text information to be extracted; inputting the medical text information into a trained BERT model to obtain a sequence of feature vectors for the medical text information; inputting the sequence of feature vectors into a trained large-scale model to extract entity relationships from the medical text information; the large-scale model is based on a gated cross-attention mechanism. This improves the accuracy and contextual consistency of complex relationship extraction.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and in particular to a method and system for jointly extracting medical text relationships by combining a BERT model with a large model. Background Technology

[0002] Medical texts (such as clinical medical records and diagnostic reports) are generally characterized by dense technical terms, complex relational structures, and lengthy sentences, making the accurate extraction of entities and relationships from such texts extremely challenging. Current mainstream methods are primarily based on pre-trained language models such as BERT, and their implementations can be divided into two categories:

[0003] Pipeline approach: Named entity recognition (NER) is performed first, followed by relation classification (RC). This method suffers from error propagation, meaning that incorrect entity recognition leads to incorrect relation classification, and there is a lack of information exchange between the two sub-task models.

[0004] Joint extraction methods: These methods use a single model (such as BERT) to simultaneously complete entity recognition and relation extraction. While this alleviates error accumulation, it is usually modeled as a rigid structure such as sequence labeling or pointer networks, which has limited ability to handle complex scenarios such as overlapping and compound relationships. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a method and system for joint extraction of medical text relationships by combining BERT models and large models, so as to solve the limitations of existing BERT-like models in extracting complex relationships from medical texts.

[0006] On one hand, embodiments of the present invention provide a method for joint extraction of medical text relationships by combining a BERT model and a large model, including the following steps:

[0007] Obtain the medical text information to be extracted;

[0008] The medical text information is input into the trained BERT model to obtain the feature vector sequence of the medical text information;

[0009] The feature vector sequence of the medical text information is input into a trained large model to extract entity relationships from the medical text information; the large model is a large model with a gated cross-attention module inserted; the gated cross-attention module includes a preprocessing gated unit, an enhancement gated unit, a cross-attention unit, a filtering gated unit, and an adapter connected in sequence.

[0010] Based on the above method, a gated cross-attention module is added after the self-attention module of each Transformer structure layer in the large model and before the feedforward network to construct a large model based on the gated cross-attention mechanism.

[0011] Based on further improvements to the above method,

[0012] The preprocessing gating unit is used to perform preliminary screening and filtering on the input information to obtain preliminary filtering features;

[0013] The enhanced gating unit is used to generate an enhanced key matrix and an enhanced value matrix based on the initial filtering features;

[0014] The cross-attention unit is used to calculate the context vector based on the current query, the boosting key matrix, and the boosting value matrix.

[0015] The filtering gating unit is used to fuse the context vector and the current query and filter them to obtain the final filtering features;

[0016] The adapter is used to transform the final filtered features in the representation space and then use them as the output of the gated cross-attention module.

[0017] Based on a further improvement of the above method, the preprocessing gating uses the following method to perform preliminary screening and filtering of the input information to obtain preliminary filtering features:

[0018] ;

[0019] ;

[0020] in, This represents the learnable weight matrix. Indicates bias. This represents the sigmoid function. A sequence of feature vectors representing medical text information. Indicates importance score, This indicates the initial filtering features.

[0021] Based on a further improvement of the above method, the enhancement gating unit generates the enhancement key matrix and enhancement value matrix based on the initial filtering features in the following manner:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, Indicates the current query. Indicates average pooling. The dimension parameter represents the tensor operation. Indicates preliminary filtering features. This indicates average pooling query. Indicates the average pooling key value. Indicates the enhancement coefficient. Represents the augmented gating vector. Represents the tanh function. Indicates splicing, This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the enhanced bond projection matrix. Represents the augmentation value projection matrix. Represents the key projection matrix. Represents the value projection matrix, Indicates dimensional expansion, The feature dimension of each token in the feature vector sequence representing medical text information. This indicates the dimension of the enhancement bond projection matrix. This indicates the dimension of the augmentation projection matrix, and the superscript T indicates transpose. Represents the enhancement bond matrix, Represents the augmentation value matrix. This represents element-wise product.

[0031] Based on the further improvement of the above method, the filtering gating unit fuses and filters the context vector and the current query to obtain the final filtering features in the following manner:

[0032] ;

[0033] ;

[0034] ;

[0035] in, Indicates fusion characteristics, Represents the tanh function. Indicates splicing, Represents the context vector. Indicates the current query. The dimension parameter represents the tensor operation. This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the filtering gate vector. Indicates the final filtering features. This represents element-wise product.

[0036] Based on a further improvement of the above method, the cross-attention unit uses a multi-head attention mechanism to calculate the context vector based on the current query, the enhanced key matrix, and the enhanced value matrix.

[0037] Based on a further improvement of the above method, the adapter includes:

[0038] Dimensionality reduction layer, used to reduce and compress input data;

[0039] The recovery layer is used to restore the dimensions of the compressed data.

[0040] The residual layer is used to perform residual connections between the features output by the recovery layer and the final filtered features.

[0041] Layer normalization is used to normalize the features after residual connection as the output of the gated cross-attention module.

[0042] On the other hand, embodiments of the present invention provide a medical text relationship joint extraction system combining a BERT model and a large model, comprising the following steps:

[0043] The medical information acquisition module is used to acquire the medical text information to be extracted.

[0044] The feature vector sequence extraction module is used to input the medical text information into the trained BERT model to obtain the feature vector sequence of the medical text information.

[0045] The entity relation extraction module is used to input the feature vector sequence of the medical text information into a trained large model to extract the entity relations of the medical text information; the large model is a large model with a gated cross-attention module inserted; the gated cross-attention module includes a preprocessing gate unit, an enhancement gate unit, a cross-attention unit, a filtering gate unit, and an adapter connected in sequence.

[0046] Based on the further improvement of the above system, a gated cross-attention module is added after the self-attention module of each Transformer structure in the large model and before the feedforward network to construct a large model based on the gated cross-attention mechanism.

[0047] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0048] 1. The BERT model focuses on deep semantic encoding of raw medical text, providing high-quality semantic representations rich in contextual information, without having to undertake complex structural prediction tasks, thus capturing medical terms and contextual semantics more accurately.

[0049] 2. The large model does not directly process the original text, but rather generates structured extraction results based on the deep semantic representation provided by BERT and combined with prompting engineering. This fully leverages its powerful logical reasoning and generative capabilities.

[0050] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0051] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0052] Figure 1 This is a flowchart of a medical text relationship joint extraction method combining a BERT model and a large model, as described in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the Transformer structure with a gated cross-attention module added in an embodiment of the present invention;

[0054] Figure 3 This is a block diagram of a medical text relationship extraction system that combines a BERT model with a large model, according to an embodiment of the present invention. Detailed Implementation

[0055] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0056] A specific embodiment of the present invention discloses a method for joint extraction of medical text relationships by combining a BERT model and a large model, such as... Figure 1 As shown, it includes the following steps:

[0057] S1. Obtain the medical text information to be extracted;

[0058] S2. Input the medical text information into the trained BERT model to obtain the feature vector sequence of the medical text information;

[0059] S3. Input the feature vector sequence of the medical text information into the trained large model to extract the entity relationships of the medical text information; the large model is a large model with a gated cross-attention module inserted; the gated cross-attention module includes a preprocessing gated unit, an enhancement gated unit, a cross-attention unit, a filtering gated unit and an adapter connected in sequence.

[0060] Compared with existing technologies, the medical text relation extraction method combining BERT model and large model provided in this embodiment obtains the feature vector sequence of medical text information through the trained BERT model, providing rich semantic features for subsequent steps. Instead of the medical text information itself, the feature vector sequence is input into the trained large model for entity relation extraction. This allows the large model to adjust feature weights based on the feature vector sequence using gated cross-attention, avoiding semantic disconnect and outputting a more consistent output with the original text logic. This enhances the interpretability of the large model and improves the accuracy and contextual consistency of complex relation extraction.

[0061] In practice, a pre-trained BERT model can be used instead of a trained BERT model. The pre-training task could be, for example, a medical MLM task. MLM is a task that involves randomly masking parts of the input text, allowing the model to predict the masked words while observing the other unmasked words.

[0062] During implementation, medical text information is input into the trained BERT model, and features from the output of the last layer of the model are extracted. As a sequence of feature vectors for medical text information. For context-aware semantic representation.

[0063] The feature vector sequence of medical text information is used as one of the inputs to a large model for entity relation extraction. In implementation... It is in each layer of the Transformer structure of the input large model.

[0064] In practice, to improve the reasoning ability of the large model, the large model constructed in this invention is a large model based on a gated cross-attention mechanism.

[0065] Specifically, a gated cross-attention module is added to each layer of the Transformer structure in the large model to construct a large model based on the gated cross-attention mechanism. In implementation, the gated cross-attention module is inserted after the self-attention module and before the feedforward network in each Transformer layer. The Transformer structure with the gated cross-attention module in implementation is as follows: Figure 2 As shown. To ensure training stability, a corresponding layer normalization is added before the gated cross-attention module (as shown). Figure 2 LayerNorm2 in the module adds residual connections after the gated cross-attention module.

[0066] In implementation, a cross-attention layer is introduced into each Transformer structure, and a three-stage gating design (preprocessing, enhancement, and filtering) is added to the cross-attention mechanism. Through the three-stage gating design (preprocessing, enhancement, and filtering), the model can dynamically and selectively retrieve the information most relevant to the current layer's processing task from the source text feature vector sequence extracted by BERT, thereby achieving dynamic soft retrieval of key information in the source text and enhancing the interpretability of the large model.

[0067] Specifically, the gated cross-attention module includes a preprocessing gated unit, an enhancement gated unit, a cross-attention unit, a filtering gated unit, and an adapter connected in sequence.

[0068] The preprocessing gating unit is used to perform preliminary screening and filtering on the input information to obtain preliminary filtered features;

[0069] The enhancement gating unit is used to generate the enhancement key matrix and enhancement value matrix based on the initial filtering features;

[0070] The cross-attention unit is used to calculate the context vector based on the current query, the boosting key matrix, and the boosting value matrix.

[0071] The filtering gate unit is used to fuse and filter the context vector and the current query to obtain the final filtered features;

[0072] The adapter is used to transform the final filtered features into a representation space and then output them as the gated cross-attention module.

[0073] Specifically, the preprocessing gating uses the following method to initially filter the input information to obtain preliminary filtered features:

[0074] ;

[0075] ;

[0076] in, This represents the learnable weight matrix. Indicates bias. This represents the sigmoid function. A sequence of feature vectors representing medical text information. Indicates importance score, This indicates the initial filtering features.

[0077] During implementation, the gated cross-attention module first processes the semantic representation of the BERT output. A preliminary screening process is conducted to remove noise and irrelevant information. Among these steps, Indicates batch size, Indicates the length of the sequence (the length of the token). This represents the feature dimension of each token in the sequence.

[0078] In implementation, an importance score for each semantic unit in the semantic representation is calculated using a fully connected layer and an activation function. The semantic units are then evaluated based on their importance scores. Weighted semantic units are then filtered out by removing those with low scores.

[0079] During implementation, initial filtering features are performed. As the source of keys K and values ​​V for cross-attention.

[0080] In order to correct the attention weights during the cross-attention calculation process and enhance the retrieval of key information, the gated cross-attention module adjusts the attention distribution by enhancing the gating unit.

[0081] Specifically, the enhancement gating unit generates the enhancement key matrix and enhancement value matrix based on the initial filtering features in the following manner:

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] in, Indicates the current query. Indicates average pooling. The dimension parameter represents the tensor operation. Indicates preliminary filtering features. This indicates average pooling query. Indicates the average pooling key value. Indicates the enhancement coefficient. Represents the augmented gating vector. Represents the tanh function. Indicates splicing, This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the enhanced bond projection matrix. Represents the augmentation value projection matrix. Represents the key projection matrix. Represents the value projection matrix, Indicates dimensional expansion, The feature dimension of each token in the feature vector sequence representing medical text information. This indicates the dimension of the enhancement bond projection matrix. This indicates the dimension of the augmentation projection matrix, and the superscript T indicates transpose. Represents the enhancement bond matrix, This represents the augmentation value matrix.

[0091] Wherein, the value projection matrix Key projection matrix These are learnable parameters.

[0092] in, Indicates will according to Expand the dimensions to match the weight matrix. ; Indicates will according to Expand the dimensions to match the weight matrix. .

[0093] During implementation, the enhanced gating unit first determines the current query... (The enhancement coefficients are obtained by linearly transforming the features output by the attention mechanism module of the current Transformer layer through the query projection matrix, which is a learnable parameter.) and calculating the context. Then, the enhancement gating vector is calculated based on the enhancement coefficient, and the enhancement projection matrix is ​​generated based on the enhancement gating vector. and the augmented value projection matrix This results in an enhanced bond matrix. and enhanced value matrix .

[0094] After obtaining the enhancement key matrix and enhancement value matrix, cross-attention calculation is performed.

[0095] Specifically, the cross-attention unit uses a multi-head attention mechanism to calculate the context vector based on the current query, the boosted key matrix, and the boosted value matrix. .

[0096] The context vector is obtained in the following way. :

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] Where h represents the number of attention heads. This represents the calculation result of the i-th attention head. Indicates splicing, This represents the multi-head attention output projection matrix. This represents the query projection matrix of the i-th attention head. The projection matrix represents the value of the i-th attention head. This represents the key projection matrix of the i-th attention head, where the superscript T indicates transpose. The dimension representing a single attention head. This represents the query for the i-th attention head. The key representing the i-th attention head. This represents the value of the i-th attention head, and softmax() represents the softmax function.

[0103] After obtaining the cross-attention output, the fused semantic representation is further filtered and adjusted by a filtering gating unit to ensure that only the most relevant information is retained.

[0104] Specifically, the filtering gating unit fuses and filters the context vector and the current query to obtain the final filtering features in the following way:

[0105] ;

[0106] ;

[0107] ;

[0108] in, Indicates fusion characteristics, Represents the tanh function. Indicates splicing, Represents the context vector. Indicates the current query. The dimension parameter represents the tensor operation. This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the filtering gate vector. Indicates the final filtering features. This represents element-wise product.

[0109] The filtering gating unit outputs the context vector from the cross-attention function. Compared with the current query The information is then merged and controlled via a gating mechanism (fully connected layer + sigmoid). The filtering gating unit determines which information should be retained and passed to the next layer.

[0110] In practice, to better adapt to the representation space of large models, the gated cross-attention module performs a nonlinear transformation on the final filtered features.

[0111] Specifically, the adapter includes:

[0112] Dimensionality reduction layer, used to reduce and compress input data;

[0113] The recovery layer is used to restore the dimensions of the compressed data.

[0114] The residual layer is used to perform residual connections between the features output by the recovery layer and the final filtered features.

[0115] Layer normalization is used to normalize the features after residual connection as the output of the gated cross-attention module.

[0116] The dimensionality reduction layer uses the following formula to compress and reduce the dimensionality of the input data:

[0117] ;

[0118] ;

[0119] in, This represents the learnable weight coefficients. Indicates bias. This refers to the gelu function. This represents the output of the dimensionality reduction layer. express Dimensions It is Compressed to 1 / 4 dimension.

[0120] The recovery layer uses the following formula for dimensional recovery:

[0121] ;

[0122] in, This represents the learnable weight coefficients. Indicates bias.

[0123] The residual layers are connected using the following formula:

[0124] .

[0125] The layer normalization layer obtains its output in the following way:

[0126] ;

[0127] in, Representation layer normalization.

[0128] The adapter first compresses the input to a smaller dimension through a dimensionality reduction layer and then transforms it to obtain the hidden representation. Then restore to the original dimension to reduce intermediate computation.

[0129] During implementation, the trained large model is obtained using the following method.

[0130] Data preparation phase: Training is performed using a labeled medical text relation extraction dataset. Each training sample contains:

[0131] Original medical text: It has been de-identified, which means that the patient information in the medical text has been de-identified; the entity relationship triplet (head entity, relationship type, tail entity) is labeled.

[0132] Input construction phase: Perform the following operations for each training sample:

[0133] a. Input the original medical text into the pre-trained BERT model, extract the last hidden state as a feature vector sequence, denoted as... .

[0134] b. Construct task prompt text, formatted as: "Based on the following medical text semantic information, extract all entity relations:"

[0135] c. Convert the prompt text into an embedding vector sequence through the word embedding layer of the large model, and use it as input to the large model.

[0136] d. The BERT feature vector sequence As an external context, it is input into the gated cross-attention module of the large model, and cross-attention calculation is performed in each Transformer layer.

[0137] Output construction phase: The labeled entity relation triples are converted into normalized text sequences as training targets. The conversion rules are as follows:

[0138] Each triple is converted to: "Head entity: [Entity 1], Relation: [Relation type], Tail entity: [Entity 2]";

[0139] Multiple triples are separated by semicolons.

[0140] Model forward propagation phase:

[0141] a. The large model receives a sequence of cue word embedding vectors as input.

[0142] b. In each Transformer layer of the model, the BERT feature vector sequence It interacts with the self-attention output of the large model through a gated cross-attention module.

[0143] c. The model generates the output sequence in an autoregressive manner.

[0144] Loss calculation stage: The standard autoregressive cross-entropy loss function is used, which calculates the cross-entropy loss of the entity relationship triplet extracted from the large model and the labeled entity relationship triplet component.

[0145] It should be noted that the BERT feature vector sequence is used internally through a gated cross-attention mechanism, so it will not appear directly in the loss function formula.

[0146] Parameter update phase:

[0147] a. Keep the BERT model parameters completely frozen, without performing gradient calculations or updates.

[0148] b. Optimize only all parameters of the large model, including:

[0149] The original Transformer layer parameters of the large model;

[0150] New parameters for the gated cross-attention module.

[0151] On the other hand, embodiments of the present invention provide a medical text relation joint extraction system that combines a BERT model with a large model, such as... Figure 3 As shown, it includes the following steps:

[0152] The medical information acquisition module is used to acquire the medical text information to be extracted.

[0153] The feature vector sequence extraction module is used to input the medical text information into the trained BERT model to obtain the feature vector sequence of the medical text information.

[0154] The entity relation extraction module is used to input the feature vector sequence of the medical text information into a trained large model to extract the entity relations of the medical text information; the large model is a large model with an inserted gated cross-attention module; the gated cross-attention module includes a preprocessing gate unit, an enhancement gate unit, a cross-attention unit, a filtering gate unit, and an adapter connected in sequence.

[0155] Based on the further improvement of the above system, a gated cross-attention module is added after the self-attention module of each Transformer structure in the large model and before the feedforward network to construct a large model based on the gated cross-attention mechanism.

[0156] The above-described method and system embodiments are based on the same principles, and their related aspects can be referenced from each other to achieve the same technical effects. For specific implementation processes, please refer to the foregoing embodiments, which will not be repeated here.

[0157] Electronic device example:

[0158] One specific implementation of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method embodiment that combines the BERT model with a large model for joint extraction of medical text relationships.

[0159] Examples of readable storage media:

[0160] One specific implementation of this application discloses a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method embodiment for jointly extracting medical text relationships by combining a BERT model and a large model.

[0161] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0162] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for jointly extracting medical text relationships by combining BERT model and large-scale model, characterized in that, Includes the following steps: Obtain the medical text information to be extracted; The medical text information is input into the trained BERT model to obtain the feature vector sequence of the medical text information; The feature vector sequence of the medical text information is input into a trained large model to extract entity relationships from the medical text information; the large model is a large model with a gated cross-attention module inserted; the gated cross-attention module includes a preprocessing gated unit, an enhancement gated unit, a cross-attention unit, a filtering gated unit, and an adapter connected in sequence; A gated cross-attention module is added after the self-attention module of each Transformer layer in the large model and before the feedforward network to build a large model based on the gated cross-attention mechanism; The preprocessing gating unit is used to perform preliminary screening and filtering on the input information to obtain preliminary filtering features; The enhanced gating unit is used to generate an enhanced key matrix and an enhanced value matrix based on the initial filtering features; The cross-attention unit is used to calculate the context vector based on the current query, the boosting key matrix, and the boosting value matrix. The filtering gating unit is used to fuse the context vector and the current query and filter them to obtain the final filtering features; The adapter is used to transform the final filtered features in the representation space and use them as the output of the gated cross-attention module. The enhanced gating unit generates the enhanced key matrix and enhanced value matrix based on the initial filtering features in the following manner: ； ； ； ； ； ； ； ； in, Indicates the current query. Indicates average pooling. The dimension parameter represents the tensor operation. Indicates preliminary filtering features. This indicates average pooling query. Indicates the average pooling key value. Indicates the enhancement coefficient. Represents the augmented gating vector. Represents the tanh function. Indicates splicing, This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the enhanced bond projection matrix. Represents the augmentation value projection matrix. Represents the key projection matrix. Represents the value projection matrix, Indicates dimensional expansion, The feature dimension of each token in the feature vector sequence representing medical text information. This indicates the dimension of the enhancement bond projection matrix. This indicates the dimension of the augmentation projection matrix, and the superscript T indicates transpose. Represents the enhancement bond matrix, Represents the augmentation value matrix. This represents element-wise product.

2. The method for joint extraction of medical text relationships combining BERT model and large model as described in claim 1, characterized in that, The preprocessing gating unit performs preliminary filtering on the input information to obtain preliminary filtering features in the following manner: ； ； in, This represents the learnable weight matrix. Indicates bias. This represents the sigmoid function. The feature vector sequence representing medical text information. Indicates importance score, This indicates the initial filtering features.

3. The method for joint extraction of medical text relationships combining BERT model and large model as described in claim 1, characterized in that, The filtering gating unit fuses and filters the context vector and the current query to obtain the final filtering features in the following way: ； ； ； in, Indicates fusion features, Represents the tanh function. Indicates splicing, Represents the context vector. Indicates the current query. The dimension parameter represents the tensor operation. This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the filtering gate vector. Indicates the final filtering features. This represents element-wise product.

4. The method for joint extraction of medical text relationships combining BERT model and large model as described in claim 1, characterized in that, The cross-attention unit calculates the context vector by using a multi-head attention mechanism based on the current query, the augmented key matrix, and the augmented value matrix.

5. The method for joint extraction of medical text relationships combining BERT model and large model according to claim 1, characterized in that, The adapter includes: Dimensionality reduction layer, used to reduce and compress input data; The recovery layer is used to restore the dimensions of the compressed data. The residual layer is used to perform residual connections between the features output by the recovery layer and the final filtered features. Layer normalization is used to normalize the features after residual connection as the output of the gated cross-attention module.

6. A medical text relation joint extraction system combining BERT model and large model, characterized in that, Includes the following steps: The medical information acquisition module is used to acquire the medical text information to be extracted. The feature vector sequence extraction module is used to input the medical text information into the trained BERT model to obtain the feature vector sequence of the medical text information. An entity relation extraction module is used to input the feature vector sequence of the medical text information into a trained large model to extract the entity relations of the medical text information; the large model is a large model with a gated cross-attention module inserted; the gated cross-attention module includes a preprocessing gate unit, an enhancement gate unit, a cross-attention unit, a filtering gate unit, and an adapter connected in sequence; a gated cross-attention module is added after the self-attention module of each Transformer structure layer of the large model and before the feedforward network to construct a large model based on the gated cross-attention mechanism; The preprocessing gating unit is used to perform preliminary screening and filtering on the input information to obtain preliminary filtering features; The enhanced gating unit is used to generate an enhanced key matrix and an enhanced value matrix based on the initial filtering features; The cross-attention unit is used to calculate the context vector based on the current query, the boosting key matrix, and the boosting value matrix. The filtering gating unit is used to fuse the context vector and the current query and filter them to obtain the final filtering features; The adapter is used to transform the final filtered features in the representation space and use them as the output of the gated cross-attention module. The enhanced gating unit generates the enhanced key matrix and enhanced value matrix based on the initial filtering features in the following manner: ； ； ； ； ； ； ； ； in, Indicates the current query. Indicates average pooling. The dimension parameter represents the tensor operation. Indicates preliminary filtering features. This indicates average pooling query. Indicates the average pooling key value. Indicates the enhancement coefficient. Represents the augmented gating vector. Represents the tanh function. Indicates splicing, This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the enhanced bond projection matrix. Represents the augmentation value projection matrix. Represents the key projection matrix. Represents the value projection matrix, Indicates dimensional expansion, The feature dimension of each token in the feature vector sequence representing medical text information. This indicates the dimension of the enhancement bond projection matrix. This indicates the dimension of the augmentation projection matrix, and the superscript T indicates transpose. Represents the enhancement bond matrix, Represents the augmentation value matrix. This represents element-wise product.

7. The medical text relationship joint extraction system combining BERT model and large model according to claim 6, characterized in that, The preprocessing gating unit performs preliminary filtering on the input information to obtain preliminary filtering features in the following manner: ； ； in, This represents the learnable weight matrix. Indicates bias. This represents the sigmoid function. The feature vector sequence representing medical text information. Indicates importance score, This indicates the initial filtering features.

8. The medical text relationship joint extraction system combining BERT model and large model according to claim 6, characterized in that, The filtering gating unit fuses and filters the context vector and the current query to obtain the final filtering features in the following way: ； ； ； in, Indicates fusion features, Represents the tanh function. Indicates splicing, Represents the context vector. Indicates the current query. The dimension parameter represents the tensor operation. This represents the sigmoid function. and Each represents a learnable weight matrix. and Both represent bias. Represents the filtering gate vector. Indicates the final filtering features. This represents element-wise product.

Images