Text matching method and system based on dynamic hierarchical attention mechanism and storage medium
The text matching method using a dynamic hierarchical attention mechanism solves the problem of low semantic matching accuracy in complex texts by traditional methods, and achieves efficient and accurate semantic understanding and matching of long texts.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUNNAN DAILY NEWSPAPER GRP
- Filing Date
- 2026-01-27
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional text matching methods suffer from low semantic matching accuracy when dealing with complex texts. Their attention mechanisms are rigid and their matching strategies are limited, making it difficult to adapt to the dynamically changing semantic focus in long texts.
A text matching method based on dynamic hierarchical attention mechanism is adopted. Semantic features are obtained through a domain-fine-tuned BERT model, word-level and sentence-level vectors are obtained by hierarchical acquisition using an improved bidirectional GRU network, and weights are assigned through dynamic hierarchical attention mechanism. The results are output by combining a hybrid matching algorithm.
The model's sensitivity to vertical domain terms has been enhanced, capturing word-level local features and sentence-level global features, prioritizing entity words and core sentences, and improving the accuracy and robustness of semantic matching.
Smart Images
Figure CN121598110B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to a text matching method, system and storage medium based on a dynamic hierarchical attention mechanism. Background Technology
[0002] With the explosive growth of internet information, efficiently and accurately understanding and associating massive amounts of text has become one of the core challenges in the field of natural language processing. Text matching, as a key technology in this field, is widely used in many important areas such as question-answering systems, search engines, intelligent customer service, legal document comparison, and medical diagnostic assistance. Therefore, developing more efficient and adaptive text matching methods is of great significance for improving information retrieval quality and promoting the application of artificial intelligence.
[0003] Traditional text matching methods (such as TF-IDF and BM25) mainly rely on word frequency statistics for matching. This method is effective when dealing with simple text, but its semantic matching accuracy is low when dealing with complex text. Neural network-based models (such as Siamese LSTM and BERT) have improved semantic understanding capabilities, but they still have the following problems: First, the attention mechanism is rigid. Traditional self-attention mechanisms use a fixed weight allocation method, which is difficult to adapt to the dynamically changing semantic focus in long texts. Second, the matching strategy is simplistic. Most methods rely only on simple similarity calculations (such as cosine similarity), which is difficult to accurately characterize complex semantic relationships.
[0004] These issues limit the effectiveness of existing methods in practical applications, especially when dealing with long texts that are semantically complex and structurally varied. Summary of the Invention
[0005] The main purpose of this application is to provide a text matching method based on a dynamic hierarchical attention mechanism, which aims to solve the problems of rigid attention mechanism and single matching strategy in text matching.
[0006] To achieve the above objectives, this application provides a text matching method based on a dynamic hierarchical attention mechanism, the method comprising:
[0007] S10: Obtaining semantic features of the text to be matched using a domain-based fine-tuned BERT model;
[0008] S20: Based on the improved bidirectional GRU network, word-level vectors and sentence-level vectors of the semantic features are obtained hierarchically, wherein the first layer obtains the word-level vectors and the second layer obtains the sentence-level vectors;
[0009] S30: Assign attention weights to the word-level vectors and sentence-level vectors through a dynamic hierarchical attention mechanism;
[0010] S40: The attention weights of the word-level vectors and the sentence-level vectors are fused through a gating network to obtain the final attention weights;
[0011] S50: Input semantic features based on final attention weights into the pre-trained text matching model, and output the text matching result through a hybrid matching algorithm.
[0012] Optionally, the field fine-tuning includes:
[0013] Acquire the corpus of the matching task domain, including source domain and target domain data;
[0014] An adversarial training objective function is constructed based on a domain classifier and a matching task domain corpus. Adversarial training is achieved by minimizing the objective function, thereby completing fine-tuning.
[0015] The objective function expression for adversarial training is:
[0016]
[0017] In the formula, It is the loss function for adversarial training. This indicates that the input sample x comes from the source domain dataset. This indicates that the input sample x comes from the target domain dataset. For the domain classifier, and These are source domain and target domain data, respectively.
[0018] Optionally, before obtaining the semantic features of the text to be matched, the text to be matched needs to be preprocessed. The preprocessing involves standardizing the text to be matched, including domain-dictionary-enhanced word segmentation, stop word filtering, stemming, and named entity recognition. After preprocessing, the semantic features of the text to be matched are obtained as follows:
[0019] The domain-fine-tuned BERT model is retrained using the matching task domain corpus;
[0020] Extract the weighted sum of the hidden states from the {LN}th to the Lth layer of the retrained, domain-fine-tuned BERT model as the semantic features of the text to be matched.
[0021] Optionally, the improved bidirectional GRU network includes improved word-level encoding layers and sentence-level encoding layers, as well as the introduction of a cross-block attention mechanism, including:
[0022] The first word-level coding layer uses a bidirectional GRU with residual connections;
[0023] The second sentence-level coding layer divides semantic blocks into semantic blocks using trainable segmented gating units, and each semantic block is independently encoded using bidirectional GRU.
[0024] The expression for introducing the cross-block attention mechanism is as follows:
[0025]
[0026] In the formula, Let be the final enhanced representation vector of the j-th semantic block. The degree of attention paid by the j-th semantic block to the k-th semantic block. Let M be the original vector representation of the k-th semantic block, and M be the number of semantic blocks. These are trainable parameters.
[0027] Optionally, the dynamic hierarchical attention mechanism includes dynamically obtaining word-level attention weights and sentence-level attention weights, wherein the dynamic nature is achieved through a dynamic weight adjustment factor, and the word-level attention weights are attention weights assigned to the word-level vectors, expressed as:
[0028]
[0029] In the formula, For word-level attention weights, and The GRU hidden state of a word is represented by the output of the first layer of a bidirectional GRU network. , , Here are the trainable parameters, and d is the dimension. This is a matrix transpose operation. This is a dynamic weight adjustment factor;
[0030] The sentence-level attention weights are the attention weights assigned to the sentence-level vectors, and are expressed as follows:
[0031]
[0032] In the formula, Sentence-level attention weights and The aggregated representation of the sentence is output by the second layer of the bidirectional GRU network. , , For trainable parameters, For dimensions;
[0033] Wherein, the dynamic weight adjustment factor The expression is:
[0034]
[0035] In the formula, For the Sigmoid function, The number of terms matched by the domain dictionary in the input sample x. To input the total number of words in the sample, The maximum depth of the dependency syntax tree, These are trainable parameters.
[0036] Optionally, the final attention weight expression is:
[0037]
[0038]
[0039]
[0040] In the formula, For the final attention weight, and are the average weights of the attention to the word-level vector and the sentence-level vector, respectively, and u and c are trainable parameters.
[0041] Optionally, the hybrid matching algorithm combines cosine similarity with bilinear tensor calculation, and its expression is:
[0042]
[0043] In the formula, S represents the overall similarity score; a higher score indicates a higher degree of matching. For a trainable parameter matrix, and These are the feature vectors of the two texts to be matched. This is a dynamic adjustment coefficient;
[0044] The expression for the dynamic adjustment coefficient is as follows:
[0045]
[0046] In the formula, and For trainable parameters, This indicates a vector concatenation operation.
[0047] Optionally, the text matching model is pre-trained using a hybrid loss function, the expression of which is:
[0048]
[0049] In the formula, For the total loss, To compare the losses, For cross-entropy loss, and The balance coefficient between the comparison loss and the cross-entropy loss.
[0050] Furthermore, to achieve the above objectives, this application also provides a text matching system based on a dynamic hierarchical attention mechanism. The text matching system based on the dynamic hierarchical attention mechanism includes: a memory, a processor, and a text matching program based on the dynamic hierarchical attention mechanism stored in the memory and executable on the processor. When the text matching program based on the dynamic hierarchical attention mechanism is executed by the processor, it implements the steps of the text matching method based on the dynamic hierarchical attention mechanism as described in any of the preceding claims.
[0051] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a text matching program based on a dynamic hierarchical attention mechanism, wherein when the text matching program based on the dynamic hierarchical attention mechanism is executed by a processor, it implements the steps of the text matching method based on the dynamic hierarchical attention mechanism as described in any of the preceding claims.
[0052] This application has at least the following beneficial effects:
[0053] 1. By introducing domain-adaptive fine-tuning during the BERT fine-tuning stage, domain classification adversarial training is introduced to enhance the model's sensitivity to vertical domain terms;
[0054] 2. Using bidirectional GRU hierarchical encoding, word-level local features and sentence-level global features are captured by two layers of GRU respectively, and residual connections alleviate gradient vanishing;
[0055] 3. Introduce dynamic hierarchical attention, design word-sentence two-level attention, and dynamically adjust weight allocation through gating network to prioritize entity words and core sentences;
[0056] 4. A hybrid matching algorithm is proposed, which integrates bilinear tensor calculation and cosine similarity. It uses a learnable parameter matrix to capture nonlinear semantic interactions and optimizes the model by introducing a hybrid loss function. Contrastive loss is used to enhance the distinction between positive and negative samples, and cross-entropy loss is used to optimize the classification boundary, thereby improving the robustness of the model. Attached Figure Description
[0057] Figure 1 This is a flowchart illustrating the steps of the text matching method based on the dynamic hierarchical attention mechanism in this application.
[0058] Figure 2 This is a schematic diagram of the dynamic hierarchical attention mechanism of this application;
[0059] Figure 3This is a schematic diagram of the hardware operating environment of the text matching system based on a dynamic hierarchical attention mechanism involved in the embodiments of this application.
[0060] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0061] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0062] Reference Figure 1 In this embodiment, the text matching method based on the dynamic hierarchical attention mechanism includes the following steps:
[0063] S10: Obtaining semantic features of the text to be matched using a domain-based fine-tuned BERT model;
[0064] Optionally, the field fine-tuning includes:
[0065] Acquire the corpus of the matching task domain, including source domain and target domain data;
[0066] An adversarial training objective function is constructed based on a domain classifier and a matching task domain corpus. Adversarial training is achieved by minimizing the objective function, thereby completing fine-tuning.
[0067] The objective function expression for adversarial training is:
[0068]
[0069] In the formula, It is the loss function for adversarial training. This indicates that the input sample x comes from the source domain dataset. This indicates that the input sample x comes from the target domain dataset. For the domain classifier, and The data represent the source and target domains, respectively. It's understandable that the goal of adversarial training is to eliminate domain dependencies in the features, thereby optimizing the model's performance in the target domain.
[0070] The source domain data is a general text dataset (such as Wikipedia) used to provide basic language understanding capabilities; the target domain data is a professional dataset related to the matching task (such as medical records and legal provisions), containing domain-specific terms and semantic relationships.
[0071] Optionally, before obtaining the semantic features of the text to be matched, the text to be matched needs to be preprocessed. The preprocessing involves standardizing the text to be matched, including domain-dictionary-enhanced word segmentation, stop word filtering, stemming, and named entity recognition. After preprocessing, the semantic features of the text to be matched are obtained as follows:
[0072] The domain-fine-tuned BERT model is retrained using matching task domain corpus, wherein the retraining includes:
[0073] Unsupervised pre-training can be performed using unlabeled data in the domain (e.g., masked language modeling tasks).
[0074] Expand the original vocabulary using a domain-specific dictionary;
[0075] Supervised fine-tuning is performed using domain-labeled data (e.g., for classification / similarity tasks).
[0076] Extract the weighted sum of the hidden states from the {LN}th to the Lth layer of the retrained, domain-fine-tuned BERT model as the semantic features of the text to be matched;
[0077] Optionally, the semantic features can also be processed, including:
[0078] Dynamically select feature combinations of different Transformer layers (such as weighted average of the last 4 layers).
[0079] Introduce domain-related feature enhancement modules (such as domain entity tag embedding).
[0080] Special optimization processing is applied to the [CLS] tag representation, where [CLS] is the semantic summary vector of BERT.
[0081] Optionally, in some specific implementations, the word segmentation based on the domain dictionary enhancement can be: constructing a target domain terminology list (such as drug names and diagnostic terms in the medical field); loading the target domain terminology list into an open-source word segmentation tool, and outputting the segmentation results for subsequent processing.
[0082] S20: Based on the improved bidirectional GRU network, word-level vectors and sentence-level vectors of the semantic features are obtained hierarchically, wherein the first layer obtains the word-level vectors and the second layer obtains the sentence-level vectors;
[0083] Optionally, the improved bidirectional GRU network includes improved word-level encoding layers and sentence-level encoding layers, as well as the introduction of a cross-block attention mechanism, including:
[0084] The first word-level coding layer uses a bidirectional GRU with residual connections;
[0085] The second sentence-level coding layer divides semantic blocks into semantic blocks using trainable segmented gating units, and each semantic block is independently encoded using bidirectional GRU.
[0086] The expression for introducing the cross-block attention mechanism is as follows:
[0087]
[0088] In the formula, Let be the final enhanced representation vector of the j-th semantic block. The degree of attention paid by the j-th semantic block to the k-th semantic block. Let M be the original vector representation of the k-th semantic block, and M be the number of semantic blocks. These are trainable parameters.
[0089] The specific implementation of the residual connection is as follows:
[0090] First, let the input word vector be... The hidden state of the bidirectional GRU output is ;
[0091] Then, the residual connection preserves the original input information through skip connections:
[0092]
[0093] in, Let be the final output feature vector of the i-th word. These are the parameters for linear transformation; LayerNorm is the layer normalization operation to prevent numerical instability.
[0094] Finally, during backpropagation, the gradient can be directly passed to the lower layer through the residual path, avoiding gradient decay caused by multiple layers of GRU.
[0095] S30: Assign attention weights to the word-level vectors and sentence-level vectors through a dynamic hierarchical attention mechanism;
[0096] Optionally, refer to Figure 2 The dynamic hierarchical attention mechanism includes dynamically obtaining word-level attention weights and sentence-level attention weights. The dynamic nature is achieved through a dynamic weight adjustment factor. The word-level attention weights are assigned to the word-level vectors, and the expression is:
[0097]
[0098] In the formula, For word-level attention weights, and The GRU hidden state of a word is represented by the output of the first layer of a bidirectional GRU network. , , Here are the trainable parameters, and d is the dimension. This is a matrix transpose operation. This is a dynamic weight adjustment factor;
[0099] The sentence-level attention weights are the attention weights assigned to the sentence-level vectors, and are expressed as follows:
[0100]
[0101] In the formula, Sentence-level attention weights and The aggregated representation of the sentence is output by the second layer of the bidirectional GRU network. , , For trainable parameters, For dimensions;
[0102] Wherein, the dynamic weight adjustment factor The expression is:
[0103]
[0104] In the formula, For the Sigmoid function, The number of terms matched by the domain dictionary in the input sample x. To input the total number of words in the sample, The maximum depth of the dependency syntax tree, These are trainable parameters.
[0105] The GRU hidden state is generated through the following steps:
[0106] The input text is encoded using BERT to obtain the token embedding sequence {e1,…,e n};
[0107] The embedded sequence is input into the first layer of the bidirectional GRU network, and the forward and backward hidden states are concatenated:
[0108]
[0109] in, Let be the hidden state at the i-th position of the forward GRU. Let i be the hidden state at the i-th position of the backward GRU. The hidden layer dimension of the unidirectional GRU is 256 in this embodiment.
[0110] S40: The attention weights of the word-level vectors and the sentence-level vectors are fused through a gating network to obtain the final attention weights;
[0111] Optionally, the gated network consists of a fully connected layer and a sigmoid activation function, used to balance the attention contributions of the word-level vectors and the sentence-level vectors.
[0112] Optionally, the final attention weight expression is:
[0113]
[0114]
[0115]
[0116] In the formula, For the final attention weight, and are the average weights of the attention to the word-level vector and the sentence-level vector, respectively, and u and c are trainable parameters.
[0117] S50: Input semantic features based on final attention weights into the pre-trained text matching model, and output the text matching result through a hybrid matching algorithm.
[0118] Optionally, the hybrid matching algorithm combines cosine similarity with bilinear tensor calculation, and its expression is:
[0119]
[0120] In the formula, S is the overall similarity score, with a higher score indicating a higher degree of matching, and cos is the cosine similarity calculation. For a trainable parameter matrix, and These are the feature vectors of the two texts to be matched. This is a dynamic adjustment coefficient;
[0121] The expression for the dynamic adjustment coefficient is as follows:
[0122]
[0123] In the formula, and For trainable parameters, This indicates a vector concatenation operation.
[0124] Optionally, the text matching model is pre-trained using a hybrid loss function, the expression of which is:
[0125]
[0126] In the formula, For the total loss, To compare the losses, For cross-entropy loss, and The balance coefficient between the comparison loss and the cross-entropy loss.
[0127] Optionally, as an example, the text matching method based on the dynamic hierarchical attention mechanism described in this embodiment can be used for legal text matching, specifically:
[0128] Input text:
[0129] Text 1: "If the patentee fails to pay the annual fee on time, the patent right shall terminate on the date on which the annual fee should have been paid."
[0130] Text 2: "If the patentee fails to pay the annual fee by the due date, the patent right shall expire after the payment deadline."
[0131] Implementation steps:
[0132] Step 1: Enhance word segmentation using a legal dictionary to identify entities such as "patentee" and "annual fee";
[0133] Step 2: Use the BERT model fine-tuned by legal documents to generate vectors. During fine-tuning, adversarial training is added to prevent overfitting.
[0134] Step 3: The first layer of the bidirectional GRU network outputs word-level features, and the second layer aggregates sentence meanings;
[0135] Step 4: Word-level attention focuses on "termination" and "invalidity," while sentence-level attention weights the legal logic of the entire sentence;
[0136] Step 5: The hybrid algorithm outputs a similarity of 0.92, which is considered a high match.
[0137] As one implementation scheme, Figure 3 This is a schematic diagram of the hardware operating environment of the text matching system based on a dynamic hierarchical attention mechanism involved in the embodiments of this application.
[0138] like Figure 3As shown, the text matching system based on a dynamic hierarchical attention mechanism may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0139] Those skilled in the art will understand that Figure 3 The text matching system architecture based on the dynamic hierarchical attention mechanism shown in the figure does not constitute a limitation on the text matching system based on the dynamic hierarchical attention mechanism. It may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.
[0140] like Figure 3 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the text matching system based on a dynamic hierarchical attention mechanism, as well as the computer programs and the execution of other software or programs.
[0141] exist Figure 3 In the text matching system based on the dynamic hierarchical attention mechanism shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used to communicate data with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.
[0142] In this embodiment, the text matching system based on a dynamic hierarchical attention mechanism includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:
[0143] When processor 1001 calls the text matching program based on the dynamic hierarchical attention mechanism stored in memory 1005, it performs the following operations:
[0144] S10: Obtaining semantic features of the text to be matched using a domain-based fine-tuned BERT model;
[0145] S20: Based on the improved bidirectional GRU network, word-level vectors and sentence-level vectors of the semantic features are obtained hierarchically, wherein the first layer obtains the word-level vectors and the second layer obtains the sentence-level vectors;
[0146] S30: Assign attention weights to the word-level vectors and sentence-level vectors through a dynamic hierarchical attention mechanism;
[0147] S40: The attention weights of the word-level vectors and the sentence-level vectors are fused through a gating network to obtain the final attention weights;
[0148] S50: Input semantic features based on final attention weights into the pre-trained text matching model, and output the text matching result through a hybrid matching algorithm.
[0149] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a text matching system based on a dynamic hierarchical attention mechanism to implement the process steps of the embodiments of the above methods.
[0150] Therefore, this application also provides a computer-readable storage medium storing a text matching program based on a dynamic hierarchical attention mechanism. When the text matching system program based on the dynamic hierarchical attention mechanism is executed by a processor, it implements the various steps of the text matching method based on the dynamic hierarchical attention mechanism as described in the above embodiments.
[0151] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0152] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.
[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0154] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0157] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0158] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0159] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A text matching method based on a dynamic hierarchical attention mechanism, characterized in that, The method includes the following steps: S10: Obtaining semantic features of the text to be matched using a domain-based fine-tuned BERT model; S20: Based on the improved bidirectional GRU network, word-level vectors and sentence-level vectors of the semantic features are obtained hierarchically, wherein the first layer obtains the word-level vectors and the second layer obtains the sentence-level vectors; S30: Assign attention weights to the word-level vectors and sentence-level vectors through a dynamic hierarchical attention mechanism; S40: The attention weights of the word-level vectors and the sentence-level vectors are fused through a gating network to obtain the final attention weights; S50: Input the semantic features based on the final attention weight into the pre-trained text matching model, and output the text matching result through the hybrid matching algorithm; The improved bidirectional GRU network includes improved word-level and sentence-level encoding layers, as well as the introduction of a cross-block attention mechanism, including: The first word-level coding layer uses a bidirectional GRU with residual connections; The second sentence-level coding layer divides semantic blocks into semantic blocks using trainable segmented gating units, and each semantic block is independently encoded using bidirectional GRU. The expression for introducing the cross-block attention mechanism is as follows: ； In the formula, Let be the final enhanced representation vector of the j-th semantic block. The degree of attention paid by the j-th semantic block to the k-th semantic block. Let M be the original vector representation of the k-th semantic block, and M be the number of semantic blocks. These are trainable parameters; The dynamic hierarchical attention mechanism includes dynamically obtaining word-level attention weights and sentence-level attention weights. The dynamic nature is achieved through a dynamic weight adjustment factor. The word-level attention weights are assigned to the word-level vectors, and the expression is: ； In the formula, For word-level attention weights, and The GRU hidden state of a word is represented by the output of the first layer of a bidirectional GRU network. , , Here are the trainable parameters, and d is the dimension. This is a matrix transpose operation. This is a dynamic weight adjustment factor; The sentence-level attention weights are the attention weights assigned to the sentence-level vectors, and are expressed as follows: ； In the formula, Sentence-level attention weights and The aggregated representation of the sentence is output by the second layer of the bidirectional GRU network. , , For trainable parameters, For dimensions; Wherein, the dynamic weight adjustment factor The expression is: ； In the formula, For the Sigmoid function, The number of terms matched by the domain dictionary in the input sample x. To input the total number of words in the sample, The maximum depth of the dependency syntax tree, These are trainable parameters.
2. The text matching method based on a dynamic hierarchical attention mechanism as described in claim 1, characterized in that, The aforementioned fine-tuning includes: Acquire the corpus of the matching task domain, including source domain and target domain data; An adversarial training objective function is constructed based on a domain classifier and a matching task domain corpus. Adversarial training is achieved by minimizing the objective function, thereby completing fine-tuning. The objective function expression for adversarial training is: ； In the formula, It is the loss function for adversarial training. This indicates that the input sample x comes from the source domain dataset. This indicates that the input sample x comes from the target domain dataset. For the domain classifier, and These are source domain and target domain data, respectively.
3. The text matching method based on a dynamic hierarchical attention mechanism as described in claim 2, characterized in that, Before obtaining the semantic features of the text to be matched, the text to be matched needs to be preprocessed. This preprocessing involves standardizing the text, including domain-dictionary-enhanced word segmentation, stop word filtering, stemming, and named entity recognition. After preprocessing, the semantic features of the text to be matched are obtained as follows: The domain-fine-tuned BERT model is retrained using the matching task domain corpus; Extract the weighted sum of the hidden states from the {LN}th to the Lth layer of the retrained, domain-fine-tuned BERT model as the semantic features of the text to be matched.
4. The text matching method based on a dynamic hierarchical attention mechanism as described in claim 1, characterized in that, The final attention weight expression is: ； ； ； In the formula, For the final attention weight, and are the average weights of the attention to the word-level vector and the sentence-level vector, respectively, and u and c are trainable parameters.
5. The text matching method based on a dynamic hierarchical attention mechanism as described in claim 1, characterized in that, The hybrid matching algorithm combines cosine similarity with bilinear tensor calculation, and its expression is: ； In the formula, S is the overall similarity score, with a higher score indicating a higher degree of matching, and cos is the cosine similarity calculation. For a trainable parameter matrix, and These are the feature vectors of the two texts to be matched. This is a dynamic adjustment coefficient; The expression for the dynamic adjustment coefficient is as follows: ； In the formula, and For trainable parameters, This indicates a vector concatenation operation.
6. The text matching method based on a dynamic hierarchical attention mechanism as described in claim 1, characterized in that, The text matching model is pre-trained using a hybrid loss function, the expression of which is: ； In the formula, For the total loss, To compare the losses, For cross-entropy loss, and The balance coefficient between the comparison loss and the cross-entropy loss.
7. A text matching system based on a dynamic hierarchical attention mechanism, characterized in that, The text matching system based on the dynamic hierarchical attention mechanism includes: a memory, a processor, and a text matching program based on the dynamic hierarchical attention mechanism stored in the memory and executable on the processor. When the text matching program based on the dynamic hierarchical attention mechanism is executed by the processor, it implements the steps of the text matching method based on the dynamic hierarchical attention mechanism as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a text matching program based on a dynamic hierarchical attention mechanism, which, when executed by a processor, implements the steps of the text matching method based on a dynamic hierarchical attention mechanism as described in any one of claims 1 to 6.