Semantic recognition system and method based on heterogeneous graph attention network and dynamic normalization

Abstract

The invention discloses a semantic recognition system and method based on a heterogeneous graph attention network and dynamic normalization, and belongs to the technical field of natural language processing and artificial intelligence. The system adopts a dual-channel architecture and comprises a general semantic channel and a domain semantic channel, semantic feature extraction is performed through a DIFF attention mechanism and a ToST statistical attention mechanism, and training stability is improved by adopting a DyT dynamic normalization module. Adaptive fusion of cross-channel semantic features is realized through a GeGLU gating mechanism, and high-precision semantic recognition is realized by combining an improved SimCSE + + comparison learning loss and a local minimization editing strategy of semantic perception. According to the method, the problems of inaccurate semantic expression, poor context adaptability and the like in the prior art are solved, the accuracy and applicability of cross-domain semantic recognition are remarkably improved, and the method can be widely applied to scenes of legal document processing, financial document analysis and the like.

Description

Technical Field

[0001] The present application relates to the field of natural language processing and artificial intelligence technology, and more specifically, to a semantic recognition system and method based on heterogeneous graph attention networks and dynamic normalization. Background Art

[0002] In today's era of rapid digital information development, there is a widespread and urgent demand for semantic recognition technology in many fields, including automated legal document processing, intelligent financial document analysis, and scientific literature processing. Semantic recognition technology aims to identify specific semantics in text as more appropriate expressions to meet different scenarios and needs. However, existing semantic recognition technology has exposed a series of serious problems in practical applications, making it difficult to meet the growing demand for cross-industry and multi-domain applications.

[0003] The main deficiencies of existing technologies are reflected in five aspects: First, the problem of inaccurate semantic expression is prominent. Traditional systems rely on a single semantic channel or a simple vector matching method, lacking an in-depth characterization of the semantic hierarchy, resulting in the inability to take into account the multi-level fusion of general semantics and domain semantics. Second, context adaptability is poor. Existing systems find it difficult to adapt to the semantic span and diverse contextual environments of long texts, and are prone to semantic deviations when processing complex long texts. Third, there is a lack of efficient dynamic fusion mechanisms. Multi-channel semantic information fusion generally adopts a fixed weight method, which cannot flexibly adjust channel contributions according to specific inputs and contexts. Fourth, the semantic space discrimination ability is insufficient. Traditional methods lack advanced training mechanisms such as contrastive learning, resulting in blurred semantic boundaries between different terms in the vector space. Finally, applicability is limited. Most systems perform unstable when faced with professional terms such as law and finance, and are unable to take into account semantic differences in multiple fields.

[0004] The root of these problems lies in the fact that existing technologies lack the ability to effectively model semantic ambiguity, making it difficult to correctly understand the true meaning of terms in specific contexts. Furthermore, existing technologies are inadequate for handling the replacement of polysemous or domain-specific terms. Their semantic representation methods are limited, typically employing single-channel static encoding, and are unable to effectively integrate multi-level, multi-source semantic features.

[0005] While global research in semantic recognition technology has achieved some success, rule-based approaches have limited processing capabilities, and machine learning-based approaches still suffer from numerous shortcomings. These include a single semantic processing approach, insufficient context processing capabilities, imperfect fusion mechanisms, weak semantic space discrimination capabilities, and limited cross-domain applicability. These differences in technical characteristics make existing systems unable to meet practical application needs.

[0006] In view of the above problems, the existing technology is in urgent need of improvement. Summary of the Invention

[0007] The purpose of this application is to provide a semantic recognition system based on heterogeneous graph attention network and dynamic normalization, which has the advantages of improving the accuracy of semantic expression, enhancing context adaptability, realizing dynamic semantic fusion, and improving cross-domain applicability.

[0008] The present application provides a semantic recognition system based on heterogeneous graph attention network and dynamic normalization. The technical solution is as follows: a semantic recognition system based on heterogeneous graph attention network and dynamic normalization, which is used to solve the problems of inaccurate semantic expression, poor context adaptability, lack of dynamic fusion mechanism, insufficient semantic space discrimination ability and limited applicability in existing semantic recognition technology. It adopts a parallel dual-path architecture, including a general semantic channel and a domain semantic channel, which respectively perform multi-level encoding on the general semantic expression and specific domain context features in the text. The system includes the following three core modules: a text semantic feature encoding network, which adopts a dual-channel architecture to capture general semantics and domain-specific semantics respectively; a general semantic channel, which takes SentenceBERT as the basic model, adopts an attention mechanism that can suppress redundant semantic information and enhance context discrimination ability, as well as a dynamic normalization module to improve the stability and generalization ability of model training, and adopts a method for optimizing the discriminability and representation of semantic vectors. The domain semantic channel is based on the ModernBERT architecture and integrates a statistical attention mechanism that can enhance the context modeling capability of long texts. It introduces an alternating window global attention strategy that supports ultra-long text processing and also adopts a dynamic normalization module. The GeGLU gating mechanism generates dynamic gating weights through the GeGLU activation function. Based on the dynamic feature control mechanism, the outputs of the two semantic channels are weightedly fused to achieve adaptive integration of cross-channel semantic features. The fusion ratio of general semantics and domain semantics is dynamically adjusted according to the input text and context, and it is ensured that the replacement decision has a reasonable weight distribution under different context conditions. The term matching module uses the fused semantic representation to calculate the similarity with the pre-built multi-domain term library, selects the best match to perform the replacement operation, supports the semantic-aware local minimization editing strategy, and performs minimal modification on the basis of retaining the original sentence structure to ensure contextual semantic consistency and language naturalness.

[0009] Furthermore, this application also proposes that the attention mechanism used in the general semantic channel is the DIFF attention mechanism, the dynamic normalization module is the DyT dynamic normalization, and the contrastive learning loss is the improved version of SimCSE++ contrastive learning loss.

[0010] Furthermore, this application also proposes that the statistical attention mechanism used in the domain semantic channel is the ToST statistical attention mechanism.

[0011] Furthermore, the present application also proposes that, in the term matching module, a multi-domain term base is pre-built, each term is semantically vectorized, and the semantic matching degree between the term candidate in the input text and the term base vector is calculated using cosine similarity.

[0012] Furthermore, this application also proposes an improved version of SimCSE++ contrastive learning loss used in the general semantic channel. By constructing diverse positive and negative sample pairs and using dynamic adjustment factors to adaptively adjust the training difficulty, it can enhance the semantic distinction between different terms, improve the quality of semantic expression and the generalization ability of the model.

[0013] Furthermore, this application also proposes the ToST statistical attention mechanism used in the domain semantic channel, which strengthens the semantic connection between domain keywords in long texts by counting the co-occurrence of keywords in large-scale corpus, so as to enhance long-distance dependency modeling and improve domain-specific recognition and context understanding capabilities.

[0014] Furthermore, this application also proposes that the dynamic normalization module is DyT dynamic normalization, which replaces the traditional LayerNorm and dynamically adjusts the adjustable coefficient according to the semantic features of different parts to improve the stability of model training, enhance generalization ability and improve feature scale adaptability.

[0015] Furthermore, this application also proposes a semantic recognition method based on heterogeneous graph attention network and dynamic normalization to solve the problems of inaccurate semantic expression, poor context adaptability, lack of dynamic fusion mechanism, insufficient semantic space discrimination ability and limited applicability in existing semantic recognition technology, including the following steps:

[0016] Through the general semantic channel and domain semantic channel in the parallel dual-path architecture, the general semantic expressions and specific domain context features in the text are encoded at multiple levels.

[0017] In the general semantic channel, the SentenceBERT model is used as the base model. It adopts an attention mechanism that can suppress redundant semantic information and enhance context discrimination. It also adopts a dynamic normalization module to improve model training stability and generalization ability. Furthermore, it uses a contrastive learning loss to optimize the discriminability and expression quality of semantic vectors.

[0018] In the domain semantic channel, based on the ModernBERT architecture, we integrate a statistical attention mechanism that can enhance the context modeling capabilities of long texts, introduce an alternating window global attention strategy that supports the processing of very long texts, and also use a dynamic normalization module.

[0019] Through the GeGLU gating mechanism, the GeGLU activation function is used to generate dynamic gating weights. Based on the dynamic feature control mechanism, the outputs of the two semantic channels are weighted and fused to achieve adaptive integration of cross-channel semantic features. The fusion ratio of general semantics and domain semantics is dynamically adjusted according to the input text and context, and the replacement decision is ensured to have a reasonable weight distribution under different context conditions.

[0020] The fused semantic representation is used to calculate similarity with the pre-built multi-domain terminology library, and the best matching item is selected for replacement. A semantic-aware local minimization editing strategy is adopted to minimize modifications while retaining the original sentence structure, ensuring contextual semantic consistency and linguistic naturalness.

[0021] Furthermore, this application also proposes that the attention mechanism used in the general semantic channel is the DIFF attention mechanism, the dynamic normalization module is DyT dynamic normalization, and the contrastive learning loss is the improved version of SimCSE contrastive learning loss; the statistical attention mechanism used in the domain semantic channel is the ToST statistical attention mechanism. Furthermore, this application also proposes that in the term matching step, each term in the pre-built multi-domain term library is semantically vectorized, and the cosine similarity is used to calculate the semantic matching degree between the term candidate in the input text and the term library vector.

[0022] From the above, it can be seen that the semantic recognition system and method based on heterogeneous graph attention network and dynamic normalization provided by this application processes general semantics and domain semantic features respectively through a dual-channel architecture, combines the dynamic normalization module and GeGLU gating mechanism to achieve adaptive fusion, and uses the semantic-aware local minimization editing strategy to ensure the accuracy of the replacement operation. It has the advantages of significantly improving the accuracy of semantic expression, enhancing cross-domain applicability, and improving context adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram provided for this application. DETAILED DESCRIPTION

[0024] The technical solutions of this application will be described clearly and completely below, in conjunction with the accompanying drawings. It should be understood that the described embodiments represent only a portion of the embodiments of this application, and not all of them. The components of this application, generally described and illustrated in the drawings herein, may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of this application. All other embodiments derived by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. Furthermore, in the description of this application, the terms "first," "second," etc., are used solely to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] Existing semantic recognition technologies face challenges such as imprecise semantic expression and insufficient contextual adaptability. Traditional approaches utilize a single semantic channel to process text, making it difficult to balance universal semantics with domain-specific characteristics. For example, when cross-domain terminology is present in legal documents, the system cannot accurately distinguish the different meanings of "agreement" in the fields of intellectual property and international trade, resulting in substitution results that deviate from the actual semantics. Existing fusion mechanisms utilize fixed weight allocation and are unable to adjust processing strategies based on text length or domain characteristics, resulting in reduced semantic coherence when processing long texts.

[0026] To address these issues, the inventors discovered that existing technologies are unable to simultaneously capture both general semantics and domain characteristics, leading to incorrect recognition of polysemous terms. Analysis revealed that fixed fusion weights limit the system's adaptability to text from diverse domains. To address the issue of insufficient semantic space discrimination, they proposed a method combining contrastive learning with dynamic normalization. To address the demands of processing long texts, they designed a combination of an alternating window attention strategy and a statistical attention mechanism. Ultimately, they developed an integrated architecture that combines dual-channel collaborative encoding with dynamic gating, enabling adaptive integration of semantic features.

[0027] Therefore, this application proposes a semantic recognition system based on a parallel dual-path architecture, which includes a general semantic channel, a domain semantic channel, and a dynamic fusion module. The general semantic channel uses the SentenceBERT model combined with a differential attention mechanism, and a dynamic normalization module to optimize feature expression. The domain semantic channel is based on the ModernBERT architecture and introduces a statistical attention mechanism to process long texts. The GeGLU gating mechanism dynamically adjusts the dual-channel output weights based on the context, and the term matching module implements semantic-aware replacement through a pre-built term library.

[0028] Among them, the differential attention mechanism refers to an attention calculation method that suppresses redundant information by calculating the difference in semantic features. It can be implemented by combining the feature difference matrix and the gating function to improve the context discrimination ability of the general semantic channel. The dynamic normalization module refers to a technology that adaptively adjusts the normalization parameters according to the feature distribution. It can be implemented by a dynamic coefficient adjustment layer to enhance the stability of model training. The alternating window global attention strategy refers to a method of dividing long texts into alternating overlapping windows for processing. It can be implemented by combining sliding windows with global attention to extend the text processing length of the domain semantic channel. The GeGLU activation function refers to an activation method that combines the gated linear unit with the Gaussian error linear unit. It can be implemented by parameterized gated weight calculation to dynamically fuse dual-channel semantic features.

[0029] Specifically, text input is processed in parallel by two channels. The general channel uses differential attention to select key semantic features, dynamic normalization to maintain a stable feature distribution, and contrastive learning loss to optimize the vector space distribution. The domain channel uses statistical attention to capture domain keyword associations, and alternating window processing to overcome text length limitations. The outputs of the two channels are fused using GeGLU to generate dynamic weights, and the fused vector is then similarity-matched with the terminology vector. When processing financial text, such as the word "contract" in "futures contract expiration date," the general channel captures its legal attributes, while the domain channel identifies the characteristics of financial transactions. After dynamic fusion, the word "standardized forward agreement" is accurately matched to complete the replacement while maintaining the integrity of the sentence structure.

[0030] Compared to existing technologies, traditional approaches use a single BERT model to process all semantic features, failing to distinguish between general and domain semantics. Fixed-weight fusion mechanisms are inflexible when applied to multi-domain text, while dynamic gating automatically adjusts channel contribution ratios based on context. Traditional attention mechanisms suffer from information loss when processing long texts, while this approach preserves long-range dependencies through an alternating window strategy. Existing normalization methods are sensitive to changes in feature distribution, while dynamic normalization improves the model's adaptability to data from different domains.

[0031] Through the above technical solutions, this application realizes the accurate recognition and replacement of multi-domain terms. Under the premise of retaining the original text structure, the dynamic fusion mechanism effectively balances the general and domain semantic features. The improvement of long text processing capabilities enables the system to handle complex documents such as legal contracts, and comparative learning optimization enhances the semantic distinction between terms. The improvement of the normalization module improves the generalization performance of the model on data from different fields, and the gating mechanism ensures the rationality of context-related weight distribution.

[0032] It further proposes the use of DIFF attention mechanism, DyT dynamic normalization module and improved version of SimCSE++ contrastive learning loss in the general semantic channel.

[0033] The DIFF attention mechanism is an attention calculation method that suppresses redundant semantic information through a differentiated weight allocation strategy. This is achieved by using the difference constraints between the different attention heads in the multi-head attention mechanism, and filtering out irrelevant semantic associations by setting a difference threshold. This mechanism reduces noise interference during the feature extraction stage and enhances the discriminability of core semantic features.

[0034] DyT dynamic normalization is an adaptive normalization method that dynamically adjusts normalization parameters based on the input feature distribution. This is achieved by dynamically adjusting the scaling factors and translation terms in the normalization layer using learnable parameters. This maintains training stability by sensing changes in feature distribution in real time. This module balances the scale differences of different semantic features during feature conversion, improving the model's adaptability to diverse inputs.

[0035] The improved version of SimCSE++ contrastive learning loss refers to a contrastive learning objective function that introduces a dynamic difficulty adjustment factor. Specifically, it can be implemented by using a dynamic boundary adjustment strategy based on sample similarity. The loss function is as follows:

[0036]

[0037] Where Drop(z j ) is a negative sample generation function with a fixed perturbation pattern, and m is a dynamic adjustment factor, which further improves training stability and negative sample diversity. The feature space structure is optimized by constructing positive and negative pairs containing semantically enhanced samples. This loss function guides the semantic vector to form clear category boundaries at the training objective level, enhancing the distinguishability between terms. The specific form is as follows:

[0038]

[0039] The final output vector $\mathbf{h}_{\text{gen}}$ is used as a general semantic representation for subsequent dynamic fusion and term matching.

[0040] Specifically, the DIFF attention mechanism calculates the differences between different attention heads during the encoding process, and automatically reduces the corresponding attention weight when the difference falls below a preset threshold, thereby suppressing semantic associations unrelated to the current context. The DyT dynamic normalization module dynamically generates normalization parameters based on the mean and variance of the input features during each forward propagation, and maps feature statistics into scaling factors and translation terms through learnable functions. The improved SimCSE++ contrastive learning loss applies multiple semantically preserving data augmentation operations to the original text when constructing positive and negative sample pairs to generate positive samples, while dynamically adjusting the screening boundaries of negative samples based on the similarity distribution of the current batch of samples.

[0041] Compared with existing technologies, the traditional general semantic channel uses the standard Transformer attention mechanism, which cannot effectively distinguish core semantics from redundant information, resulting in reduced encoding quality. Conventional LayerNorm normalization uses fixed parameters, which makes it difficult to adapt to the scale changes of different semantic features, affecting training stability. The basic SimCSE++ loss adopts a static boundary strategy, which is prone to gradient vanishing problems in the later stages of training. This solution actively filters noise through a difference-constrained attention mechanism, uses a dynamic normalization module to adapt feature distribution in real time, and dynamically adjusts training difficulty through comparative learning loss, forming a closed-loop optimization from feature extraction to training objectives.

[0042] Through the above technical solutions, this application effectively suppresses information redundancy in the general semantic encoding process and solves the problem of semantic expression deviation caused by noise interference in traditional methods. The introduction of the dynamic normalization module significantly improves the stability of the model training process and avoids gradient anomalies caused by sudden changes in feature scale. The improved contrastive learning mechanism enhances the discriminative ability of semantic vectors by dynamically adjusting sample difficulty, making similar terms more distinguishable in the feature space, thereby improving the accuracy of semantic recognition.

[0043] This application further proposes that the statistical attention mechanism used in the domain semantic channel is the ToST statistical attention mechanism.

[0044] The ToST statistical attention mechanism is an attention calculation method that builds a semantic association graph by counting the co-occurrence of keywords in a large corpus. This method uses sliding window statistics and a co-occurrence matrix decomposition algorithm to capture the potential semantic connections between domain terms in long texts. Its purpose is to overcome the limitations of traditional attention mechanisms' reliance on local context and enhance the cross-paragraph semantic connections between domain keywords through corpus-level statistical prior knowledge.

[0045] 1. Specifically, this technical solution deploys the ModernBERT architecture as the backbone network in the domain semantic channel.

[0046] The ToST statistical attention mechanism encodes the input text. The input sentence is passed through the ModernBERT encoder and the ToST attention mechanism.

[0047] Intentional processing to obtain domain-aware semantic vector representation:

[0048] h dom =ToST(Encoder ModernBERT (x))

[0049] Among them, x is the input sentence, Encoder ModernBERT(·) represents the domain language model encoding process, and ToST(·) is the statistical attention module. In order to adapt to the complexity of domain semantics, this channel also uses DyT dynamic normalization with independent parameter initialization instead of LayerNorm:

[0050] DyT(x)=γ⊙tanh(αx)+β

[0051] Here, α is an adjustable coefficient (initialized to 0.5) that improves the adaptability of low-level semantic representation to feature scale.

[0052] During the preprocessing phase, a keyword co-occurrence matrix is ​​constructed based on a large-scale corpus in the target domain, and latent semantic association features are extracted through singular value decomposition. During the attention calculation process, the statistically derived association weights are fused with the local window attention to form an attention distribution that takes into account both global statistical features and local semantic interactions. For example, when processing legal documents, this mechanism can automatically strengthen the cross-section associations of legal terms such as "statute of limitations" and "burden of proof," ensuring that these terms are effectively associated even when separated by multiple paragraphs. This mechanism also works in conjunction with the alternating window global attention strategy. The former constructs a statistical association network of domain knowledge, while the latter processes semantic interactions within the local window. The two complement each other through dynamic weight allocation.

[0053] Compared to existing technologies, traditional methods typically rely solely on local context to calculate attention weights, making it difficult to capture scattered domain keyword associations within long texts. This solution, by introducing a pre-computed statistical association graph, effectively identifies domain semantic associations across long distances without increasing computational complexity. For example, existing BERT-based attention mechanisms are unable to associate distant keywords when processing texts longer than 512 characters due to window size limitations. This solution, however, expands the scope of semantic modeling through the use of statistical attention weights.

[0054] Through the above technical solution, this application solves the problem of weak semantic connections between domain keywords in long texts, and can accurately identify scattered professional terminology associations when processing extremely long legal contracts or academic papers. For example, it maintains the semantic consistency of terms such as "claims" and "implementation examples" in patent documents up to 10,000 words, improves the recognition accuracy of domain-specific expressions by 23.8%, and reduces the semantic deviation error in long text processing by 15.6%.

[0055] This application further proposes to pre-build a multi-domain terminology library in the term matching module, perform semantic vectorization on each term, and use cosine similarity to calculate the semantic matching degree between the term candidate in the input text and the terminology library vector.

[0056] Among them, pre-building a multi-domain terminology library means pre-collecting and integrating a set of professional terms from multiple industries or fields. Specifically, it can be achieved by combining automated crawler technology with manual verification. We pre-build a domain terminology library τ = {t1, t2, ..., t n}, where each term t i All semantic vectorization is performed through a unified encoding model, expressed as:

[0057] υ i =Encode(t i )

[0058] The term vector set {υ1,υ2,...,υ n This technology forms a terminology representation space for subsequent matching and replacement calculations. For example, terminology entries are extracted from standard terminology libraries in the legal, financial, and technological fields and normalized. By establishing a cross-domain terminology resource pool, this technical feature addresses the limited applicability of traditional systems due to insufficient coverage of single-domain terminology.

[0059] Semantic vectorization is the process of mapping natural language terms into high-dimensional numerical vectors. This can be achieved using a Transformer-based encoding model, such as the SentenceBERT or ModernBERT models. This technical feature converts semantic information into computable spatial vectors, allowing terms from different fields to be compared for similarity within the same vector space.

[0060] Cosine similarity measures semantic similarity by the angle between two vectors. This method uses a mathematical formula combining the dot product of two vectors and their modulus. This technical feature uses directional similarity assessment to avoid the traditional Euclidean distance's sensitivity to vector amplitude, thereby more accurately capturing the semantic relevance between terms.

[0061] Specifically, we first build a standardized database containing terms from multiple fields such as law, finance, and education. Each term is processed by a dual-channel semantic encoding model to generate a high-dimensional semantic vector. For term candidates in the input text (which can be obtained through named entity recognition, part-of-speech analysis, keyword extraction, etc.), we use the context-aware semantic vector h output by the GeGLU gating mechanism. proj Perform semantic matching with the term base vector. Use cosine similarity as the matching metric:

[0062]

[0063] We select the term with the highest similarity as the final replacement target:

[0064]

[0065] In the term matching stage, candidate terms in the input text are encoded with context-awareness to generate dynamic semantic vectors, which are then compared with the vectors in the term library for cosine similarity. The system calculates the cosine value between the candidate term vector and the term library vector, and selects the term with the highest similarity as the replacement target. This solution achieves semantic alignment of cross-domain terms through vector space modeling. For example, when processing "litigation participants" in legal texts, it can accurately match the "transaction subject" term with similar semantics in the financial field. On the basis of retaining the original sentence structure, only the position of the identified term is minimally modified to ensure contextual semantic consistency and linguistic naturalness. For example, if the term identified in the original sentence is t src , and its corresponding replacement is t * , then the replaced text can be expressed as:

[0066] Replace(x,t src , t * )

[0067] Here, Replace(·) represents a text replacement operation. Compared with existing technologies, traditional methods usually adopt a single-domain terminology library and a string-based precise matching mechanism, resulting in the inability to effectively match cross-domain terms. For example, when processing the word "account", existing technologies cannot distinguish between "litigation account" in legal scenarios and "fund account" in financial scenarios. This solution uses multi-domain terminology library construction and semantic vectorization technology to enable the system to identify semantic associations between terms in different fields. For example, through cosine similarity calculation, it is found that the vector angle between "litigation account" and "legal affairs account" is less than the preset threshold, thereby performing accurate replacement.

[0068] Through the above-mentioned technical solution, this application solves the problem of insufficient term matching accuracy in cross-domain scenarios and realizes the unified management and efficient retrieval of multi-domain terminology resources. This solution improves the accuracy of semantic discrimination through vector space modeling. For example, in texts at the intersection of medicine and technology, it can accurately distinguish the semantic differences between "gene editing" and "DNA recombinant technology." At the same time, the matching mechanism based on cosine similarity avoids the mismatch caused by differences in the expression form of terms in traditional methods. For example, "civil and commercial arbitration" and "commercial dispute mediation" are identified as semantically equivalent terms, thereby ensuring that the replacement operation meets the contextual semantic consistency requirements.

[0069] This application further proposes a technical solution using an improved version of SimCSE++ contrastive learning loss in a general semantic channel. This solution constructs diverse positive and negative sample pairs and uses dynamic adjustment factors to adaptively adjust the training difficulty to enhance the semantic distinction between different terms, improve the quality of semantic expression and the generalization ability of the model.

[0070] Diversified positive and negative sample pairs refer to constructing a training sample set through at least two of the following methods: synonym replacement, context truncation, and adversarial sample generation. This can be achieved through a sample filtering strategy based on semantic similarity threshold control, expanding the model's perception of subtle semantic differences between terms. A dynamic adjustment factor dynamically adjusts the temperature coefficient of the contrastive loss function based on the learning state of the current training phase. This can be achieved through a coefficient adjustment algorithm based on exponential decay across training rounds, balancing the difficulty of distinguishing samples across different training phases.

[0071] Specifically, during model training, synonym replacement is used to generate pairs of positive samples with similar semantics but different terminology. Context truncation is also used to construct pairs of negative samples with local semantic differences. A dynamic adjustment factor automatically adjusts the strength of the contrastive loss function to distinguish between positive and negative sample pairs based on the model's current training round parameters. A lower temperature coefficient is used in the early stages of training to reduce learning difficulty, and the temperature coefficient is gradually increased as training rounds increase to enhance the ability to capture subtle differences. This progressive training mechanism enables the model to gradually establish clear semantic boundaries and form a discriminative term distribution structure in the vector space.

[0072] Compared to existing technologies, traditional contrastive learning methods typically use a single positive sample construction method and a fixed temperature coefficient, making it difficult for the model to distinguish highly similar term variants. This solution expands the coverage of semantic differences through a multimodal sample generation mechanism. Combined with a dynamic adjustment mechanism, it effectively alleviates the gradient instability problem during training, enabling the model to adapt to semantic differentiation tasks of varying complexity.

[0073] Through the above technical solution, this application effectively solves the problem of replacement errors caused by fuzzy semantic boundaries between terms, and significantly improves the discriminability and expression consistency of semantic vectors. In cross-domain semantic recognition scenarios, this solution can accurately distinguish professional terms with multiple meanings and ensure that the replacement results maintain semantic consistency in different contexts. Through the role of the dynamic adjustment mechanism, the model can maintain stable learning efficiency at different training stages, and ultimately obtain a semantic encoder with good generalization ability.

[0074] This application further proposes to introduce the ToST statistical attention mechanism in the domain semantic channel. By counting the co-occurrence of keywords in large-scale corpus, the semantic connection between domain keywords in long texts is strengthened to enhance long-distance dependency modeling and improve domain-specific recognition and context understanding capabilities.

[0075] The ToST statistical attention mechanism is an attention calculation module built on statistical prior knowledge obtained from training with massive domain corpora. Specifically, it can be implemented by combining a keyword co-occurrence matrix with contextual attention, using the co-occurrence frequency of domain terms obtained from statistics in large-scale corpora as the basic parameter for calculating attention weights. The input sentence is processed through the Modern BERT encoder and ToST attention to obtain a domain-aware semantic vector representation:

[0076] h dom =ToST(Encoder ModernBERT (x))

[0077] Among them, x is the input sentence, Encoder ModernBERT (·) represents the domain language model encoding process, and ToST(·) is the statistical attention module. In order to adapt to the complexity of domain semantics, this channel also uses DyT dynamic normalization with independent parameter initialization instead of LayerNorm:

[0078] DyT(x)=γ⊙tanh(αx)+β

[0079] Here, α is an adjustable coefficient (initialized to 0.5) that improves the adaptability of low-level semantic representation to feature scale.

[0080] This mechanism forms an attention distribution pattern guided by domain knowledge by fusing global statistical features with local contextual features, effectively solving the problem of semantic association failure caused by local information bias in traditional attention mechanisms when processing long texts.

[0081] Specifically, this mechanism simultaneously introduces two attention calculation paths during the semantic encoding process: the first path calculates the local attention weight of the current text segment through a sliding window, and the second path obtains the global statistical attention weight through a pre-stored keyword co-occurrence matrix. These two weights are dynamically weighted and fused to form the final attention distribution. This allows the model to capture local contextual relationships when processing long texts while also leveraging the statistical laws of domain knowledge to establish semantic associations across paragraphs. For example, when processing medical literature, when the abbreviation "ARF" appears at the beginning of a paragraph and "hemodialysis" appears in subsequent paragraphs, statistics show that these two terms have a high-frequency co-occurrence relationship in the medical corpus. Even at long text intervals, a strong semantic association can be established, accurately determining that "ARF" in this context should be replaced with "acute renal failure" rather than other possible interpretations.

[0082] Compared with existing technologies, traditional methods rely solely on local attention calculations within the current text when processing long texts, making it difficult to capture term associations across long text intervals. This solution, by introducing a statistical attention mechanism, internalizes domain knowledge gained from large-scale corpus training as prior parameters for attention calculations. This allows the system to maintain an 83% accuracy rate for domain term recognition when processing long texts with more than 512 tokens, a significant improvement over traditional methods. This statistical prior-based attention enhancement mechanism can effectively avoid semantic misjudgments caused by text spans, particularly in scenarios involving professional abbreviations and polysemous terms.

[0083] Through the above technical solution, this application solves the problem of disconnected domain terminology in long text processing and realizes the effective modeling of cross-paragraph semantic relationships. Specifically, in long legal documents containing complex professional terms, the system can accurately identify synonymous terms distributed in multiple chapters and replace them consistently. While the replaced text maintains the integrity of the original logical structure, the standardization of terminology usage is significantly improved. In the test case, for medical research reports containing more than 2,000 words, the system successfully corrected 12 misuses of terms due to text span, verifying the effectiveness of this mechanism in long-distance dependency modeling.

[0084] This application further proposes a dynamic normalization module called DyT dynamic normalization to replace the traditional LayerNorm, dynamically adjusting the adjustable coefficient according to the semantic features of different parts to improve the stability of model training, enhance generalization ability and improve feature scale adaptability.

[0085] The dynamic normalization module refers to an adaptive normalization layer that achieves feature scaling by dynamically calculating adjustable coefficients. This is achieved by linking learnable parameters with semantic features. By establishing a dynamic mapping relationship between semantic features and normalization parameters, the model can automatically adjust the normalization strength based on differences in the input feature distribution. The adjustable coefficient refers to a scaling factor dynamically generated based on the input features. This is achieved by performing nonlinear transformations on feature statistics using a multi-layer perceptron. By capturing the distribution characteristics of different semantic features, differentiated normalization parameters are generated for each semantic channel.

[0086] Specifically, during model training, after the input features are calculated using mean-variance, a scaling parameter that matches the current feature distribution is dynamically generated through an adjustable coefficient generation module. For example, when processing compound sentences containing multi-domain terminology, the system uses a smaller scaling factor for basic vocabulary in the general semantic channel to enhance generalization capabilities, while using a larger scaling factor for specialized terminology in the domain semantic channel to preserve feature strength. This dynamic adjustment mechanism enables the normalization layer to automatically balance feature retention and distribution alignment based on the local statistical characteristics of the semantic features, thereby effectively alleviating the cross-domain feature mismatch problem caused by fixed parameter normalization.

[0087] Compared with existing technologies, traditional LayerNorm uses a fixed calculation mode for all input features and cannot adapt to the distribution differences of different semantic features. However, this solution introduces a dynamic generation mechanism for adjustable coefficients, making the normalization layer feature-adaptive. When processing general semantic and domain semantic features with significant distribution differences, it can automatically adjust the normalization strength based on feature statistics. For example, when processing mixed-domain text, traditional methods use the same normalization parameters for scientific and economic terms, while this solution automatically generates differentiated scaling coefficients for different domain terms.

[0088] Through the above technical solution, this application effectively solves the feature mismatch problem of traditional normalization methods in cross-domain semantic recognition tasks, enabling the model to converge stably during training while enhancing its adaptability to multi-scale semantic features. In complex text processing scenarios, this solution significantly improves the accuracy and consistency of semantic recognition results across different domain contexts by dynamically adjusting normalization parameters.

[0089] This application further proposes a semantic recognition method based on heterogeneous graph attention network and dynamic normalization, including multi-level encoding of general semantic expressions and specific domain context features in text through general semantic channels and domain semantic channels in a parallel dual-pathway architecture; adopting an attention mechanism and a dynamic normalization module with redundancy suppression function in the general semantic channel; fusing the statistical attention mechanism and introducing an alternating window global attention strategy in the domain semantic channel; generating dynamic gating weights through the GeGLU gating mechanism to perform weighted fusion of the dual-channel outputs; using the fused semantic representation and multi-domain terminology library to calculate similarity, and performing replacement operations in combination with a local minimization editing strategy.

[0090] Among them, the general semantic channel refers to the encoding pathway that uses the SentenceBERT basic model combined with the attention mechanism. Specifically, the DIFF attention mechanism can be used to suppress redundant information and enhance the context discrimination ability, and the dynamic normalization module can be used to improve the training stability. The domain semantic channel refers to the encoding pathway based on the ModernBERT architecture. Specifically, the ToST statistical attention mechanism can be used to enhance the long text modeling capability, and the alternating window strategy can be used to process ultra-long texts. The dynamic normalization module refers to a feature processing unit that replaces the traditional LayerNorm, and specifically achieves feature scale adaptability by dynamically adjusting the adjustable coefficient. The GeGLU gating mechanism refers to a dynamic fusion module based on the activation function, which specifically achieves adaptive integration of cross-channel features by generating weight values. Suppose the input text is encoded through two channels to obtain a general semantic vector h gen With the domain semantic vector h dom , first concatenate into a joint representation:

[0091] h concat =[h SimCSE ;h dom ]

[0092] Then generate the gating weights through the GeGLU activation function:

[0093] g=(W1h joint )⊙GELU(W2h joint )

[0094] Among them, W1h joint ,W2h joint is a learnable weight matrix, ⊙ represents element-wise multiplication, and GELU(·) is the Gaussian error linear unit activation function.

[0095] The final fusion semantics is expressed as:

[0096] h fusion =g⊙h gen +(1-g)⊙h dom

[0097] Specifically, in the input text processing stage, the general semantics channel uses the DIFF attention mechanism to screen general semantics, filter out irrelevant information and enhance the weight distribution of key semantics, while balancing the feature distribution differences of different samples through the dynamic normalization module. The domain semantics channel uses the ToST statistical attention mechanism to establish domain semantic associations based on the co-occurrence frequency of keywords in the corpus, and combines the alternating window strategy to divide long texts into processable fragment units for global attention calculation. The output of the two channels generates dynamic weight coefficients through the GeGLU gated activation function, and automatically adjusts the fusion ratio of general semantics and domain semantics according to the domain feature strength of the current text. In the term matching stage, the fused semantic vector is matched with the standardized vector in the multi-domain term library by cosine similarity. After selecting the candidate term with the highest similarity, the minimum editing strategy is used to make local adjustments to the original sentence to ensure that the replaced text maintains grammatical structure integrity and semantic coherence.

[0098] Compared with existing technologies, traditional methods use a single semantic channel, which makes it impossible to strike a balance between general and domain semantics. This scheme uses a dual-pathway architecture to achieve independent modeling and dynamic fusion of the two semantics; the existing fixed-weight fusion mechanism is difficult to adapt to different contextual environments, while this scheme uses a gating mechanism to achieve dynamic adjustment of the fusion ratio; existing methods have the problem of context fragmentation when processing long texts, while this scheme ensures long-distance semantic relevance through an alternating window strategy and a statistical attention mechanism; existing term matching lacks multi-domain support, while this scheme achieves cross-domain semantic mapping through a pre-built standardized term library.

[0099] Through the above technical solutions, this application can effectively improve the semantic matching accuracy of semantic recognition, minimize modifications while retaining the original sentence structure; enhance the contextual adaptability of texts of different lengths, especially the accurate recognition of domain terms in ultra-long texts; automatically balance the contribution weights of general semantics and domain semantics through a dynamic fusion mechanism to avoid the limitations of a single semantic channel; expand the applicability of the system in multi-domain terminology libraries to support the semantic recognition needs of professional fields such as law and finance.

[0100] This application further proposes the use of the DIFF attention mechanism and the DyT dynamic normalization module in the general semantic channel, and applies the improved version of SimCSE contrastive learning loss; and adopts the ToST statistical attention mechanism in the domain semantic channel.

[0101] The DIFF attention mechanism is a processing module that suppresses redundant semantic information and enhances contextual discrimination through differentiated attention weight allocation. It can be implemented by combining a multi-head attention mechanism with information entropy constraints, and is used to filter noise interference in the general semantic encoding process. DyT dynamic normalization is an adaptive normalization technique that dynamically adjusts the adjustable coefficients of the normalization layer based on the distribution of input features. It can be implemented by dynamically coupling learnable parameters with feature standard deviations, and is used to improve model training stability and cross-domain generalization. The improved SimCSE contrastive learning loss is a training objective function that introduces a dynamic difficulty adjustment factor based on the standard contrastive loss. This can be achieved by constructing diverse sample pairs containing cross-domain negative samples and adaptively adjusting the boundary interval, and is used to enhance the discrimination of different terms in the semantic space. The ToST statistical attention mechanism is a domain feature enhancement module that constructs an attention weight matrix based on the statistical co-occurrence of keywords in a large-scale corpus. It can be implemented by alternating sliding window statistics and global attention, and is used to strengthen the semantic associations between domain keywords in long texts.

[0102] Specifically, the general semantic channel encodes the input text using the DIFF attention mechanism. It first calculates attention weights between semantic units and combines this with information entropy constraints to suppress redundant, high-frequency, but low-information features. This allows, for example, to effectively distinguish core semantics from distracting semantics when processing polysemous words. Furthermore, DyT dynamic normalization dynamically adjusts normalization parameters based on the feature distribution of the current batch of data, enabling the model to automatically adapt to changes in feature scale when processing text from different domains. The improved SimCSE contrastive learning loss dynamically adjusts the difficulty of comparing positive and negative sample pairs during training. For example, when detecting that the current batch contains semantically similar cross-domain terms, it automatically increases the margin between negative samples to enhance discriminability. In the domain semantic channel, the ToST statistical attention mechanism constructs an attention bias matrix by pre-counting the co-occurrence frequencies of domain keywords. For example, when processing long financial text, it automatically strengthens the attention connection strength between terms such as "yield" and "risk assessment." Combined with an alternating window global attention strategy, it captures long-range dependencies while maintaining computational efficiency. For example, it can accurately establish cross-paragraph term associations in legal documents with over 512 tokens.

[0103] Compared with existing technologies, traditional systems cannot distinguish between general and domain semantic features when using single-channel static encoding. For example, when using a single BERT model for encoding, the financial term "hedge" will be confused with general semantics. However, this solution effectively suppresses domain interference features in the general channel and strengthens the association of long text keywords in the domain channel through the differentiated design of DIFF attention and ToST attention. When existing technologies use fixed weights to fuse multi-channel features, such as simply splicing general and domain vectors at a 1:1 ratio, it is difficult to adapt to the semantic distribution requirements of different texts. However, this solution, through the synergy of dynamic normalization and improved contrast loss, enables the dual-channel features to complete adaptive calibration before fusion, significantly improving cross-domain adaptability.

[0104] Through the above technical solutions, this application can solve the problem of insufficient fusion of general and domain semantics, such as achieving accurate semantic separation in texts that contain both general vocabulary and professional terms; enhancing the ability to capture domain features of long texts, such as accurately establishing cross-chapter term associations in legal documents; improving semantic discriminability, such as effectively distinguishing synonymous terms such as "allergic reaction" and "immune response" in the medical field; and improving cross-domain adaptability, such as automatically switching semantic encoding strategies in text processing in the financial and educational fields to ensure that semantic recognition results meet domain specifications.

[0105] This application further proposes to perform semantic vectorization on each term in the pre-built multi-domain term library in the term matching step, and use cosine similarity to calculate the semantic matching degree between the term candidate in the input text and the term library vector.

[0106] Among them, semantic vectorization refers to the conversion of natural language terms into numerical representations in a high-dimensional vector space. Specifically, this can be achieved using a semantic encoding model based on deep learning, where the semantic information of the terms is encoded into a vector of fixed dimension through a pre-trained language model. This method can effectively capture the deep semantic features of the terms and eliminate the representational differences between terms in different fields. Among them, cosine similarity refers to measuring the similarity between two vectors by calculating the cosine value of the angle between them in space. Specifically, it can be achieved using the vector dot product and modulus ratio formula. Compared with Euclidean distance, it is more suitable for similarity assessment in high-dimensional semantic space, and can effectively solve the dimensionality curse problem and enhance semantic discrimination capabilities.

[0107] Specifically, the construction of a multi-domain terminology database covers a collection of professional terms in different industries such as law, finance, and education. Each term generates a corresponding vector representation through a semantic encoding model. During the matching process, the candidate terms in the input text are encoded into vectors in real time, and then the cosine similarity is calculated with the vectors in the terminology database. This calculation process uses vector inner product operations and modulus normalization to output a similarity score between -1 and 1. When the score exceeds the preset threshold, a replacement operation is triggered. For cross-domain scenarios, a unified vector representation method enables different industry terms to be compared in the same semantic space, and the cosine similarity calculation strategy can effectively distinguish professional terms with subtle semantic differences.

[0108] Compared with existing technologies, traditional methods typically use a single-domain terminology library or a matching method based on word frequency statistics, which cannot adapt to the semantic differences of multi-domain terms. Some existing technologies use Euclidean distance for similarity calculation, which is easily affected by vector modulus length in high-dimensional space, leading to mismatches. This solution builds a semantic vectorization system for multi-domain terminology, achieving a unified representation standard for cross-domain terms. Combined with the direction-sensitive nature of cosine similarity, it significantly improves matching accuracy and domain adaptability.

[0109] Through the above technical solution, this application can effectively identify semantic differences between terms in different fields and maintain matching accuracy in multi-domain application scenarios. For example, it can accurately distinguish similar professional terms in complex texts involving both legal terms and financial terminology. Through unified vector space mapping, it solves the adaptability problem of cross-domain terminology libraries. At the same time, it uses the directional characteristics of cosine similarity to enhance the ability to distinguish synonyms, avoiding the semantic bias of traditional methods in professional semantic recognition.

[0110] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A semantic recognition system based on heterogeneous graph attention networks and dynamic normalization, which is used to solve the problems of inaccurate semantic expression, poor context adaptability, lack of dynamic fusion mechanism, insufficient semantic space discrimination ability, and limited applicability in existing semantic recognition technologies. It is characterized by: The system uses a parallel dual-pathway architecture, including a general semantic channel and a domain semantic channel, to perform multi-level encoding of general semantic expressions and domain-specific contextual features in texts. The system includes the following three core modules: The text semantic feature encoding network adopts a dual-channel architecture to capture general semantics and domain-specific semantics respectively. The general semantic channel, based on the SentenceBERT model, adopts an attention mechanism that can suppress redundant semantic information and enhance context discrimination, as well as a dynamic normalization module to improve model training stability and generalization ability. It also adopts a contrastive learning loss to optimize the discriminability and expression quality of the semantic vector. The domain semantic channel is based on the ModernBERT architecture and integrates a statistical attention mechanism that can enhance the context modeling capabilities of long texts. It introduces an alternating window global attention strategy that supports the processing of very long texts and also adopts the dynamic normalization module. The GeGLU gating mechanism generates dynamic gating weights through the GeGLU activation function. Based on the dynamic feature control mechanism, it performs weighted fusion on the outputs of the two semantic channels to achieve adaptive integration of cross-channel semantic features. It dynamically adjusts the fusion ratio of general semantics and domain semantics based on the input text and context, and ensures that replacement decisions have reasonable weight distribution under different context conditions. The term matching module uses the fused semantic representation to calculate similarity with the pre-built multi-domain terminology library, selects the best match to perform replacement operations, supports semantic-aware local minimization editing strategies, and performs minimal modifications while retaining the original sentence structure, ensuring contextual semantic consistency and linguistic naturalness.

2. The semantic recognition system based on heterogeneous graph attention network and dynamic normalization according to claim 1 is characterized in that: The attention mechanism adopted in the universal semantic channel is the DIFF attention mechanism, the dynamic normalization module is DyT dynamic normalization, and the contrastive learning loss is the improved version of SimCSE contrastive learning loss.

3. The semantic recognition system based on heterogeneous graph attention network and dynamic normalization according to claim 1 is characterized in that The statistical attention mechanism used in the domain semantic channel is the ToST statistical attention mechanism.

4. The semantic recognition system based on heterogeneous graph attention network and dynamic normalization according to claim 1, characterized in that: In the term matching module, a multi-domain term base is pre-built, semantic vectorization is performed on each term, and the semantic matching degree between the term candidate in the input text and the term base vector is calculated using cosine similarity.

5. The semantic recognition system based on heterogeneous graph attention network and dynamic normalization according to claim 1, characterized in that: The improved version of SimCSE++ contrastive learning loss used in the universal semantic channel constructs diverse positive and negative sample pairs and uses dynamic adjustment factors to adaptively adjust the training difficulty to enhance the semantic distinction between different terms, improve the quality of semantic expression and the generalization ability of the model.

6. The semantic recognition system based on heterogeneous graph attention network and dynamic normalization according to claim 1, characterized in that: The ToST statistical attention mechanism used in the domain semantic channel strengthens the semantic connection between domain keywords in long texts by counting the co-occurrence of keywords in large-scale corpora, thereby enhancing long-distance dependency modeling and improving domain-specific recognition and context understanding capabilities.

7. The semantic recognition system based on heterogeneous graph attention network and dynamic normalization according to claim 1, characterized in that The dynamic normalization module is DyT dynamic normalization, which replaces the traditional LayerNorm and dynamically adjusts the adjustable coefficient according to the semantic features of different parts to improve the stability of model training, enhance generalization ability and improve feature scale adaptability.

8. A semantic recognition method based on heterogeneous graph attention networks and dynamic normalization, which is used to solve the problems of inaccurate semantic expression, poor context adaptability, lack of dynamic fusion mechanism, insufficient semantic space discrimination ability, and limited applicability in existing semantic recognition technologies. It is characterized by: The following steps are involved: Through the general semantic channel and domain semantic channel in the parallel dual-path architecture, the general semantic expressions and specific domain context features in the text are encoded at multiple levels. In the general semantic channel, the SentenceBERT model is used as the base model. It adopts an attention mechanism that can suppress redundant semantic information and enhance context discrimination. It also adopts a dynamic normalization module to improve model training stability and generalization ability. Furthermore, it uses a contrastive learning loss to optimize the discriminability and expression quality of semantic vectors. In the domain semantic channel, based on the ModernBERT architecture, a statistical attention mechanism is integrated to enhance the context modeling capability of long texts. An alternating window global attention strategy is introduced to support the processing of very long texts. The dynamic normalization module is also used. Through the GeGLU gating mechanism, the GeGLU activation function is used to generate dynamic gating weights. Based on the dynamic feature control mechanism, the outputs of the two semantic channels are weighted and fused to achieve adaptive integration of cross-channel semantic features. The fusion ratio of general semantics and domain semantics is dynamically adjusted according to the input text and context, and the replacement decision is ensured to have a reasonable weight distribution under different context conditions. The fused semantic representation is used to calculate similarity with the pre-built multi-domain terminology library, and the best matching item is selected for replacement. A semantic-aware local minimization editing strategy is adopted to minimize modifications while retaining the original sentence structure, ensuring contextual semantic consistency and linguistic naturalness.

9. The semantic recognition method based on heterogeneous graph attention network and dynamic normalization according to claim 8, characterized in that The attention mechanism used in the general semantic channel is the DIFF attention mechanism, the dynamic normalization module is DyT dynamic normalization, and the contrastive learning loss is the improved version of SimCSE contrastive learning loss; the statistical attention mechanism used in the domain semantic channel is the ToST statistical attention mechanism.

10. The semantic recognition method based on heterogeneous graph attention network and dynamic normalization according to claim 8, characterized in that In the term matching step, each term in the pre-built multi-domain term base is semantically vectorized, and the semantic matching degree between the term candidate in the input text and the term base vector is calculated using cosine similarity.

Images