AI semantic analysis and data processing method based on large language model

Abstract

The invention relates to the field of data processing, in particular to an AI semantic analysis and data processing method based on a large language model, which comprises the following steps: firstly, extracting professional terms and context windows in a vertical field text, extracting hidden layer vectors by using the large language model, calculating spatial dispersion, and dividing the terms into three-state working modes according to semantic stability; aiming at ambiguous terms, independent semantic branches are identified through vector clustering, accurate semantic affiliation under a new context is realized by adopting co-occurrence word matching and vector distance judgment, dynamic segmentation adjustment is performed on static word frequency (TF-IDF) weights according to the accurate semantic affiliation, and finally, the improved weights are injected into a downstream analysis process. According to the method, the same term is endowed with differentiated weight expressions in different contexts, the problem of term ambiguity is efficiently solved through a low-overhead cascade disambiguation mechanism, structured data confusion and document error clustering are avoided, and the accuracy of tasks such as text matching, information extraction and document classification is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, specifically to an AI semantic analysis and data processing method based on a large language model. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have demonstrated powerful text understanding capabilities in the field of natural language processing. AI semantic analysis and data processing methods based on LLMs have now become core technologies in text mining, information extraction, and knowledge graph construction. This technology mainly uses deep neural networks to extract features, segment words, and encode semantics from raw text, thereby achieving efficient and automated analysis of massive amounts of unstructured text data.

[0003] In real-world vertical sectors such as healthcare, systems often need to process massive amounts of specialized documents in batches, including clinical records, treatment guidelines, and medical literature. In the daily workflow of these specific scenarios, semantic analysis systems need to accurately identify specialized entities, classify and cluster documents, and archive structured data. Because texts in specialized scenarios contain extremely dense technical terms, and the final business requirements demand extremely high accuracy in data processing, how well the system accurately understands the true meaning of each technical term in the document within its specific context directly affects the usability of downstream business operations.

[0004] However, existing semantic analysis solutions face a pressing practical pain point when delving into vertical business scenarios: in real-world cross-business scenarios, such as processing medical records from different departments, many professional terms exhibit severe polysemy. Current technologies typically employ fixed, static term frequency weights (such as standard TF-IDF) to treat terms in a one-size-fits-all manner, failing to recognize the semantic differentiation that occurs within the context. This leads to a situation where, in actual structured information entry or medical record archiving scenarios, the system easily confuses drastically different diagnostic and treatment methods across different business contexts, resulting in serious operational anomalies such as errors in structured data field entry, confusion in business ledgers, and misclassification of medical records, ultimately reducing the accuracy of data processing in specialized fields. Summary of the Invention

[0005] To address the issues raised by existing technologies that typically employ fixed static word frequency weights, potentially leading to errors in structured data field input, this invention proposes an AI semantic analysis and data processing method based on a large language model. This method includes: performing word segmentation and term extraction on vertical domain text; constructing a contextual fragment set by extracting local context windows using term occurrence positions as anchor points; calculating the static basic weights of each term; feeding the contextual fragments into the encoding layer of a large language model to extract hidden layer output vectors; classifying terms into three semantic stability levels based on the dispersion of all hidden layer output vectors relative to the centroid vector of the same term; and constructing a system containing a first working mode, a second working mode, and a third working mode. The three-state semantic working mode state machine maps terms to corresponding modes according to the semantic stability level. The first working mode maintains the static basic weights unchanged, while the second working mode is subject to a fixed ratio decay. Hidden layer output vector clustering is performed on the terms of the third working mode to identify independent semantic branches, and the characteristic co-occurrence word set and centroid vector of each branch are extracted. For the terms of the third working mode in the target document, the semantic branch affiliation is determined through a two-level cascade mechanism of co-occurrence word matching and vector distance determination, and segmented weight adjustment is performed based on the branch universality. The adjusted weights replace the static basic weights and are injected into the semantic analysis process.

[0006] Compared to traditional text processing methods that rely solely on static word frequency (such as TF-IDF) or single context analysis, this invention can accurately identify the polysemy and semantic drift of the same professional term in different contexts in vertical professional fields with extremely high requirements for word meaning, such as medical diagnostic records and legal contract analysis. By introducing hidden layer vectors of a large language model and a three-state semantic working mode state machine, it not only avoids retrieval omissions or misjudgments caused by rare word meanings in professional terms being masked by mainstream public semantics, but also dynamically and adaptively restores the true weight of the term in the current specific business scenario based on the context. This significantly improves the accuracy and recall rate of vertical industry knowledge graph construction, intelligent question answering systems, and precise document retrieval.

[0007] Furthermore, the method of extracting a local context window using the term's location as an anchor point also includes: expanding forward and backward along the term's location by a preset number of words to form a symmetrical context window; when the term's location is less than the number of words to be expanded from the beginning or end of the document, extracting all words within the available range and filling the insufficient portion with preset blank placeholders to maintain a consistent total number of words in all context windows.

[0008] Compared to conventional variable-length truncation or unidirectional truncation strategies, when processing texts of varying lengths in automated tag extraction or document summarization scenarios, this approach ensures that the local semantic perception field of the large language model always maintains strict symmetry and uniformity. It prevents the model's attention mechanism from shifting due to missing information at document edges, thus guaranteeing the system's extremely high stability when extracting contextual features at any text location.

[0009] Furthermore, the degree of dispersion is characterized by the coefficient of dispersion, which is calculated as follows: the arithmetic mean of all hidden layer output vectors of the same term is calculated to obtain the centroid vector; the Euclidean distance between each hidden layer output vector and the centroid vector is calculated one by one to obtain a set of deviation distance values; and the ratio of the standard deviation of the deviation distance values ​​to the arithmetic mean is used as the coefficient of dispersion of the term.

[0010] Compared to the traditional method of judging term stability by relying solely on human experience or simple word frequency fluctuations, this method provides a standardized vector space quantitative index system. In the scenario of large-scale automated cleaning of massive engineering and technical documents, it can more scientifically and objectively filter out highly discrete words that are prone to ambiguity, effectively reducing data noise interference caused by subjective judgment or different standards of different cleaning personnel.

[0011] Furthermore, the three semantic stability levels are divided as follows: the discrete coefficients of all terms in the current corpus are statistically analyzed, the 30th percentile of the discrete coefficient distribution is taken as the first level boundary, and the 70th percentile is taken as the second level boundary; terms with discrete coefficients lower than the first level boundary are classified into the first stability level, terms with discrete coefficients higher than the second level boundary are classified into the third stability level, and terms between the two level boundaries are classified into the second stability level.

[0012] Furthermore, the fixed-proportion attenuation applied to the static base weights in the second working mode is achieved through an attenuation factor, the value of which ranges from 0.50 to 0.85.

[0013] Furthermore, the hidden layer output vector clustering adopts the K-means clustering algorithm, and the number of clusters is adaptively determined by the optimal silhouette coefficient criterion. The specific determination method is as follows: starting from the number of clusters equal to 2, gradually increasing to the preset maximum number of clusters, performing clustering for each candidate number of clusters and calculating the corresponding silhouette coefficient, and selecting the number of clusters when the silhouette coefficient reaches the maximum value as the optimal number of semantic branches for the term.

[0014] Compared to traditional clustering algorithms that require a fixed number of categories, when analyzing the ever-emerging new polysemous words in scientific research papers or patent documents, this algorithm can automatically find the number of semantic branches that best fits the actual evolution of the term by relying on the contour coefficient. This avoids semantic adhesion caused by forced classification or semantic fragmentation caused by excessive segmentation, and gives the system a strong unsupervised adaptive capability when facing novel professional vocabulary expressions.

[0015] Furthermore, the extraction method of the iconic co-occurrence word set is as follows: perform word frequency statistics on the context fragments within each cluster, calculate the ratio between the occurrence frequency of each non-stop word in the target cluster and its occurrence frequency in all other clusters, and include words whose ratio exceeds a preset discrimination multiple threshold into the iconic co-occurrence word set of the corresponding semantic branch; at the same time, mark the semantic branch containing the most occurrence positions as the mainstream semantic branch, and the rest as non-mainstream semantic branches.

[0016] Compared to conventional feature extraction methods that only count the frequency of a single word, this method can overcome noisy words and accurately extract truly distinctive and distinctive related words when facing extremely complex contextual environments such as news aggregation and massive text mining. By clearly defining the primary and secondary semantic branches, it ensures that the system can also perform independent feature labeling and tracking when encountering rare usages in niche professional fields, preventing important long-tail semantics from being completely submerged by commonly used semantics.

[0017] Furthermore, the two-level cascade mechanism includes two levels of determination. The first level of determination is fast co-occurrence word matching, specifically: extracting non-stop words from the local context window of the term's occurrence position in the target document to form the current context word set; performing an intersection operation between the current context word set and the characteristic co-occurrence word set of each semantic branch to count the number of hits; when the difference between the first and second ranked branches in terms of the number of hits exceeds a preset determination interval threshold, the branch is determined to be the first ranked branch; the second level of determination is activated when the first level of determination does not give a valid result, sending the current context fragment of the term into the encoding layer of the large language model to extract the hidden layer output vector, and determining the branch with the smallest cosine distance to the centroid vector of each semantic branch as the semantic belonging.

[0018] Furthermore, segmented weight adjustments are performed based on the generality of the branches, including: when a term belongs to the mainstream semantic branch in the target document, its static basic weight remains unchanged; when a term belongs to a non-mainstream semantic branch, the ratio of the total number of semantic branches of the term to the number of mainstream branches is used as an enhancement factor, and the static basic weight is multiplied by the enhancement factor to compensate for the frequency contribution of non-mainstream semantic branches being diluted by mainstream semantic branches during the statistical process.

[0019] Furthermore, it also includes the following anomaly handling mechanism: when the cosine distance between the hidden layer output vector and the centroid vectors of all semantic branches in the second-level determination exceeds the preset maximum attribution distance threshold, the term is output with weight according to the fixed attenuation strategy of the second working mode.

[0020] The technical effects of this invention are as follows: This invention calculates the discreteness by extracting hidden layer vectors from a large model, dynamically classifying specialized terms into three semantic stability levels. For polysemous words, it innovatively employs vector clustering to identify independent semantic branches, and combines a two-level judgment mechanism of co-occurrence words / vector distance with a niche frequency compensation strategy to accurately restore the true weight of terms in complex contexts. This solution addresses the pain point of non-mainstream and rare semantics being diluted by mainstream semantics in professional scenarios, significantly improving the accuracy of AI retrieval and information extraction. Attached Figure Description

[0021] Figure 1 This is a schematic flowchart illustrating an AI semantic analysis and data processing method based on a large language model according to an embodiment of the present invention. Figure 2 This is a schematic bar chart illustrating the comprehensive performance comparison of the two methods in different downstream tasks in the embodiments of the present invention; Figure 3 This is a schematic graph illustrating the impact of unregistered new terms in open-domain text on system robustness in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The server-side inference cluster uses servers equipped with high-performance graphics processing units (GPUs) as its core computing units. Each server is configured with at least two GPUs capable of accelerating tensor operations, with each GPU having at least 40GB of video memory to support the coding layer inference tasks of large language models. The servers are interconnected via a high-speed network with a bandwidth of at least 100Gbps to support parameter synchronization and intermediate result transmission during parallel model inference. For example, in this embodiment, the inference cluster consists of four servers with dual GPUs, each with 80GB of video memory, for a total available video memory of 640GB. This is sufficient to fully load a large language model with seven billion parameters and reserve ample inference buffer space. Preferably, the number of servers in the inference cluster can be adjusted from 2 to 16 to accommodate the deployment requirements of large language models with different parameter scales.

[0024] The data preprocessing node employs a general-purpose computing server equipped with a multi-core CPU. The CPU has at least 32 cores, a clock speed of at least 2.5 GHz, and at least 128 GB of system memory. It handles computationally intensive preprocessing tasks such as text segmentation, term index construction, word frequency statistics, and context window extraction, but does not rely on GPU acceleration. The preprocessing node and the inference cluster communicate via a message queue middleware for task scheduling and data transfer. The message queue uses a first-in, first-out (FIFO) persistent model to ensure that tasks are not lost due to network interruptions or node restarts.

[0025] At the storage level, both the original text dataset and intermediate computation results are stored in a distributed file system with a total storage capacity of no less than 10TB and a read / write throughput of no less than 1GB / s, to meet the requirements of batch reading of large-scale corpora and high-frequency writing of clustering results. Structured metadata, such as the term index table, semantic stability level labels, working mode labels, and the set of characteristic co-occurring words for each semantic branch, are stored in a relational database, with an index built using term identifiers as the primary key to support fast queries in the subsequent online service phase.

[0026] It should be noted that the above hardware configuration is a preferred solution in this embodiment. In other embodiments, a single high-configuration server can be used to replace the distributed cluster to achieve the same function, or elastic computing instances provided by a cloud computing platform can be used to allocate resources on demand. This invention does not limit the specific hardware brand, model and deployment form.

[0027] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] An example of an AI semantic analysis and data processing method based on a large language model: like Figure 1As shown, the AI ​​semantic analysis and data processing method based on a large language model according to the present invention includes: S101. Perform word segmentation on the original text of the vertical domain and use terms as anchors to extract local context windows to construct a set of contextual fragments of terms.

[0029] First, the system loads raw text datasets from the vertical domain in batches from the distributed file system. In this embodiment, the dataset originates from clinical medical records, treatment guidelines, and medical literature in the medical field, containing 50,000 documents with a total character count of approximately 200 million characters. The system performs domain-based dictionary-enhanced word segmentation on each document. This segmentation employs a maximum probability path segmentation algorithm based on directed acyclic graphs and loads a pre-compiled medical terminology dictionary as a user-defined dictionary to ensure that professional terms are not incorrectly segmented during the word segmentation stage. For example, the terminology dictionary contains 120,000 entries, covering categories such as disease names, generic drug names, surgical procedures, and laboratory indicators. Preferably, the size of the terminology dictionary can be adjusted between 50,000 and 500,000 entries to adapt to the terminology density of different vertical domains.

[0030] Furthermore, after word segmentation is completed, the system iterates through the segmentation results of all documents, extracts all terms matched in the terminology dictionary, assigns a unique term identifier to each term, and builds a term index table. This index table records all occurrence positions of each term in the corpus, and each position record includes the document identifier of the document and the starting position offset of the term in the document's token sequence.

[0031] Then, using each occurrence of a term as an anchor point, a local context window is extracted from the token sequence of the document containing that term. The extraction rule is that the number of tokens extending forward from the term's occurrence position is equal to the number of tokens extending backward, both set to a fixed window radius. In this embodiment, the window radius is preferably set to 25 tokens, meaning each context window contains the term itself and 25 tokens before and after it, for a total window length of 51 tokens. This length is determined based on the fact that in Chinese medical texts, the average length of a complete clinical descriptive sentence is approximately 15 to 30 tokens. A window radius of 25 tokens can cover the complete main structure of the sentence containing the term and part of the content of at least one adjacent sentence, thus fully capturing the local semantic environment of the term at the current position. In practical applications, depending on the average sentence length differences in the target domain texts, this window radius can be adjusted within the range of 15 to 50 tokens.

[0032] Next, the above-mentioned truncation operation is performed on all occurrences of the same term in the corpus, and all the obtained context windows are aggregated into a set of contextual fragments for that term. The system simultaneously counts the number of times each term appears in each document and the number of documents in the entire corpus where the term appears. Following the standard word frequency multiplied by the inverse document frequency calculation process, the static basic weight of each term in each document is obtained.

[0033] Specifically, if a term appears in a document at a distance less than one window radius from the beginning or end of the document (i.e., it touches the document boundary when expanding forward or backward), the system will truncate all tokens within the available range as the window content on that side and fill the insufficient portion with preset blank placeholders. This boundary handling mechanism ensures that the total number of tokens in each context window remains consistent, avoiding tensor concatenation anomalies caused by inconsistent input lengths in subsequent encoding stages.

[0034] It should be noted that although this embodiment uses a fixed window radius for truncation, other embodiments may also use a complete sentence truncation method based on punctuation marks, that is, expanding to the boundaries of adjacent complete sentences on both sides of the sentence containing the term as the core, to achieve the same context coverage effect.

[0035] S102. Input all contextual fragments of the term into the coding layer of the large language model to extract the hidden layer output vector and calculate the degree of discreteness of the vector space to determine the semantic stability level of the term.

[0036] First, the system groups the set of contextual fragments for each term obtained in the previous step according to the term identifier. For each term, all its contextual fragments are fed into the encoding layer of the large language model for forward inference. In this embodiment, the large language model adopts a pre-trained language model based on the Transformer architecture, whose encoding layer contains 32 layers of multi-head self-attention modules, with a hidden layer dimension of 4096 and 32 attention heads. After each contextual fragment is processed by the encoding layer, the system extracts the output vector of the last hidden layer at the corresponding token position of the term. The dimension of this vector is consistent with the hidden layer dimension of the model, which is 4096 dimensions. Preferably, the hidden layer output vector can also be taken from the second-to-last or fourth-to-last layer of the encoding layer. The outputs of different layers have different levels of semantic abstraction, and the optimal extraction layer can be selected according to the specific task requirements.

[0037] Furthermore, for the same term, the system collects its hidden layer output vectors at all occurrence locations, forming a vector sample set for that term. Then, spatial distribution morphology analysis is performed on this sample set: first, the arithmetic mean of all vectors is calculated dimension-wise to obtain the centroid vector of the term; then, the Euclidean distance between each hidden layer output vector and the centroid vector is calculated one by one, resulting in a set of deviation distance values; next, the arithmetic mean and standard deviation of this set of deviation distances are calculated, and finally, the ratio of the standard deviation to the arithmetic mean is used as the coefficient of variation for the term. It is important to note that when the arithmetic mean is 0, the coefficient of variation is directly determined to be 0 to avoid division by zero errors.

[0038] Then, when the discrete coefficient approaches zero, it indicates that the hidden vectors of the term are tightly clustered around the centroid at all occurrence positions. The large language model's encoding of the term is highly consistent across different contexts, and the semantic expression of the term does not change substantially with the context, belonging to a semantically stable monosemous term. A significantly high discrete coefficient indicates that the hidden vectors are dispersed over a large area around the centroid in high-dimensional space. The large language model provides significantly different semantic encodings for the term in different contexts, and the semantics of the term drifts significantly with the context, exhibiting polysemy.

[0039] Next, the system needs to determine two threshold values ​​required to classify terms into three semantic stability levels. In this embodiment, the system calculates the discrete coefficient values ​​of all terms in the current corpus, arranges these values ​​in ascending order, and, given the significant differences in the overall semantic fluctuation benchmarks of corpora from different vertical domains, takes the value at the 30th percentile as the first threshold and the value at the 70th percentile as the second threshold. For example, in the medical corpus of this embodiment, the calculated result of the first threshold is 0.12, and the calculated result of the second threshold is 0.38. In practical applications, the percentile value of the first threshold can be adjusted within the range of the 20th to 40th percentile, and the percentile value of the second threshold can be adjusted within the range of the 60th to 80th percentile.

[0040] Subsequently, the system determines the semantic stability level of each term based on the two hierarchical boundaries mentioned above. Terminators with a dispersion coefficient below the first hierarchical boundary are labeled as having a first stability level, i.e., a semantically stable level. Terminators with a dispersion coefficient above the second hierarchical boundary are labeled as having a third stability level, i.e., a highly semantically divergent level. Terminators with a dispersion coefficient between the first and second hierarchical boundaries are labeled as having a second stability level, i.e., a slightly semantically drifting level.

[0041] Specifically, if the frequency of a term in the corpus is lower than a preset minimum sample size threshold, the system directly labels the term as the second stability level and records the low sample count. In this embodiment, the minimum sample size threshold is preferably set to 10 times. When the frequency of a term is too low, the sample size of its hidden layer vector is insufficient to support the statistical reliability of the discrete coefficients, and forced calculation may produce artificially high or low results due to sampling bias. Classifying such terms into the second stability level is a conservative strategy, allowing them to undergo appropriate weight decay in subsequent processes to avoid extreme misjudgments due to insufficient statistics. Preferably, the minimum sample size threshold can be adjusted within the range of 5 to 30 times.

[0042] It should be noted that although this embodiment uses the coefficient of variation as a measure of the degree of dispersion, in other embodiments, the mean cosine distance can also be used as a measure. That is, the cosine distance between each hidden layer vector and the centroid vector is calculated and the average value is taken, which replaces the coefficient of variation under the Euclidean distance system to achieve the same semantic stability evaluation effect.

[0043] S103. Construct a three-state semantic working mode state machine and map each term to the corresponding discrete working mode according to the semantic stability level to execute a hierarchical processing strategy.

[0044] First, the system initializes a semantic state machine containing three discrete working modes. The three working modes are defined as the first working mode, the second working mode, and the third working mode, respectively. There are no dynamic jump paths between the modes, and each term belongs to only one unique working mode within the processing cycle of the current batch of corpus.

[0045] Furthermore, the system reads the semantic stability level label assigned to each term in the previous step and assigns the terms to the corresponding working modes according to a deterministic one-time mapping rule. Terminators labeled with the first stability level enter the first working mode, terms labeled with the second stability level enter the second working mode, and terms labeled with the third stability level enter the third working mode.

[0046] Then, the system executes a pass-through strategy for terms entering the first working mode. Since the semantics of the terms corresponding to the first working mode are highly consistent across all occurrence positions, the large language model's encoding of them does not substantially change with the context. Any additional weight correction operation is redundant computational overhead and may introduce unnecessary numerical perturbations to the already accurate weight values. Therefore, the specific operation of the pass-through strategy is as follows: read the static basic weight value calculated for the term in step S101, apply no correction operation, write the weight value into the weight output buffer, and wait for direct invocation in subsequent semantic matching calculation stages.

[0047] Next, the system executes a fixed attenuation strategy on the terms entering the second working mode. The specific operation of the fixed attenuation strategy is as follows: read the static base weight value of the term, multiply it by a preset attenuation factor, and write the product into the weight output buffer. Since the semantics of the terms corresponding to the second working mode remain consistent in most occurrence positions, but identifiable shifts occur in a few specific contexts, these shifts have not yet formed independent semantic branches, but have already mixed uncertainties caused by contextual differences into the weight contribution of the term. In this embodiment, the attenuation factor is preferably set to 0.70, that is, the attenuated weight is 70% of the original weight. Attenuating the weight to 70% retains the main semantic contribution of the term while compressing the influence of uncertainties to a level that does not interfere with the downstream ranking results. In practical applications, depending on the average degree of polysemy of terms in the target domain, this attenuation factor can be adjusted within the range of 0.50 to 0.85.

[0048] Subsequently, the system marks the terms that have entered the third working mode as terms to be disambiguated and adds them to the term queue. The semantic encoding of the terms corresponding to the third working mode shows significant differentiation in different contexts, forming multiple distinguishable semantic branches, which need to be activated in subsequent steps to complete the polysemy disambiguation process.

[0049] Specifically, if the system detects an inconsistency between the term index table and the semantic stability level label table during the execution mode assignment process—that is, a term exists in the index table but is missing in the level label table—the system defaults the working mode of that term to the second working mode and generates an exception log record. This fault tolerance mechanism ensures that even if a single term is missed in the middle of the data pipeline, the overall process can still continue to execute without being interrupted by a single point of failure.

[0050] It should be noted that although the state machine mode in this embodiment belongs to a one-time static mapping, an incremental update mechanism can also be introduced in other embodiments. That is, when the system receives a new batch of corpus data, it updates the discrete coefficients of existing terms in a rolling manner and re-evaluates their stability level and working mode based on the updated discrete coefficients to adapt to the application scenario of continuous corpus expansion.

[0051] S104. Perform hidden layer vector clustering operation on the terms to be disambiguated in the third working mode and determine the optimal number of semantic branches based on the silhouette coefficient to identify each independent semantic branch.

[0052] First, the system reads term identifiers one by one from the queue of terms to be disambiguated, and retrieves the hidden layer output vector of the term at all occurrence positions from the vector sample set cached in step S102.

[0053] Furthermore, the system performs clustering operations on all hidden layer output vectors of the term to group semantically similar occurrences into the same group, with each group corresponding to an independent semantic branch of the term. The hidden layer output vector clustering employs a centroid-based iterative K-means clustering algorithm. Before clustering, the system needs to determine the optimal number of clusters, i.e., the actual number of semantic branches existing for the term. This embodiment uses the optimal silhouette coefficient criterion for adaptively determining the number of clusters: the system starts with a cluster number of 2 and gradually increases it to a preset maximum cluster number upper limit, performing a complete K-means clustering process once for each candidate cluster number and calculating the silhouette coefficient corresponding to the clustering result. In this embodiment, since the number of independent semantic branches of a single professional term in vertical domain text rarely exceeds 8, clustering results exceeding this upper limit usually reflect over-segmentation caused by noise rather than true semantic differentiation in engineering practice. Therefore, the maximum cluster number upper limit is preferably set to 8, and preferably, the maximum cluster number upper limit can be adjusted within the range of 4 to 12.

[0054] Then, the system compares the silhouette coefficient values ​​corresponding to the number of candidate clusters and selects the number of clusters with the maximum silhouette coefficient as the optimal number of semantic branches for that term. The silhouette coefficient measures the balance between intra-cluster compactness and inter-cluster separation; a higher value indicates that the members within each cluster are more tightly packed and the intervals between clusters are clearer. The number of clusters corresponding to the peak of the silhouette coefficient means that each semantic branch maintains semantic cohesion internally while also having discernible semantic intervals between them.

[0055] Next, the system uses the optimal number of semantic branches as input parameter to re-execute K-means clustering on all hidden layer output vectors of the term, obtaining the final clustering result. In this embodiment, the maximum number of iterations of the K-means algorithm is set to 300, and the convergence criterion is that the maximum value of the centroid displacement between two consecutive iterations is less than 1e-4. Preferably, the maximum number of iterations can be adjusted within the range of 100 to 500. After clustering, each semantic branch corresponds to a cluster, and the system records the centroid vector of each cluster and the indices of all occurrence positions contained in that cluster.

[0056] Subsequently, the system performs word frequency statistics on the context fragments within each cluster to extract characteristic words that can identify that semantic branch. Specifically, it counts the frequency of each non-stop word in all context fragments within a target cluster, and simultaneously counts the frequency of that word in the context fragments of all other clusters, calculating the ratio between the frequency within the target cluster and the frequencies in other clusters. It is important to note that in the above calculation, a basic constant of 1 needs to be pre-added to both the numerator and denominator frequency statistics before calculating the ratio, thus avoiding the risk of data errors caused by a denominator of 0.

[0057] If the ratio exceeds a preset discrimination multiple threshold, the word is included in the set of iconic co-occurring words of the semantic branch corresponding to the target cluster. In this embodiment, the discrimination multiple threshold is preferably set to 3.0, meaning that a word must appear at least 3.0 times more frequently in the target cluster than in all other clusters to be selected. Preferably, this threshold can be adjusted within the range of 2.0 to 5.0. The system performs the above operation for each cluster of the term, ultimately obtaining the set of iconic co-occurring words for each semantic branch.

[0058] Furthermore, the system counts the number of occurrence positions contained in each semantic branch, marking the branch with the most occurrence positions as the mainstream semantic branch, and the remaining branches as non-mainstream semantic branches. This marking will be used in the weight adjustment strategy in subsequent steps.

[0059] Specifically, if the silhouette coefficients corresponding to all candidate clusters of a term are lower than a preset minimum silhouette coefficient threshold (set to 0.15 in this embodiment), it indicates that although the hidden layer vectors of the term have a high degree of overall dispersion, they do not form a clearly separable cluster structure in the vector space, but rather exhibit a diffuse distribution. In this case, the system forcibly sets the semantic branch number of the term to 1, does not perform subsequent co-occurrence word extraction operations, and downgrades the term's working mode from the third working mode to the second working mode, processing its weights according to a fixed attenuation strategy.

[0060] It should be noted that although this embodiment uses the K-means algorithm as the clustering method, in other embodiments, the density-based DBSCAN clustering algorithm or the Gaussian mixture model-based EM clustering algorithm can also be used to achieve the same semantic branch recognition effect. When using the DBSCAN algorithm, there is no need to preset the number of clusters, and it can automatically discover cluster structures of arbitrary shapes, making it suitable for scenarios where semantic branches are non-convex in the vector space.

[0061] S105. Construct a two-level cascaded semantic attribution determination mechanism and perform fast co-occurrence word matching and fine vector distance determination on polysemous terms in the target document to determine the semantic branch attribution.

[0062] First, when the system receives a new target document for online semantic analysis, it performs the same word segmentation and term recognition operations as in step S101 to obtain all the terms appearing in the target document and their position information in the document token sequence.

[0063] Furthermore, the system queries the working mode label assigned to each term in step S103 one by one. For terms in the first and second working modes, the system directly obtains their weight output according to the direct pass strategy and fixed attenuation strategy determined in step S103, without proceeding to the subsequent determination process in this step. For terms in the third working mode, the system initiates a two-level cascaded semantic attribution determination mechanism.

[0064] Then, the system performs the first-level judgment, namely, fast co-occurrence word matching. Specifically, using the location of the term in the target document as the anchor point, a local context window with the same radius as in step S101 is extracted, and all non-stop words within the window are extracted to form the current context word set. Next, the current context word set is intersected with the characteristic co-occurrence word sets of each semantic branch of the term in step S104, and the number of hits between the current context word set and the characteristic co-occurrence word sets of each branch is counted. The system sorts the number of hits for all branches. If the difference between the number of hits of the first-ranked branch and the number of hits of the second-ranked branch exceeds a preset judgment interval threshold, the system determines that the first-level judgment result is valid and identifies the semantic belonging of the term in the current document as the branch with the highest number of hits. In this embodiment, the judgment interval threshold is preferably set to 2, meaning that the number of hits of the first-ranked branch must be at least 2 more than the number of hits of the second-ranked branch for direct judgment. Preferably, this threshold can be adjusted within the range of 1 to 4.

[0065] Next, if the first-level judgment fails to produce a valid result, i.e., the difference in the number of hits between the first and second ranked branches does not reach the judgment interval threshold, the system automatically initiates the second-level judgment, namely, fine-grained judgment based on embedding vector distance. Specifically, the local context window of the term in the target document is fed into the encoding layer of the large language model, and the hidden layer output vector of the term in the current context is extracted. Then, the cosine distance between this vector and the centroid vectors of each semantic branch in step S104 is calculated, and the branch with the smallest cosine distance is determined as the semantic affiliation of the term in the current document.

[0066] The cascaded design of the two-level decision mechanism follows a progressive principle from low overhead to high accuracy. Co-occurrence word fast matching is a lightweight set intersection operation that does not involve forward inference of the large language model, resulting in extremely low computational overhead and efficient handling of most cases with distinct contextual features. Embedded vector distance decision requires calling the large language model's encoding layer to perform forward inference once, with significantly higher computational overhead than the first-level decision, but it can handle boundary cases where contextual words lack significant distinguishability. For example, in the medical corpus of this embodiment, the effective hit rate of the first-level decision is approximately 75%, meaning that about three-quarters of the ambiguous terms can be directly determined through co-occurrence word matching, and only about one-quarter of the terms require triggering the second-level decision. This mechanism reduces the frequency of calls to the large language model's encoding layer during the online service phase by approximately 75% compared to a scheme that performs vector distance decision for all ambiguous terms.

[0067] Specifically, if the cosine distance between the hidden layer output vector of a term and the centroid vectors of all semantic branches exceeds a preset maximum attribution distance threshold (set to 0.85 in this embodiment), the system will determine that the semantics of the term in the current context do not fall within the coverage of any known branch and are therefore considered unseen semantics. In this case, the system downgrades the weight processing strategy for the term to a fixed decay strategy in the second working mode, outputting weights conservatively, and simultaneously records the context fragment in the pending review log for subsequent manual review and incremental expansion of semantic branches. In open-domain text processing, it is impossible to exhaustively enumerate all semantic uses of a term during the corpus construction phase. When a new context not covered during the training phase appears in the target document, forcibly classifying it into an existing branch will lead to uncontrollable misjudgment risks. Therefore, a decay strategy can be used to suppress the weight influence of the term and retain evidence for subsequent iterations.

[0068] It should be noted that although this embodiment uses the number of hits of iconic co-occurring words as the matching metric in the first-level determination, other embodiments may also use co-occurring word matching scores based on TF-IDF weighting. That is, each hit iconic co-occurring word is weighted and accumulated according to its TF-IDF weight in the target cluster, in order to replace the simple counting method and achieve a more refined contextual discrimination effect.

[0069] S106. Based on the semantic branch attribution results, implement a segmented weight adjustment strategy based on branch universality for the terms of the third working mode to complete the improvement of static word frequency weight.

[0070] First, the system reads the semantic branch attribution results determined for each third working mode term in the previous step, and at the same time reads the mainstream and non-mainstream labeling information of each semantic branch recorded in step S104, as well as the total number of semantic branches of the term.

[0071] Furthermore, the system performs segmented adjustments to the static base weights of the term based on the semantic branch attribution results. The adjustment strategy is based on the conditional discrimination rule of branch prevalence: since the mainstream semantic branch appears most frequently in the corpus, its semantic connotation is closest to the single semantic assumption implicit in the original TF-IDF statistics, and the original weights already have sufficient accuracy in representing the mainstream semantics, no additional correction is needed. Therefore, if the term is determined to belong to the mainstream semantic branch in the current target document, the system maintains its original static base weights without modification.

[0072] Then, if the term is determined to belong to a non-mainstream semantic branch in the current target document, the system applies a boosting factor to its original static base weight. The value of the boosting factor is directly determined by the total number of semantic branches for the term, and its specific value is equal to the total number of semantic branches divided by the number of mainstream branches. In this embodiment, the number of mainstream branches is 1, that is, each term retains only one branch containing the most frequent occurrences as the mainstream branch. For example, if a term is identified as having 3 semantic branches after clustering, the boosting factor for the term belonging to the non-mainstream branch is 3.0, that is, the weight is boosted to three times the original value.

[0073] The statistical mechanism of TF-IDF, when calculating term frequency and inverse document frequency, mixes the frequency information of a term at all occurrence positions into the same statistic. This results in the frequency contribution of the minority non-mainstream semantic branches being severely diluted by the majority mainstream semantic branches. Therefore, the multiplicative compensation effect of the boosting factor is precisely the reverse correction against this dilution effect. Its value is equal to the dilution factor itself, thereby restoring the weight of the non-mainstream semantic branches to their proper level in an independent statistical context.

[0074] Next, the system writes the segmented and adjusted weight values ​​into the weight output buffer. At this point, all terms in all three working modes have obtained their final weight outputs. The system then aggregates all term weight data in the weight output buffer to form the complete term weight vector of the target document.

[0075] When a term has too many semantic branches, some branches may contain only a very small number of occurrences, reducing the statistical reliability of its characteristic co-occurrence word set. The multiplicative compensation of the boosting factor may lead to an extreme case where the weight value is abnormally high. Specifically, if the total number of semantic branches identified after clustering in step S104 exceeds a preset alarm threshold for the number of branches (set to 6 in this embodiment), the system will generate an excessive branch warning log while performing weight adjustment. This warning log provides system maintenance personnel with a basis for investigation, facilitating manual review of the term's clustering results during subsequent corpus updates.

[0076] It should be noted that although the boosting factor in this embodiment is calculated as the ratio of the total number of semantic branches to the number of mainstream branches, in other embodiments, the inverse of the proportion of the frequency of each branch can also be used as the boosting factor, that is, the inverse of the proportion of the frequency of the mainstream branch to the total frequency of the branches can be used as the boosting factor of the non-mainstream branches, so as to achieve a more refined dilution compensation effect.

[0077] S107. The term weights adjusted by the state-driven segmentation strategy are injected into the semantic analysis process of the large language model to replace the static weights and perform the final semantic matching and data processing.

[0078] First, the system reads the complete term weight vector of the target document formed in the previous step. Each component in the vector corresponds to the final weight value of a term after being adjusted by the three-state working mode segmentation strategy.

[0079] Furthermore, the system injects the term weight vectors into the semantic analysis processing flow of the large language model, replacing the original static TF-IDF weights. The specific injection method varies depending on the type of downstream semantic analysis task.

[0080] In the text semantic matching task, the system uses improved term weights to weight and modulate the term-level attention scores output by the encoding layer of the large language model. Specifically, the improved weight value of each term is used as a multiplicative coefficient for its corresponding attention score, and the row or column corresponding to that term in the attention score matrix is ​​scaled element-wise. After scaling, semantically stable terms maintain their original attention influence, the attention influence of terms with slight semantic drift is moderately compressed, and the attention influence of highly semantically divergent terms is differentiated based on their specific semantic belonging in the current document. This ensures that when the same ambiguous term participates in the similarity calculation between different document pairs, it is no longer treated with a single fixed weight value, but rather receives differentiated weight expressions based on its actual semantic contribution in each document.

[0081] In the information extraction task, the system uses the improved term weights as the prior confidence input for the entity recognition stage. For ambiguous terms identified as belonging to the third working mode, the system simultaneously includes the semantic branch attribution identifier of the term in the current document when outputting the entity label. The downstream data structuring module then maps the occurrence of the same term in different contexts to different structured data fields. For example, in the medical text processing scenario of this embodiment, the term "intervention" is assigned to the semantic branch corresponding to percutaneous coronary intervention in the cardiovascular medical record context, and to the semantic branch corresponding to interventional embolization of tumors in the oncology medical record context. The data structuring module fills the former into the "cardiovascular surgery type" field and the latter into the "tumor treatment method" field, avoiding confusion in structured data caused by term ambiguity.

[0082] In document clustering and classification tasks, the system constructs weighted term vector representations of documents using improved term weights. Specifically, for each term in the target document, its improved weight is used as the numerical value for that term's dimension, and these are combined to form a weighted vector of the target document in the global term space. The semantic similarity between two documents is obtained by calculating the cosine distance between their respective weighted vectors. Because ambiguous terms receive differentiated weight representations in different documents, document pairs that were incorrectly grouped closer by static weighting methods due to sharing the same ambiguous term exhibit the proper semantic distance differences under the improved weighting system, thus avoiding being incorrectly clustered into the same category.

[0083] like Figure 2 As shown, this paper compares the comprehensive performance of existing methods and the method of this invention in different downstream tasks. The bar chart clearly compares the processing accuracy of the two methods in three core downstream task scenarios: text semantic matching, information extraction, and document clustering and classification. It can be seen that after injecting term weights adjusted by the state-driven segmentation strategy, the processing accuracy of the method of this invention in the above tasks is significantly higher than that of existing methods, verifying the effectiveness of the hierarchical processing strategy of this invention in downstream data processing.

[0084] Specifically, if during online service, the system detects that a term in the target document is not registered in the terminology index table, meaning it is a new term not present in the training corpus, the system will calculate its static weight according to the standard TF-IDF process and output it directly without performing any three-state weight adjustment operations. The system will then add this new term to the list of terms to be added to the database, for inclusion in the next corpus update and model retraining. This ensures the system's basic compatibility with new open-domain terms and avoids online service interruptions due to limited coverage of the terminology database.

[0085] like Figure 3As shown, this illustrates the impact of unregistered new terms in open-domain text on system robustness. The graph reflects the dynamic trend of online service success rates between existing methods and the method of this invention as the proportion of unregistered new terms increases. Data shows that when the proportion of unregistered new terms increases and enters the open-domain new word emergence range, the online service success rate of existing methods shows a significant downward trend; while the method of this invention, thanks to its dynamic compatibility mechanism for new terms, can still maintain a high online service success rate, significantly improving the system's robustness when facing unknown words in the open domain.

Claims

1. An AI semantic analysis and data processing method based on a large language model, characterized in that, The method includes: performing word segmentation and term extraction on the vertical domain text, constructing a set of context fragments by extracting local context windows with the position of the term as the anchor point, and calculating the static basic weight of each term; The context fragments are fed into the coding layer of the large language model to extract the hidden layer output vectors. Based on the degree of dispersion of all hidden layer output vectors of the same term relative to the centroid vector, the term is divided into three semantic stability levels. Construct a three-state semantic working mode state machine containing a first working mode, a second working mode, and a third working mode. Map terms to corresponding modes according to the semantic stability level. The first working mode keeps the static basic weight unchanged, and the second working mode is subject to a fixed ratio of attenuation. Hidden layer output vector clustering is performed on the terms in the third working mode to identify independent semantic branches, and the iconic co-occurring word set and centroid vector of each branch are extracted. For the terms in the third working mode described in the target document, the semantic branch affiliation is determined through a two-level cascade mechanism of co-occurrence word matching and vector distance determination, and segment weight adjustment is performed based on the branch universality; the adjusted weights are then injected into the semantic analysis process to replace the static basic weights.

2. The AI ​​semantic analysis and data processing method based on a large language model according to claim 1, characterized in that, Capture a local context window using the location where the term appears as the anchor point, including: Expand the term forward and backward by a preset number of words to form a symmetrical context window; When a term appears less than the specified number of expanded terms from the beginning or end of the document, all terms within the available range are truncated and the insufficient portion is filled with preset blank placeholders to maintain a consistent total number of terms across all context windows.

3. The AI ​​semantic analysis and data processing method based on a large language model according to claim 1, characterized in that, The degree of dispersion is characterized by the coefficient of variation, which is calculated as follows: The centroid vector is obtained by calculating the arithmetic mean of all hidden layer output vectors for the same term in each dimension. The Euclidean distance between each hidden layer output vector and the centroid vector is calculated one by one to obtain a set of deviation distance values. The ratio of the standard deviation of the deviation distance values ​​to the arithmetic mean is used as the dispersion coefficient of the term.

4. The AI ​​semantic analysis and data processing method based on a large language model according to claim 3, characterized in that, The three semantic stability levels are divided as follows: The discrete coefficients of all terms in the current corpus are statistically analyzed. The 30th percentile of the discrete coefficient distribution is taken as the first classification boundary, and the 70th percentile is taken as the second classification boundary. Terms with a dispersion coefficient lower than the first classification boundary are classified into the first stability level, terms with a dispersion coefficient higher than the second classification boundary are classified into the third stability level, and terms between the two classification boundaries are classified into the second stability level.

5. The AI ​​semantic analysis and data processing method based on a large language model according to claim 1, characterized in that, The second working mode applies a fixed percentage attenuation to the static base weights through an attenuation factor, which ranges from 0.50 to 0.

85.

6. The AI ​​semantic analysis and data processing method based on a large language model according to claim 1, characterized in that, The hidden layer output vector clustering uses the K-means clustering algorithm, and the number of clusters is adaptively determined by the silhouette coefficient optimality criterion. The specific determination method is as follows: Starting with the number of clusters equal to 2, the number is gradually increased to the preset maximum number of clusters. Clustering is performed for each candidate number of clusters and the corresponding silhouette coefficient is calculated. The number of clusters when the silhouette coefficient reaches its maximum value is selected as the optimal number of semantic branches for the term.

7. The AI ​​semantic analysis and data processing method based on a large language model according to claim 6, characterized in that, The method for extracting the set of iconic co-occurring words is as follows: For each cluster, perform word frequency statistics on the context fragments, calculate the ratio between the frequency of each non-stop word in the target cluster and its frequency in all other clusters, and include words whose ratio exceeds a preset discrimination multiple threshold into the set of iconic co-occurring words of the corresponding semantic branch. At the same time, the semantic branch with the most occurrences is marked as the mainstream semantic branch, and the rest are marked as non-mainstream semantic branches.

8. The AI ​​semantic analysis and data processing method based on a large language model according to claim 1, characterized in that, The two-level cascading mechanism includes two levels of judgment. The first level of judgment is fast matching of co-occurring words, specifically: Extract non-stop words from the local context window where terms appear in the target document to form the current context word set. Perform an intersection operation on the current context word set and the characteristic co-occurrence word set of each semantic branch to count the number of hits. When the difference between the first and second ranked branches in terms of the number of hits exceeds the preset judgment interval threshold, it is determined to belong to the first ranked branch. The second-level decision is activated when the first-level decision does not provide a valid result. It sends the current context fragment of the term to the encoding layer of the large language model to extract the hidden layer output vector, and determines the branch with the smallest cosine distance to the centroid vector of each semantic branch as the semantic affiliation.

9. The AI ​​semantic analysis and data processing method based on a large language model according to claim 1, characterized in that, Perform segment weight adjustments based on branch universality, including: When a term belongs to the mainstream semantic branch in the target document, its static base weight remains unchanged; When a term belongs to a non-mainstream semantic branch, the ratio of the total number of semantic branches of that term to the number of mainstream branches is used as an enhancement factor. The static basic weight is multiplied by the enhancement factor to compensate for the frequency contribution of non-mainstream semantic branches being diluted by mainstream semantic branches during the statistical process.

10. The AI ​​semantic analysis and data processing method based on a large language model according to claim 8, characterized in that, It also includes the following exception handling mechanism: when the cosine distance between the hidden layer output vector and the centroid vectors of all semantic branches in the second-level determination exceeds the preset maximum attribution distance threshold, the term is output with a weight according to the fixed attenuation strategy of the second working mode.

Images