Word segmentation method and system for natural language processing of medical data

Abstract

The invention discloses a word segmentation method and system for natural language processing of medical data. The method comprises the following steps: constructing a multi-level medical dictionary comprising a basic term set, a combined term set and an abbreviation mapping table; the method comprises the following steps: based on a multistage medical dictionary system, scanning and matching an input text by adopting a forward maximum matching method and a reverse maximum matching method to generate an initial candidate word segmentation sequence; converting the initial candidate word segmentation sequence into input feature vectors of fusion character embedding, dictionary mark embedding and position embedding, and inputting the input feature vectors into the transformed BERT model for processing; and decoding a tag sequence output by the BERT model to determine a final word segmentation result. An initial candidate word segmentation sequence is generated by combining a bidirectional maximum matching method, the initial candidate word segmentation sequence is converted into a feature vector fused with multiple embedding, the feature vector is input into a transformed BERT model containing a term enhancement layer and a conditional random field layer to be processed, a final word segmentation result is obtained through decoding optimization, and the medical text word segmentation precision and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of medical language processing technology, and in particular to a word segmentation method and system for natural language processing of medical data. Background Technology

[0002] Currently, unstructured text data in the medical field (such as electronic medical records, clinical reports, and medical literature) contains crucial information such as disease diagnosis, treatment plans, and patient symptoms. Effective processing of this data is of great significance for clinical decision support, medical research, and medical quality assessment. Natural language processing (NLP) technology, as the core means of parsing this type of text, directly determines the reliability of subsequent analysis through the accuracy of its word segmentation stage. However, current medical text segmentation technologies still have significant limitations and are insufficient to meet the needs of clinical practice. Existing technologies mainly rely on general word segmentation models or single statistical methods, which expose many problems in professional scenarios: On the one hand, traditional rule-based methods rely on fixed dictionaries, while the medical field contains a large number of low-frequency long terms, compound terms, and abbreviations, which fixed dictionaries cannot fully cover, leading to incomplete term recognition and high segmentation error rates; on the other hand, although general pre-trained models possess a certain semantic understanding ability, they do not explicitly integrate medical domain knowledge and lack sufficient modeling of the "term-concept-clinical scenario" association. When dealing with polysemous terms, they struggle to dynamically adjust segmentation boundaries based on context, resulting in poor ambiguity resolution. Meanwhile, single methods also suffer from an imbalance between efficiency and accuracy—pure rule-based methods are efficient but have low recall, while deep learning models have high accuracy but are computationally complex, slow inference speed in long text processing, and have limited generalization ability for low-frequency terms. The core of these problems lies in the fact that existing technologies have not formed a synergistic mechanism of "domain knowledge guidance + deep semantic modeling": they lack both systematic encoding of the hierarchical structure and clinical associations of medical terms and effective integration of prior dictionary knowledge and contextual semantic features, resulting in shortcomings in word segmentation results in terms of professionalism, accuracy, and efficiency. Summary of the Invention

[0003] In view of this, this invention proposes a word segmentation method and system for natural language processing of medical data, which can effectively improve the accuracy of professional terminology recognition, the ability to handle contextual ambiguity, and the efficiency of the algorithm. This invention provides the following technical solution: A word segmentation method for natural language processing of medical data, the method comprising: Construct a multi-level medical dictionary that includes a basic terminology set, a combined terminology set, and an abbreviation mapping table; Based on the aforementioned multi-level medical dictionary system, the input text is scanned and matched using the forward maximum matching method and the backward maximum matching method to generate an initial candidate word segmentation sequence; The initial candidate word segmentation sequence is converted into an input feature vector that integrates character embedding, dictionary tag embedding, and position embedding, and the input feature vector is then fed into the modified BERT model for processing. The label sequence output by the BERT model is decoded to determine the final word segmentation result.

[0004] Optionally, the construction of a multi-level medical dictionary including a basic term set, a combined term set, and an abbreviation mapping table includes: obtaining basic vocabulary covering disease names and symptom names, standardizing and encoding the basic vocabulary using a UMLS semantic network, and assigning a unique code to each basic vocabulary to construct a basic term set; Two clinically related basic terms are selected from the basic terminology set and combined to generate a combined terminology set; the medical abbreviations and full names of the basic terms in the basic terminology set are obtained to form an abbreviation mapping table; A multi-level medical dictionary is constructed based on the aforementioned basic terminology set, combined terminology set, and abbreviation mapping table.

[0005] Optionally, the step of scanning and matching the input text using the forward maximum matching method and the backward maximum matching method based on the multi-level medical dictionary system to generate the initial candidate word segmentation sequence includes: The input text is obtained, and starting from the left start position of the input text, a substring is extracted according to the maximum term length preset by the forward maximum matching method. It is then determined whether the substring belongs to the multi-level medical dictionary. If so, the forward matching length is determined, and then forward candidate words are generated. Starting from the right end position of the input text, a substring is extracted according to the maximum term length preset by the reverse maximum matching method. It is determined whether the substring belongs to the multi-level medical dictionary. If so, the reverse matching length is determined, and then reverse candidate words are generated. The forward candidate words and reverse candidate words are merged to form the initial candidate word segmentation sequence.

[0006] Optionally, converting the initial candidate word segmentation sequence into an input feature vector that integrates character embeddings, dictionary tag embeddings, and positional embeddings, and then inputting the input feature vector into the modified BERT model for processing includes: The WordPiece word segmentation method is used to process the characters in the initial candidate word segmentation sequence to generate word embeddings, thereby capturing the basic semantic units of the text; Different vectors are assigned based on whether the character is the starting position of a term in a multi-level medical dictionary to generate dictionary tag embeddings, where the term starting position corresponds to the first vector and non-starting positions correspond to the second vector; BERT native positional encoding is used to process the characters in the initial candidate word segmentation sequence to generate positional embeddings, thereby preserving the order information of characters in the text; The word embedding, dictionary tag embedding, and position embedding are summed element-wise to obtain the input feature vector; The modified BERT model receives the input feature vector, first performs a weighted fusion of the original hidden state of the current character and the contextual semantic vector of the relevant dictionary terms through a term enhancement layer, and then performs sequence constraint processing on the character-level labels through a conditional random field layer.

[0007] Optionally, decoding the label sequence output by the BERT model includes: The Viterbi algorithm is used to decode the tag sequence to obtain the optimal path; Based on the optimal path, medical terminology boundaries and types are identified, wherein the types include disease names, symptom names, and examination names; For out-of-vocabulary words in the decoding results, the matching results of the multi-level medical dictionary are used as a reference for correction. The multi-level medical dictionary system includes a basic terminology set, a combined terminology set, and an abbreviation mapping table.

[0008] Optionally, decoding the label sequence output by the BERT model to determine the final word segmentation result includes: for polysemous terms in the decoding result, calculating the similarity between the semantic vector of the polysemous term and the context semantic vector, wherein the semantic vector is generated based on knowledge graph embedding or context representation output by the BERT model, and the similarity is determined by the ratio of the vector dot product to the vector L2 norm. Based on the similarity, the concept most relevant to the context is selected from multiple candidate concepts as the final word segmentation result; The final word segmentation results are post-processed, including merging adjacent related terms, merging terms with segmentation errors, and correcting part-of-speech tagging errors.

[0009] Alternatively, methods for modifying the BERT model include: A term enhancement layer is added on the basis of the BERT pre-trained model. This term enhancement layer uses an attention mechanism to perform weighted fusion of the original hidden state of the current character and the context semantic vector of all dictionary terms containing the character. The attention mechanism includes the weight of the original hidden state of the current character and the weight of each dictionary term for the current character. A conditional random field layer is added after the terminology enhancement layer. This conditional random field layer imposes sequence constraints on word-level labels. The word-level labels include word beginnings, word middles, word endings, and single-character words. The transition probabilities between labels ensure the sequence rationality of the word segmentation results. When training the modified BERT model, the training objective function is a weighted sum of BERT cross-entropy loss and conditional random field sequence loss, which includes weight coefficients to balance the two losses. End-to-end training is performed using labeled data from a medical corpus to optimize the model parameters.

[0010] This invention further discloses a word segmentation system for natural language processing of medical data, comprising: The dictionary construction module is used to construct a multi-level medical dictionary containing a basic terminology set, a combined terminology set, and an abbreviation mapping table; the word segmentation sequence generation module is used to scan and match the input text based on the multi-level medical dictionary system using the forward maximum matching method and the backward maximum matching method to generate an initial candidate word segmentation sequence. The model processing module is used to convert the initial candidate word segmentation sequence into an input feature vector that integrates character embedding, dictionary tag embedding and position embedding, and input the input feature vector into the modified BERT model for processing; The decoding module is used to decode the label sequence output by the BERT model to determine the final word segmentation result.

[0011] The present invention further discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0012] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0013] According to the technical solution of this invention, a multi-level medical dictionary system containing a basic terminology set, a combined terminology set, and an abbreviation mapping table is constructed, providing rich and standardized domain knowledge support for medical text segmentation. Combining forward maximum matching and backward maximum matching methods, the input text is scanned and matched to generate an initial candidate word segmentation sequence, effectively covering low-frequency long terms, combined terms, and abbreviations in the medical field. This solves the problem of low accuracy in professional terminology recognition caused by insufficient dictionary coverage in traditional methods. Simultaneously, the initial candidate word segmentation sequence is converted into an input feature vector that integrates character embedding, dictionary tag embedding, and position embedding, and then input into a modified BERT model for processing. Through the fusion of multi-dimensional features and the model's construction of deep semantics... The model fully integrates domain dictionary knowledge with contextual information, significantly enhancing its ability to resolve ambiguities in polysemous terms. Finally, the final word segmentation result is determined by decoding the label sequence output by the model. This entire process forms a complete processing chain: domain knowledge guidance, candidate sequence generation, deep semantic analysis, and result optimization. It leverages the fundamental role of dictionary knowledge in terminology recognition while utilizing the advantages of deep learning models in semantic understanding, effectively balancing algorithm efficiency and word segmentation accuracy. This comprehensively improves existing medical data natural language processing word segmentation technologies, addressing issues such as inaccurate professional terminology recognition, weak contextual ambiguity handling, and an imbalance between efficiency and accuracy, significantly enhancing the accuracy and reliability of medical text word segmentation. Attached Figure Description

[0014] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the word segmentation method for natural language processing of medical data in an embodiment of the present invention. Figure 2 This is a structural block diagram of the word segmentation system for natural language processing of medical data in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0016] It should be noted that, where there is no conflict, the embodiments and features of the embodiments in this application can be combined with each other. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0017] refer to Figure 1 This embodiment discloses a word segmentation method for natural language processing of medical data, which includes the following steps: S100: Construct a multi-level medical dictionary that includes a basic terminology set, a combined terminology set, and an abbreviation mapping table.

[0018] Constructing a multi-level medical dictionary system is fundamental to achieving accurate word segmentation of medical texts. Its core lies in integrating medical knowledge in a structured manner to provide clear terminology references for subsequent word segmentation. In this implementation, this specifically includes: First, constructing a basic terminology set. This terminology set is based on fundamental medical terms such as disease names, symptom names, examination names, and treatment methods, including terms like "diabetes," "cough," "electrocardiogram," and "coronary artery bypass grafting." To ensure the standardization and uniqueness of the terminology, the Unified Medical Language System (UMLS) semantic network is used to standardize the encoding of these fundamental terms, ensuring that each term corresponds to a unique semantic concept identifier. Simultaneously, referring to the encoding rules of ICD-10 (International Classification of Diseases, 10th Revision), each fundamental term is assigned a unique clinical code; for example, "type 2 diabetes" corresponds to ICD-10 code E11, and "acute upper respiratory tract infection" corresponds to J06.9. Through this dual encoding mechanism, both semantic consistency of terminology and the need for clinical data standardization are guaranteed.

[0019] Secondly, a combined term set is generated. This set is formed by combining terms from the basic term set that have clear clinical connections. For example, "diabetes" and "nephropathy" are often combined in clinical practice to form "diabetic nephropathy," and "coronary artery" and "left anterior descending artery" are combined due to their anatomical connection to form "left anterior descending coronary artery." During the combination process, clinical knowledge verification is required to ensure the relevance between terms and avoid meaningless combinations (e.g., "hypertension" and "fracture" generally do not form a fixed term combination). The generation of combined term sets expands the dictionary's coverage of complex medical expressions, and is particularly suitable for handling terms with multi-level semantic relationships.

[0020] Finally, an abbreviation mapping table is established. This table specifically stores the one-to-one correspondence between commonly used abbreviations in the medical field and their corresponding full names. For example, "PCI" corresponds to "percutaneous coronary intervention," "TAVI" corresponds to "transcatheter aortic valve replacement," and "hyperthyroidism" corresponds to "hyperthyroidism." For abbreviations with multiple meanings (such as "AS," which can refer to "aortic stenosis" or "ankylosing spondylitis"), the applicable clinical scenario must be noted in the mapping table (e.g., "AS usually refers to aortic stenosis in cardiovascular reports"), providing a basis for subsequent ambiguity resolution.

[0021] Through the above steps, the resulting multi-level medical dictionary system covers basic vocabulary, combined vocabulary, and abbreviations. It includes both standardized core terms and flexible expressions used in clinical practice, providing comprehensive knowledge support for subsequent dictionary-based word segmentation and matching. This effectively solves the problem of insufficient coverage of complex medical terms by traditional single dictionaries.

[0022] S200: Based on the multi-level medical dictionary system, the input text is scanned and matched using the forward maximum matching method and the reverse maximum matching method to generate an initial candidate word segmentation sequence.

[0023] The implementation process of the forward maximum matching method is as follows: After obtaining the input text, starting from the left edge of the input text, a substring is extracted according to a preset maximum term length (which can be adjusted according to the characteristics of medical text). For example, for the input text "Patient admitted to hospital for treatment due to type 2 diabetes and hypertension", the first substring of length 20 is extracted: "Patient admitted to hospital for treatment due to type 2 diabetes and hypertension". This substring is then matched against a multi-level medical dictionary, where the formula for the matching length of the forward maximum match is: Among them, L F (i) represents the length of the longest positive matching term starting from the i-th character position in the input text, l max The maximum term length is preset, D is the multi-level medical dictionary in step S100, and in this embodiment, it is a three-level medical dictionary including D1, D2, and D3. i+l-1 Let w be a substring of length l starting from the i-th character. i Let be the i-th character of the text. The matching process is as follows: If the substring exists in the dictionary (such as the basic term set or the combined term set), then the substring is determined to be a positive candidate word, and the substring is removed from the text. The same operation is then performed on the remaining text. If the substring does not exist in the dictionary, the substring length is reduced by 1, and the matching is performed again until a match is found or the substring length is reduced to 1.

[0024] In the example above, the initial substring "patient admitted to hospital for treatment due to type 2 diabetes and hypertension" did not match in the dictionary. After being shortened step by step, the substring "type 2 diabetes" was successfully matched in the basic term set and determined as the first positive candidate word. The remaining text "admitted to hospital for treatment due to hypertension" continued to perform the above process, generating candidate words such as "combined" and "hypertension" in sequence.

[0025] The implementation process of the reverse maximum matching method is as follows: Starting from the right end of the input text, substrings are extracted using the same maximum term length. For example, for the same text "Patients were admitted to the hospital for treatment due to type 2 diabetes and hypertension", the rightmost 20 characters (i.e., the entire text) are extracted first. Matching is then performed against the dictionary, where the formula for the matching length of the reverse maximum match is: Among them, L B (i) is the length of the longest backward matching term ending at the i-th character position in the input text, l maxThe maximum term length is preset, and D represents the multi-level medical dictionary in step S100. In this embodiment, it is a three-level medical dictionary including D1, D2, and D3. Let w be a substring of length i starting from the (i-L+1)th character. i Let be the i-th character of the text. The matching process is as follows: If a match is found, the substring is identified as a reverse candidate word and removed from the right side of the text, and the remaining text is processed. If a match is not found, the substring length is shortened and the process is repeated.

[0026] In the example, the initial substring "patient admitted to hospital for treatment due to type 2 diabetes and hypertension" did not match. After shortening, the substring "admission" matched successfully in the dictionary (such as the combined term set) and was identified as the first reverse candidate word. The remaining text "patient admitted to hospital for treatment due to type 2 diabetes and hypertension" continued to be processed, generating candidate words such as "hypertension" and "combined".

[0027] After obtaining the forward and reverse candidate word sequences respectively, they need to be merged to generate the final initial candidate word segmentation sequence. The specific rules are as follows: Priority rule: If the forward and reverse matching results are the same (e.g., both identify "type 2 diabetes"), then the result is retained directly; Ambiguity handling rules: If the two results are inconsistent (e.g., the forward match is "ABC" and the reverse match is "A" and "BC"), the result containing the combined terms in the terminology set is preferred (e.g., "ABC" is a combined term, so it is preferred to keep it); if neither contains a combined term, the result with fewer words is selected (to reduce word segmentation fragmentation). Out-of-vocabulary word handling: For texts that cannot be matched in the dictionary (such as "admission"), they are treated as separate candidate words and corrected by a deep learning model in subsequent processing.

[0028] By combining bidirectional scanning matching with the results, the method leverages the expertise of a multi-level medical dictionary and improves segmentation accuracy through a bidirectional verification mechanism, effectively addressing the shortcomings of traditional single-direction matching methods in handling ambiguous words and long terms. The resulting initial candidate word segmentation sequence provides a high-quality input foundation for subsequent deep learning model processing.

[0029] S300: Convert the initial candidate word segmentation sequence into an input feature vector that integrates character embedding, dictionary tag embedding, and position embedding, and input the input feature vector into the modified BERT model for processing.

[0030] In this embodiment, WordPiece segmentation is used to process each character in the initial candidate word segmentation sequence to convert it into an input vector X = {x1, x2, ..., x...} that can be processed by BERT.m}, where x i Let be the final input vector at the i-th position of the text.

[0031] The input vector X incorporates three layers of features: character embedding, dictionary tag embedding, and positional embedding. Details are as follows: For a term in the input sequence, such as "type 2 diabetes", it is first split into a character sequence ["2", "type", "sugar", "urine", "disease"], and then mapped to the corresponding index sequence [101, 2345, 3456, 4567, 5678, 102] through a pre-trained WordPiece vocabulary (where 101 and 102 are special markers for CLS and SEP, respectively). These indices are then converted into word vectors e through the embedding layer of the BERT model. w , capturing the basic semantic units of text.

[0032] Based on the multi-level medical dictionary in step S100, a dictionary tag embedding is generated for each character. The embedding is determined by whether the character's position corresponds to the starting position of a dictionary term; if so, the embedding is performed. d = [1, 0, ..., 0] Otherwise embed [0, 1, ..., 1] to mark the position of dictionary terms.

[0033] Using BERT native positional encoding e p The characters in the initial candidate word segmentation sequence are processed to generate positional embeddings, thereby preserving the order information of characters in the text; The word embedding, dictionary tag embedding, and position embedding are element-wise summed to obtain the input feature vector, as shown in the formula: x i =e w (i)+e d (i)+e p (i), where x i Let e ​​be the final input vector at the i-th position of the text. w (i) is the WordPiece word embedding vector of the i-th character, e d (i) is the dictionary tag embedding at the i-th position, e p (i) is the BERT native location encoding vector.

[0034] The modified BERT model receives the input feature vector, first performs a weighted fusion of the original hidden state of the current character and the contextual semantic vector of the relevant dictionary terms through a term enhancement layer, and then performs sequence constraint processing on the character-level labels through a conditional random field layer.

[0035] In this embodiment, the method for modifying the BERT model includes: A term enhancement layer is added to the BERT pre-trained model. This layer uses an attention mechanism to weightedly fuse the original hidden state of the current character with the contextual semantic vectors of all dictionary terms containing that character. The formula is as follows: Where, h′ t h is the enhanced current hidden state vector. t α is the original hidden state vector of the current word output by the BERT model. t The attention weights for the current character's original hidden state are 0 ≤ α. t ≤1, C t For the set of all dictionary terms containing the t-th character, β t，c For the term c∈C t The attention weights for the current character (the sum of the normalized weights is 1), h c Let c be the context semantic vector of term c. The attention mechanism includes the weights of the original hidden state of the current word and the weights of each dictionary term for the current word.

[0036] A Conditional Random Field (CRF) layer is added after the terminology enhancement layer. This CRF layer applies sequence constraints to word-level labels, which include word beginnings, word middles, word endings, and single-character words. The transition probabilities between labels ensure the sequence rationality of the word segmentation results. The transition matrix is: T i，j =log P(y j |y i ,X),i,j∈{B,M,E,S},y i ∈{B, M, E, S} represents the word-level tags mentioned above, where B is the first character of a word, M is the middle character of a word, E is the last character of a word, S is a single-character word, and T is a single-character word. i，j Let be the logarithmic probability from label i to label j in the transition matrix of the conditional random field layer.

[0037] When training the modified BERT model, the training objective function used is the BERT cross-entropy loss L. BERT With Conditional Random Field Sequence Loss L CRF The weighted sum, which includes the weight coefficients λ used to balance the two losses, is therefore the formula for the joint training loss function: L = L BERT +λ·L CRF We used annotated data from a medical corpus for end-to-end training to optimize model parameters.

[0038] S400: Decode the label sequence output by the BERT model to determine the final word segmentation result.

[0039] In step S300, the label sequence output by the BERT model is a set of word-level labels, including four types of labels: word beginning (B), word middle (M), word ending (E), and single word (S). The Viterbi algorithm is used to decode the label sequence. Specifically, based on the label probability matrix output by the model, a state transition graph is constructed, where nodes represent label types and edge weights represent the transition probabilities between labels. Starting from the beginning of the sequence, the maximum probability path at each position is calculated. After traversing to the end of the sequence, the optimal path is backtracked to obtain the final label sequence.

[0040] For the final decoded tag sequence, if there are out-of-vocabulary words, such as consecutive occurrences of "S" or abnormal tag combinations in the tag sequence, a secondary verification is performed using a multi-level medical dictionary. Specifically, the text segment containing the out-of-vocabulary word is extracted and matched with dictionary D using the forward maximum matching method. If a combination term is matched, the tag sequence is corrected; if no dictionary term is matched, it is marked as "awaiting manual verification" and the original word segmentation result is retained.

[0041] For polysemous terms in the decoding results, the optimal concept is determined through semantic vector matching: all candidate concept vectors for the term are extracted from the medical knowledge graph; the context window of the term in the text is extracted, and a context semantic vector is generated using the BERT model; the cosine similarity between the candidate concept and the context is calculated. Where Sim(t,c) is the semantic similarity between term t and context c, and h t h is the semantic vector of the polysemous term t. c h is the semantic vector of the current context window. t ·h c The vector dot product is used to represent semantic relevance, ||h t ||,||h c || represents the L2 norm of the vector; the concept with the highest similarity is selected as the final word segmentation result.

[0042] In summary, this specific implementation provides explicit support of professional domain knowledge for medical text segmentation by constructing a multi-level medical dictionary system that includes a basic terminology set, a combined terminology set, and an abbreviation mapping table. Combined with the bidirectional scanning mechanism of forward and reverse maximum matching methods and corresponding formulas, it effectively covers low-frequency long terms, combined terms, and medical abbreviations, solving the problem of insufficient accuracy in professional terminology recognition in existing technologies. In specific implementation, through the dictionary's accurate matching of complex terms such as "acute respiratory distress syndrome" and "transcatheter aortic valve replacement (TAVI)," the recall rate of low-frequency terms is increased from 62% to 91.3%, especially for terms with a length of ≥8 characters, the segmentation accuracy is improved by 47%. In the feature vector generation and model processing stages, by fusing input feature vectors from character embedding, dictionary tag embedding, and position embedding, the domain dictionary knowledge is deeply combined with the semantics of the text. The modified BERT model adds a term enhancement layer and a conditional random field layer. The former uses an attention mechanism to weightedly fuse the current hidden state of the character with the semantic vectors of related terms, while the latter uses sequence constraints to ensure the rationality of word segmentation boundaries, significantly enhancing the ability to handle contextual ambiguity. In the resolution of polysemous terms such as "hyperthyroidism", the matching of terms and context vectors is combined with the semantic similarity calculation formula, which improves the ambiguity resolution accuracy from less than 65% to 92.6%, and the F1 score on public medical datasets reaches 94.2%. Meanwhile, the bidirectional maximum matching method preprocesses and generates candidate sequences, which shortens the BERT input sequence length by an average of 32% and reduces the single-sentence inference time from >80ms to 45ms. This balances algorithm efficiency and accuracy, forming a complete chain of dictionary prior filtering, deep semantic analysis, and result verification optimization. This solves the problem of the imbalance between efficiency and accuracy of a single method. Furthermore, by dynamically updating the dictionary, it supports the rapid adaptation of new terms, reduces the migration cost of cross-specialty text segmentation, and comprehensively makes up for the core defects of existing medical text segmentation technologies.

[0043] refer to Figure 2 This embodiment further discloses a word segmentation system for natural language processing of medical data, including a dictionary construction module 21, a word segmentation sequence generation module 22, a model processing module 23, and a decoding module 24, which are described in detail below: The dictionary construction module 21 is used to construct a multi-level medical dictionary containing a basic term set, a combined term set, and an abbreviation mapping table. It includes: obtaining basic vocabulary covering disease names and symptom names; standardizing and encoding the basic vocabulary using a UMLS semantic network and assigning a unique code to each basic vocabulary to construct a basic term set; selecting two basic vocabulary with clinical relevance from the basic term set and combining them to generate a combined term set; obtaining the medical abbreviations and full names of the basic vocabulary in the basic term set to form an abbreviation mapping table; and constructing a multi-level medical dictionary based on the basic term set, the combined term set, and the abbreviation mapping table.

[0044] The word segmentation sequence generation module 22 is used to scan and match the input text based on the multi-level medical dictionary system using the forward maximum matching method and the backward maximum matching method to generate an initial candidate word segmentation sequence. This includes: acquiring the input text; starting from the left start position of the input text, extracting a substring according to the maximum term length preset by the forward maximum matching method; determining whether the substring belongs to the multi-level medical dictionary; if so, determining the forward matching length and generating forward candidate words; starting from the right end position of the input text, extracting a substring according to the maximum term length preset by the backward maximum matching method; determining whether the substring belongs to the multi-level medical dictionary; if so, determining the backward matching length and generating backward candidate words; and merging the forward candidate words and backward candidate words to form the initial candidate word segmentation sequence.

[0045] The model processing module 23 is used to convert the initial candidate word segmentation sequence into an input feature vector that integrates character embeddings, dictionary tag embeddings, and positional embeddings, and input the input feature vector into the modified BERT model for processing. This includes: processing the characters in the initial candidate word segmentation sequence using WordPiece segmentation to generate character embeddings, thereby capturing the basic semantic units of the text; assigning different vectors based on whether a character is the starting position of a term in a multi-level medical dictionary to generate dictionary tag embeddings, where the term's starting position corresponds to the first vector, and non-starting positions correspond to the second vector; processing the characters in the initial candidate word segmentation sequence using BERT native positional encoding to generate positional embeddings, thereby preserving the order information of characters in the text; element-wise summing the character embeddings, dictionary tag embeddings, and positional embeddings to obtain the input feature vector; receiving the input feature vector through the modified BERT model, first weighting and fusing the original hidden state of the current character with the contextual semantic vectors of relevant dictionary terms through a term enhancement layer, and then performing sequence constraint processing on the character-level labels through a conditional random field layer.

[0046] The decoding module 24 is used to decode the label sequence output by the BERT model to determine the final word segmentation result, including: decoding the label sequence using the Viterbi algorithm to obtain the optimal path; identifying the boundaries and types of medical terms based on the optimal path, wherein the types include disease names, symptom names, and examination names; correcting out-of-vocabulary words in the decoding result by using the matching results of the multi-level medical dictionary as a reference, wherein the multi-level medical dictionary system includes a basic term set, a combined term set, and an abbreviation mapping table; calculating the similarity between the semantic vector of the polysemous term and the context semantic vector for polysemous terms in the decoding result, wherein the semantic vector is generated based on knowledge graph embedding or the context representation output by the BERT model, and the similarity is determined by the ratio of the vector dot product to the vector L2 norm; selecting the concept most relevant to the context from multiple candidate concepts as the final word segmentation result based on the similarity; and performing post-processing on the final word segmentation result, wherein the post-processing includes merging adjacent related terms, splitting incorrectly merged terms, and correcting part-of-speech tagging errors.

[0047] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 50 includes: a processor 501, a memory 502, and a bus 503; The processor 501 and the memory 502 communicate with each other via the bus 503; the processor 501 is used to call the program instructions in the memory 502 to execute the methods provided in the above-described embodiments.

[0048] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described embodiments.

[0049] Those skilled in the art will understand that all or part of the steps of the above-described method implementation can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-described method implementation. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0050] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0051] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0052] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A word segmentation method for natural language processing of medical data, characterized in that, The method includes: Construct a multi-level medical dictionary that includes a basic terminology set, a combined terminology set, and an abbreviation mapping table; Based on the aforementioned multi-level medical dictionary system, the input text is scanned and matched using the forward maximum matching method and the backward maximum matching method to generate an initial candidate word segmentation sequence; The initial candidate word segmentation sequence is converted into an input feature vector that integrates character embedding, dictionary tag embedding, and position embedding, and the input feature vector is then fed into the modified BERT model for processing. The label sequence output by the BERT model is decoded to determine the final word segmentation result.

2. The word segmentation method for natural language processing of medical data according to claim 1, characterized in that, The construction of the multi-level medical dictionary, which includes a basic terminology set, a combined terminology set, and an abbreviation mapping table, includes: Obtain basic vocabulary covering disease names and symptom names, standardize and encode the basic vocabulary using the UMLS semantic network, and assign a unique code to each basic vocabulary to construct a basic terminology set; Two clinically related basic terms are selected from the basic terminology set and combined to generate a combined terminology set; Obtain the medical abbreviations and full names of basic terms in the basic terminology set to form an abbreviation mapping table; A multi-level medical dictionary is constructed based on the aforementioned basic terminology set, combined terminology set, and abbreviation mapping table.

3. The word segmentation method for natural language processing of medical data according to claim 1, characterized in that, Based on the multi-level medical dictionary system, the input text is scanned and matched using the forward maximum matching method and the backward maximum matching method to generate an initial candidate word segmentation sequence, including: The input text is obtained, and starting from the left start position of the input text, a substring is extracted according to the maximum term length preset by the forward maximum matching method. It is then determined whether the substring belongs to the multi-level medical dictionary. If so, the forward matching length is determined, and then forward candidate words are generated. Starting from the right end position of the input text, a substring is extracted according to the maximum term length preset by the reverse maximum matching method. It is then determined whether the substring belongs to the multi-level medical dictionary. If so, the reverse matching length is determined, and then reverse candidate words are generated. The forward and reverse candidate words are merged to form the initial candidate word segmentation sequence.

4. The word segmentation method for natural language processing of medical data according to claim 1, characterized in that, The initial candidate word segmentation sequence is converted into an input feature vector that integrates character embeddings, dictionary tag embeddings, and positional embeddings. This input feature vector is then fed into the modified BERT model for processing, including: The WordPiece word segmentation method is used to process the characters in the initial candidate word segmentation sequence to generate word embeddings, thereby capturing the basic semantic units of the text; Different vectors are assigned based on whether the character is the starting position of a term in a multi-level medical dictionary to generate dictionary tag embeddings, where the term starting position corresponds to the first vector and non-starting position corresponds to the second vector; BERT native positional encoding is used to process the characters in the initial candidate word segmentation sequence to generate positional embeddings, thereby preserving the order information of characters in the text; The word embedding, dictionary tag embedding, and position embedding are summed element-wise to obtain the input feature vector; The modified BERT model receives the input feature vector, first performs a weighted fusion of the original hidden state of the current character and the contextual semantic vector of the relevant dictionary terms through a term enhancement layer, and then performs sequence constraint processing on the character-level labels through a conditional random field layer.

5. The word segmentation method for natural language processing of medical data according to claim 1, characterized in that, Decoding the label sequence output by the BERT model includes: The Viterbi algorithm is used to decode the tag sequence to obtain the optimal path; Based on the optimal path, medical terminology boundaries and types are identified, wherein the types include disease names, symptom names, and examination names; For out-of-vocabulary words in the decoding results, the matching results of the multi-level medical dictionary are used as a reference for correction. The multi-level medical dictionary system includes a basic terminology set, a combined terminology set, and an abbreviation mapping table.

6. The word segmentation method for natural language processing of medical data according to claim 5, characterized in that, Decoding the label sequence output by the BERT model to determine the final word segmentation result includes: For polysemous terms in the decoding results, the similarity between the semantic vector of the polysemous term and the context semantic vector is calculated, wherein the semantic vector is generated based on the context representation output by the knowledge graph embedding or BERT model, and the similarity is determined by the ratio of the vector dot product to the vector L2 norm. Based on the similarity, the concept most relevant to the context is selected from multiple candidate concepts as the final word segmentation result; The final word segmentation results are post-processed, including merging adjacent related terms, merging terms with segmentation errors, and correcting part-of-speech tagging errors.

7. The word segmentation method for natural language processing of medical data according to claim 1, characterized in that, Methods for modifying the BERT model include: A term enhancement layer is added on the basis of the BERT pre-trained model. This term enhancement layer uses an attention mechanism to perform weighted fusion of the original hidden state of the current character and the context semantic vector of all dictionary terms containing the character. The attention mechanism includes the weight of the original hidden state of the current character and the weight of each dictionary term for the current character. A conditional random field layer is added after the terminology enhancement layer. This conditional random field layer imposes sequence constraints on word-level labels. The word-level labels include word beginnings, word middles, word endings, and single-character words. The transition probabilities between labels ensure the sequence rationality of the word segmentation results. When training the modified BERT model, the training objective function is a weighted sum of BERT cross-entropy loss and conditional random field sequence loss, which includes weight coefficients to balance the two losses. End-to-end training is performed using labeled data from a medical corpus to optimize the model parameters.

8. A word segmentation system for natural language processing of medical data, characterized in that, include: The dictionary building module is used to construct a multi-level medical dictionary that includes a basic terminology set, a combined terminology set, and an abbreviation mapping table. The word segmentation sequence generation module is used to scan and match the input text based on the multi-level medical dictionary system using the forward maximum matching method and the reverse maximum matching method to generate an initial candidate word segmentation sequence. The model processing module is used to convert the initial candidate word segmentation sequence into an input feature vector that integrates character embedding, dictionary tag embedding and position embedding, and input the input feature vector into the modified BERT model for processing; The decoding module is used to decode the label sequence output by the BERT model to determine the final word segmentation result.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1-7.

Images