A proprietary domain Chinese word segmentation method and system based on transfer learning

Abstract

This invention relates to a domain-specific Chinese word segmentation method and system based on transfer learning. The method addresses the challenges of identifying culture-loaded words in international education texts, the poor domain adaptability of general models, and insufficient semantic consistency optimization. It accurately captures cross-cultural semantic conflict words by constructing a dynamic semantic dictionary, enhances the representation of culture-loaded words by combining an attention mechanism, eliminates domain differences using adversarial training and feature distribution alignment techniques to generate domain-invariant features, optimizes segmentation boundaries by integrating sequence labeling and boundary detection tasks, and finally dynamically adjusts the segmentation path based on semantic similarity scores. This method significantly improves the segmentation accuracy of complex terms in the education domain, solves the pain points of incorrect segmentation of culture-loaded words, decreased domain transfer performance, and lack of semantic coherence, and provides highly robust word segmentation support for international education text processing systems.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, specifically to a proprietary domain Chinese word segmentation method and system based on transfer learning. Background Technology

[0002] In the field of natural language processing, Chinese word segmentation, as a fundamental step in text understanding, plays a crucial supporting role in applications such as machine translation and intelligent teaching systems. Domain-specific Chinese word segmentation methods need to be optimized for the linguistic features of specific application scenarios such as international education, healthcare, and law, in order to address complex linguistic phenomena such as domain-specific terminology and culture-loaded words that are difficult for general-purpose word segmentation models to handle. Especially in cross-cultural education scenarios, texts often contain compound terms that integrate cultural backgrounds, such as "Buddhist-style learning method" and "scenario creation method," whose semantic structures differ significantly from traditional Chinese vocabulary, placing higher demands on the accuracy and adaptability of word segmentation.

[0003] Current Chinese word segmentation technology still faces significant challenges in specialized domain applications: General-purpose segmentation models struggle to effectively identify the semantic boundaries between culturally loaded words and ordinary vocabulary when processing educational texts, often leading to incorrect segmentation of compound terms, such as segmenting "calligraphy introduction method" into "calligraphy / introduction / method". Furthermore, due to the distributional differences between educational texts and general corpora, directly transferring general models can cause domain adaptability issues, resulting in a decrease in terminology recognition rates. In addition, existing methods lack deep optimization mechanisms for the semantic consistency of culturally loaded words and context, making them susceptible to local interference when making segmentation path decisions, affecting the integrity and semantic coherence of key teaching terms. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a proprietary domain Chinese word segmentation method and system based on transfer learning that can accurately identify culturally loaded words, adapt to domain distribution features, and optimize segmentation paths.

[0005] The objective of this invention is achieved through the following solution:

[0006] In a first aspect, the present invention provides a domain-specific Chinese word segmentation method based on transfer learning, comprising the following steps:

[0007] S1: Extract culturally loaded words from the obtained international education corpus and culturally loaded word database, calculate the semantic similarity between candidate words and typical cultural context words, and filter out words with cross-cultural semantic conflicts to generate a dynamic semantic dictionary.

[0008] S2: Perform cultural semantic perception processing on the user-input text sequence and dynamic semantic dictionary, and generate culturally enhanced word vectors by weighting the vector representation of culturally loaded words through an attention mechanism;

[0009] S3: Obtain target domain data of user-output text sequence, process it in combination with pre-trained general Chinese word segmentation model, eliminate domain differences through adversarial training and feature distribution alignment, and generate domain-invariant features;

[0010] S4: Process the culture-enhanced word vectors and domain-invariant features, optimize the word segmentation boundary by fusing sequence labeling and boundary detection tasks, and generate the initial word segmentation sequence;

[0011] S5: Process the initial word segmentation sequence and dynamic semantic dictionary, calculate the semantic similarity score between culturally loaded words and their context, optimize the Chinese segmentation path, and generate Chinese word segmentation results. The Chinese word segmentation results are used to display the segmented text sequence.

[0012] In one embodiment, S1 of the domain-specific Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0013] S11: Process the acquired international education field public corpus and culture-loaded word database, calculate the cultural context weight factor of the terms, combine the TF-IDF algorithm to screen high-frequency cultural terms in the corpus, calculate the term weights through word frequency statistics and inverse document frequency weighting, and generate a candidate vocabulary set;

[0014] S12: Process the candidate vocabulary set and the cultural load word database, calculate the cosine similarity between candidate words and cultural context words based on the cultural semantic projection transformation of the word vector model, and generate a weighted similarity matrix.

[0015] S13: Perform cross-cultural conflict screening on the similarity matrix, select words with similarity higher than the preset dynamic threshold as high-conflict cultural load words, and generate a dynamic semantic dictionary. The dynamic semantic dictionary is used to indicate the recognition boundary of cultural load words.

[0016] In one embodiment, the cultural context weight factor of a domain-specific Chinese word segmentation method based on transfer learning provided by the present invention is calculated using the following formula:

[0017]

[0018] Among them, Weight culture (t) represents the cultural context weighting factor, N culture (t) represents the number of times term t appears in the cultural load word database, N total This represents the total number of terms in the corpus.

[0019] In one embodiment, S2 of the domain-specific Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0020] S21: Process the text sequence input by the user, and map the characters into dense vectors through a domain-adaptive word embedding model to generate a character vector sequence;

[0021] S22: Process the character vector sequence and dynamic semantic dictionary, call the multi-head attention mechanism to calculate the association strength between culture-loaded words and context, and generate a three-dimensional attention distribution;

[0022] S23: Process the three-dimensional attention distribution and character vector sequence, control the enhancement strength of cultural semantic components through learnable gating parameters, and generate culturally enhanced word vectors. These culturally enhanced word vectors are used to indicate the feature fusion input for multi-task joint decoding.

[0023] In one embodiment, S3 of the proprietary domain Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0024] S31: Perform parameter transfer processing on the pre-trained general Chinese word segmentation model, freeze the underlying Transformer encoder to retain general language knowledge, and generate a domain-adaptive base;

[0025] S32: Obtain target domain data of user-output text sequence, perform adversarial training processing in combination with domain adaptive base, and confuse the feature discrimination signals of source domain and target domain through gradient inversion layer and domain classifier to generate domain robust features;

[0026] S33: Perform distribution alignment processing on the domain robust features. Based on the Wasserstein distance, minimize the feature distribution difference between the source and target domains to generate domain invariant features. The domain invariant features are used to indicate the cross-domain general representation in subsequent multi-task decoding.

[0027] In one embodiment, S4 of the proprietary domain Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0028] S41: Process culturally enhanced word vectors and domain-invariant features, align the feature spaces of cultural semantic representation and domain-general representation based on cross-modal attention mechanism, and fuse the two heterogeneous features through cross-attention weights to generate a fused feature tensor;

[0029] S42: The fused feature tensor is processed by a bidirectional gated recurrent unit to capture the long-distance contextual dependencies of the text sequence, and important semantic information is preserved through the gating mechanism to generate time-aware features;

[0030] S43: Perform CRF decoding on the time-aware features, optimize the label transition probability through the Viterbi algorithm, select the globally optimal character position label sequence, and generate the initial word segmentation sequence.

[0031] In one embodiment, S5 of the domain-specific Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0032] S51: Process the initial word segmentation sequence, combine it with the dynamic semantic dictionary to construct a word position index, and generate a word mapping table to be optimized;

[0033] S52: Process the vocabulary mapping table to be optimized, extract the local context vectors of the preset window size before and after each word, aggregate the context semantic information through average pooling operation, and generate context semantic representation;

[0034] S53: Process the contextual semantic representation, measure the word-context relevance through a bilinear transformation matrix, and generate an optimized score vector;

[0035] S54: Perform Viterbi path search on the optimized score vector to find the globally optimal segmentation path and generate Chinese word segmentation results.

[0036] Secondly, this invention provides a domain-specific Chinese word segmentation system based on transfer learning, which is configured with the following modules:

[0037] The cultural load word extraction module is used to extract cultural load words from the acquired international education field public corpus and cultural load word database, calculate the semantic similarity between candidate words and typical cultural context words, filter out words with cross-cultural semantic conflicts, and generate a dynamic semantic dictionary.

[0038] The cultural semantic awareness module is used to process the text sequence input by the user and the dynamic semantic dictionary for cultural semantic awareness. It generates culturally enhanced word vectors by weighting the vector representation of culturally loaded words through an attention mechanism.

[0039] The domain feature alignment module is used to acquire target domain data of the user output text sequence, process it in combination with a pre-trained general Chinese word segmentation model, eliminate domain differences through adversarial training and feature distribution alignment, and generate domain-invariant features.

[0040] The initial word segmentation generation module is used to process culturally enhanced word vectors and domain-invariant features. It optimizes the word segmentation boundary by fusing sequence labeling and boundary detection tasks to generate the initial word segmentation sequence.

[0041] The word segmentation result optimization module is used to process the initial word segmentation sequence and dynamic semantic dictionary, calculate the semantic similarity score between culturally loaded words and their context, optimize the Chinese segmentation path, and generate Chinese word segmentation results. The Chinese word segmentation results are used to display the segmented text sequence.

[0042] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the aforementioned domain-specific Chinese word segmentation methods based on transfer learning.

[0043] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the aforementioned domain-specific Chinese word segmentation methods based on transfer learning.

[0044] In summary, the proprietary domain-specific Chinese word segmentation method provided in this application accurately identifies cross-cultural semantic conflict words by constructing a dynamic semantic dictionary, which can solve the segmentation error problem caused by cultural background differences in compound terms in the education field. By using attention mechanism to weight cultural load word vector representation, the semantic representation of key terms can be enhanced, effectively improving the recognition ability of traditional models for cultural load words such as "Buddhist learning method". Based on adversarial training and feature distribution alignment technology to eliminate domain differences, it can achieve stable transfer of general models to the education field, thereby overcoming the performance degradation caused by the mismatch between the source and target domain distributions. The multi-task joint decoding mechanism that integrates sequence labeling and boundary detection can achieve collaborative optimization of word segmentation boundaries, significantly improving the segmentation integrity of compound terms such as "calligraphy import method". By dynamically optimizing the segmentation path through semantic similarity scores, it can ensure the consistency between cultural load words and contextual semantics, thereby solving the segmentation coherence problem of traditional methods in long-distance dependency scenarios.

[0045] The method provided in this application systematically addresses three core pain points in text processing in the field of international education: inaccurate identification of culture-loaded words, poor adaptability to domain transfer, and lack of semantic coherence. It provides highly robust Chinese word segmentation support for teaching assistance systems and significantly improves the quality of text processing in scenarios such as international student essay scoring and textbook analysis.

[0046] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a domain-specific Chinese word segmentation method based on transfer learning, provided for an embodiment of this application;

[0048] Figure 2 A schematic diagram illustrating the process of generating a dynamic semantic dictionary provided in an embodiment of this application;

[0049] Figure 3 A flowchart illustrating the process of generating Chinese word segmentation results provided in this application embodiment;

[0050] Figure 4This is a schematic diagram of the structure of a proprietary domain Chinese word segmentation system based on transfer learning, provided for another embodiment of this application. Detailed Implementation

[0051] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0053] In one embodiment, such as Figure 1 As shown, a domain-specific Chinese word segmentation method based on transfer learning is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] S1: Extract culturally loaded words from the obtained international education corpus and culturally loaded word database, calculate the semantic similarity between candidate words and typical cultural context words, and filter out words with cross-cultural semantic conflicts to generate a dynamic semantic dictionary.

[0055] Specifically, the publicly available corpora in the field of international education obtained by the system are as follows: First, international curriculum textbooks, including Chinese textbooks for IB, AP, and A-Level courses, as well as Chinese translations of the curriculum outlines published by the International Baccalaureate Organization. These texts are obtained through the official websites of international education publishers and open education resource platforms such as OpenStax. The PDF and EPUB formats are converted to plain text using format conversion tools, while retaining chapter structure markers. Among them, the IB curriculum includes science, language, and social science textbooks from primary to high school, and the AP curriculum includes Chinese reference materials for calculus and psychology. Second, cross-cultural teaching research literature, taken from CNKI's "International Education" special issue and Web of Science's "Cross-Cultural Education" themed papers, including Chinese abstracts, teaching case analyses, and interview transcripts from empirical studies. The system obtains these in batches through the literature database API, filtering for literature published within the last ten years to ensure timeliness. Thirdly, there are interactive texts from education forums, covering discussion posts and replies from international school parent forums and teacher exchange communities. These were collected using web crawlers, limiting the texts to those containing keywords such as "teaching methods," "cultural adaptation," and "curriculum design," and removing advertisements and irrelevant chatter.

[0056] Preferably, the culture-loaded term database obtained in this application is pre-constructed. For example, the construction of the culture-loaded term database includes two stages: basic vocabulary selection and manual enhancement. The system extracts core vocabulary from the *Dictionary of Intercultural Communication* and the *International Handbook of Educational Terminology*, covering educational terms such as "immersive teaching," "credit recognition," and "multi-dimensional assessment," as well as descriptive terms reflecting cultural differences such as "collective learning" and "critical thinking." Subsequently, researchers with intercultural education backgrounds expand the initial vocabulary set, supplementing it with emerging terms such as "flipped classroom localization" and "intercultural group collaboration," and associating each term with a cultural context tag. The tagging system is divided into three levels: first-level tags represent cultural dimensions, such as "power distance" and "individualism-collectivism"; second-level tags represent educational scenarios, such as "classroom interaction" and "performance evaluation"; and third-level tags represent semantic orientation, such as "positive adaptation" and "cultural conflict." The system stores the vocabulary and tags as an associated data table, supporting tag-based retrieval and batch updates.

[0057] For example, after acquiring a publicly available corpus of international education and a culture-loaded word database, the system performs culture-loaded word extraction. The publicly available corpus of international education is cleaned by using regular expressions to remove noise data such as HTML tags and special symbols, and an N-gram sliding window is used to extract candidate word sequences, filtering out single-character candidates with a length less than 2. For the culture-loaded word database, the system can construct it through a dual-channel approach: a basic database integrating education-related entries from the *Dictionary of Intercultural Communication*, and an extended database collecting frequently debated words from international education forums to form an initial candidate set.

[0058] Preferably, the system calculates the semantic similarity between candidate words and typical cultural context words, calls a pre-trained language model to generate context-related vectors for the words, and the typical cultural context word set includes benchmark terms such as "Eastern educational philosophy" and "Western teaching model." The system calculates the cosine value of the angle between the vectors of candidate words and benchmark words using the cosine similarity formula. Based on the calculation results, the system filters out words with cross-cultural semantic conflicts. When the similarity difference between a candidate word and the benchmark words from both East and West reaches a set standard, it is determined to have a cross-cultural semantic conflict. Through the above processing, the system generates a dynamic semantic dictionary, which includes word strings, semantic conflict markers, cross-cultural tendency labels, and context-fit probability distributions. Preferably, this application periodically recalculates similarity using incrementally crawled corpora and employs a database expiration key strategy to eliminate low-frequency words.

[0059] S2: Perform cultural semantic perception processing on the user-input text sequence and dynamic semantic dictionary, and generate culturally enhanced word vectors by weighting the vector representation of culturally loaded words through an attention mechanism.

[0060] Specifically, the system receives a text sequence input by the user and simultaneously invokes a dynamic semantic dictionary to perform word segmentation preprocessing on the text sequence, achieving initial segmentation using basic word segmentation tools. For example, the system matches the dynamic semantic dictionary using a prefix tree structure to mark the positions of culture-loaded words in the text sequence; for instance, in text containing "tea ceremony experience class," the position of "tea ceremony" is marked. For words not matched in the dynamic semantic dictionary, the system initializes with pre-trained word vectors; for matched words, the system loads multicultural semantic vectors stored in the dictionary and constructs a culture-aware attention layer, calculating the attention weight of each word in the text sequence with the culture-loaded words in the dynamic semantic dictionary, based on the cosine similarity between word vectors. The system dynamically weights the vector representations of culture-loaded words according to the strength of the conflict markers, while non-culture-loaded words retain their original weights. A linear transformation is then used to map the weighted vectors to a specific dimensional space, generating culture-enhanced word vectors to enhance the recognizability of culture-loaded words in the semantic space.

[0061] S3: Obtain target domain data of the user-output text sequence, process it in conjunction with a pre-trained general Chinese word segmentation model, eliminate domain differences through adversarial training and feature distribution alignment, and generate domain-invariant features.

[0062] Specifically, the system acquires target domain data from user-output text sequences. This target domain data includes international curriculum standards and multinational teaching case studies. The system can use OCR technology to process scanned documents and PyPDF2 to parse electronic text. The system preprocesses the acquired target domain data, employing synonym replacement and random insertion of culturally loaded words for data augmentation. The synonym replacement is based on the WordNet Chinese thesaurus.

[0063] For example, the system combines a pre-trained general Chinese word segmentation model for processing. The pre-trained general Chinese word segmentation model uses a specified model, freezes some layer parameters after loading pre-trained weights, and unfreezes the remaining layers and classification head. The system performs initial training on the model using the Adam optimizer, and warms up the training on a general word segmentation dataset. The system eliminates domain differences through adversarial training and feature distribution alignment, introducing a domain adversarial neural network, which consists of a feature extractor, a label predictor, and a domain discriminator. The system designs an adversarial loss function, achieving adversarial training by calculating the expected values ​​of different domain data distributions. Feature distribution alignment can use the maximum mean difference to calculate the distribution distance between two domain features, selecting a Gaussian kernel function as the regeneration kernel Hilbert space mapping. The system generates domain-invariant features, using the output of the last layer of the feature extractor as the domain-invariant feature, and performs standardization through L2 regularization. Preferably, the system can use a specified framework to implement the relevant processes, periodically recording feature distribution differences during training, and stopping adversarial training when the distribution difference reaches a set condition.

[0064] S4: Process the culture-enhanced word vectors and domain-invariant features, optimize the word segmentation boundary by fusing sequence labeling and boundary detection tasks, and generate the initial word segmentation sequence.

[0065] Specifically, the system performs a fusion process on the culture-enhanced word vectors and domain-invariant features. The fused features are equal to the sum of the results after linear transformation of the culture-enhanced word vectors and domain-invariant features. The linear transformation is completed through a convolutional kernel matrix of a specific size, and the fused features are input into a dual-channel task structure. One channel executes a sequence labeling task, and a conditional random field layer is used to predict BEMS tags. For example, for "tea ceremony", the label B-E is marked. The loss function of this task is the negative log-likelihood loss. The other channel executes a boundary detection task, and the probability of word boundaries is recognized through a multi-layer convolutional neural network. For example, it is judged whether there is a boundary after "ceremony". The loss function of this task is the binary cross-entropy loss. Preferably, the system can combine the loss functions of the two tasks in a weighted summation manner and perform iterative optimization through an Adam optimizer. When the joint loss function reaches a convergence state, that is, the decrease amplitude in consecutive multiple rounds is less than the set threshold, the system outputs the initial word segmentation sequence, and the boundary detection task improves the boundary recognition effect of compound terms.

[0066] S5: Process the initial word segmentation sequence and the dynamic semantic dictionary, calculate the semantic similarity score between the culture-loaded words and their contexts, and optimize the Chinese word segmentation path to generate the Chinese word segmentation result, which is used to display the segmented text sequence.

[0067] Specifically, the system processes the initial word segmentation sequence and identifies the culture-loaded words therein, such as "Buddha-style learning method". The system extracts several words before and after the culture-loaded word to form a context window, and calculates the average semantic similarity score between the culture-loaded word and the words in the context window through the BERT (Bidirectional Encoder Representations from Transformers) model. If the score is lower than the set threshold, the system triggers the word segmentation path optimization process. The system calls the cross-cultural semantic vector of the culture-loaded word in the dynamic semantic dictionary and recalculates its matching degree with the context. If the recalculated matching degree is significantly improved compared with before, the system adjusts the corresponding segmentation boundary. For example, it adjusts "Buddha / System learning / Method" to "Buddha-style learning / Method". The system standardizes the format of the optimized word segmentation sequence, uses " / " as the separator, and stores the result in JSON format, which contains information such as the original text, the word segmentation result, and the culture-loaded word marker. The Chinese word segmentation result generated by the system supports direct calling to the display structure for presenting the segmented text sequence.

[0068] In summary, the proprietary domain-specific Chinese word segmentation method provided in this application accurately identifies cross-cultural semantic conflict words by constructing a dynamic semantic dictionary, which can solve the segmentation error problem caused by cultural background differences in compound terms in the education field. By using attention mechanism to weight cultural load word vector representation, the semantic representation of key terms can be enhanced, effectively improving the recognition ability of traditional models for cultural load words such as "Buddhist learning method". Based on adversarial training and feature distribution alignment technology to eliminate domain differences, it can achieve stable transfer of general models to the education field, thereby overcoming the performance degradation caused by the mismatch between the source and target domain distributions. The multi-task joint decoding mechanism that integrates sequence labeling and boundary detection can achieve collaborative optimization of word segmentation boundaries, significantly improving the segmentation integrity of compound terms such as "calligraphy import method". By dynamically optimizing the segmentation path through semantic similarity scores, it can ensure the consistency between cultural load words and contextual semantics, thereby solving the segmentation coherence problem of traditional methods in long-distance dependency scenarios.

[0069] The method provided in this application systematically addresses three core pain points in text processing in the field of international education: inaccurate identification of culture-loaded words, poor adaptability to domain transfer, and lack of semantic coherence. It provides highly robust Chinese word segmentation support for teaching assistance systems and significantly improves the quality of text processing in scenarios such as international student essay scoring and textbook analysis.

[0070] For example, when teachers or students input texts such as international student essays and textbook excerpts from the international education field into the system, a dynamic semantic dictionary is first automatically constructed: the system scans culture-loaded words in the corpus, calculates their semantic similarity with cultural context words, and filters out cross-cultural conflict words to form a dedicated dictionary. Then, the system processes the user-input text sequence, dynamically weighting the vector representations of culture-loaded words through an attention mechanism to generate feature vectors that integrate cultural semantics. Next, the system calls a pre-trained word segmentation model, combining adversarial training and feature alignment techniques to eliminate distribution differences between the general domain and the education domain, outputting domain-robust semantic features. Subsequently, cultural features and domain features are integrated, and sequence labeling and boundary detection are used to collaboratively optimize the word segmentation boundaries. Finally, the initial word segmentation results are semantically consistent, the positions of culture-loaded words are located, their contextual semantic information is aggregated, and a path optimization algorithm is used to generate a globally optimal segmentation scheme. The final segmented text obtained by the user is displayed intuitively in a delimited form, such as "he / use / calligraphy import method / explain / Chinese characters," which can be directly used in teaching scenarios such as essay scoring and terminology extraction, significantly improving the accuracy and efficiency of cross-cultural education text processing.

[0071] In one embodiment, such as Figure 2 As shown, S1 of the domain-specific Chinese word segmentation method based on transfer learning provided by this invention specifically includes the following steps:

[0072] S11: Process the acquired international education field public corpus and cultural load word database, calculate the cultural context weight factor of the terms, combine the TF-IDF algorithm to screen high-frequency cultural terms in the corpus, calculate the term weights through word frequency statistics and inverse document frequency weighting, and generate a candidate vocabulary set.

[0073] Specifically, after acquiring a publicly available corpus of international education and a database of culture-loaded terms, the system iterates through all entries in the culture-loaded term database, extracting cultural context tags for each term. These tags include cultural background identifiers, educational scenario classifications, and semantic information, and are compiled into a set of cultural context tags. The system performs a full scan of the publicly available corpus of international education, first using a word segmentation tool to preliminarily segment the corpus, removing stop words, and then counting the total number of all independent terms in the corpus, denoted as N. total Simultaneously, the system matches terms in the culture-loaded term database line by line, counting the number of times each term t appears in the database, denoted as N. culture (t). Preferably, the context weighting factor is calculated using the following formula:

[0074]

[0075] Among them, Weight culture (t) represents the cultural context weighting factor, N culture (t) represents the number of times term t appears in the cultural load word database, N total This represents the total number of terms in the corpus. During the calculation, the system calculates N... culture (t) and N total A division operation is performed, and the result is incremented by 1 before undergoing a natural logarithmic transformation to avoid the influence of zero values ​​and compress the weight range. Preferably, the system calls the TF-IDF algorithm to process the publicly available corpus in the field of international education. The system divides the corpus into independent units according to the chapter structure of the documents, counts the occurrence frequency of each term in a single document unit, and obtains the term frequency TF(t,d), where t is the term and d is the document unit. The system counts the number of document units containing term t, denoted as df(t), takes the logarithm of the ratio of the total number of document units D in the corpus to df(t), obtains the inverse document frequency IDF(t), and calculates the comprehensive weight W(t) for each term, using the following formula:

[0076]

[0077] W(t)=TF(t,d)×IDF(t)×Weight culture (t)

[0078] Where W(t) is the overall weight. The system sorts the terms from high to low according to W(t) and selects the top-ranked terms to form a candidate vocabulary set.

[0079] S12: Process the candidate vocabulary set and the cultural load word database, calculate the cosine similarity between candidate words and cultural context words based on the cultural semantic projection transformation of the word vector model, and generate a weighted similarity matrix.

[0080] Specifically, the system calls upon the candidate vocabulary set and the culture-loaded word database to perform word vector transformation on the words in both types of data. It then loads a pre-trained word vector model trained on a large-scale corpus containing Chinese-English bilingual educational texts, cross-cultural communication cases, and international education policy documents, with a fixed output dimension. Based on the label association rules in the culture-loaded word database, the system constructs a projection matrix and performs a cultural semantic projection transformation on the word vector space. The transformation formula is as follows: in, M represents the original word vectors, and M is the cultural semantic projection matrix. The transformed word vectors are those terms with similar cultural attributes that are brought closer together in the vector space.

[0081] For example, the system generates a word vector for each candidate word in the candidate vocabulary set. Simultaneously, word vectors for all cultural context words in the cultural load word database are generated. Preferably, the system can use the cosine similarity formula to calculate the semantic relevance between each candidate word and each cultural context word, as follows:

[0082]

[0083] in, For vector dot product, and The initial similarity value is obtained by taking the magnitudes of the vectors. The system call generates a comprehensive weight W(t) for the candidate words. i The weighted similarity value is calculated using the following formula:

[0084]

[0085] Among them, t i Let be the i-th term in the candidate vocabulary set. The system uses candidate words as row indices and cultural context words as column indices, and fills the corresponding positions in the matrix with weighted similarity values ​​to construct a weighted similarity matrix. Where m is the number of candidate words and n is the number of cultural context words.

[0086] S13: Perform cross-cultural conflict screening on the similarity matrix, select words with similarity higher than the preset dynamic threshold as high-conflict cultural load words, and generate a dynamic semantic dictionary. The dynamic semantic dictionary is used to indicate the recognition boundary of cultural load words.

[0087] Specifically, the system parses the weighted similarity matrix, extracting the maximum similarity value corresponding to each candidate word row by row, denoted as Sim. max (t i This value represents the candidate word t. i The system calculates the weighted semantic correlation between the most relevant cultural context words and the terms in the corpus. It analyzes the distribution characteristics of terms in an open corpus of international education, calculates the average similarity value μ and standard deviation σ of all terms in the corpus, and sets an initial preset dynamic threshold using the formula Th = μ + k × σ, where k is an adjustment coefficient. When a new document is added to the corpus, the system calculates the difference between the average similarity value of culturally loaded words in the new document and the original average value, obtaining the change magnitude Δ. The preset dynamic threshold is adaptively adjusted using the formula Th' = Th × (1 + Δ) to ensure that the threshold reflects the current cultural semantic distribution of the corpus.

[0088] The system will use the Sim for each candidate word max (t i The result is compared with the preset dynamic threshold Th', and Sim is selected. max (t i Candidate words with a value greater than Th′ are identified as high-conflict culture-loaded words. The system assigns a unique lexical ID to each high-conflict culture-loaded word, associates it with its multicultural semantic vector in the word vector model, and adds a conflict marker field. This field's value describes the semantic differences of the word in different cultural contexts. The system then sorts this information in ascending order by lexical ID and stores it as a structured data table, generating a dynamic semantic dictionary. This dictionary is used in subsequent processing to quickly look up the identification boundary parameters of culture-loaded words by lexical ID.

[0089] In one embodiment, S2 of the domain-specific Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0090] S21: Process the text sequence input by the user, and map the characters into dense vectors through a domain-adaptive word embedding model to generate a character vector sequence.

[0091] Specifically, the system preprocesses the user-input text sequence by splitting it at the character level, removing invisible characters, erroneous characters, and other invalid characters, while retaining Chinese characters, numbers, letters, and necessary punctuation, forming a continuous character stream. The system then performs length validation on the character stream; if it exceeds a preset maximum length, it is truncated according to sentence boundaries to ensure the integrity of the processed sequence.

[0092] The system invokes a domain-adaptive word embedding model, which is fine-tuned from a general word embedding model using domain data, incorporating terminology features from the international education domain during the fine-tuning process. The system takes each character in the processed text sequence as input and feeds it into the domain-adaptive word embedding model. The model maps each character to a fixed-dimensional dense vector by looking up a pre-built character embedding table. For characters not appearing in the embedding table, the model generates corresponding dense vectors using random initialization and updates them during subsequent training.

[0093] The system arranges the dense vectors corresponding to all characters in order to form a character vector sequence. The dimension of the character vector sequence is the product of the text sequence length and the dimension of the dense vectors. The system standardizes the generated character vector sequence by subtracting the mean and dividing by the standard deviation to ensure that the vector values ​​are within a uniform distribution range, providing stable input for subsequent processing steps.

[0094] S22: Process the character vector sequence and dynamic semantic dictionary, call the multi-head attention mechanism to calculate the association strength between culture-loaded words and context, and generate a three-dimensional attention distribution.

[0095] Specifically, after acquiring the character vector sequence and the dynamic semantic dictionary, the system extracts all high-conflict culture-loaded words from the dynamic semantic dictionary, constructs a list of culture-loaded words, and generates a corresponding index for each culture-loaded word. Simultaneously, the system segments the character vector sequence into multiple subsequences of fixed length, each subsequence containing several consecutive character vectors, facilitating parallel computation.

[0096] Preferably, the system invokes a multi-head attention mechanism, which includes multiple parallel attention heads, each independently calculating the association strength. For each subsequence, the system matches the character vectors in the subsequence with vectors in the list of culturally loaded words to determine the positions of the culturally loaded words in the subsequence. Each attention head calculates the association strength between the culturally loaded word and each character vector in the subsequence. The calculation process is based on vector dot products, and the results are adjusted by a scaling factor and then passed through a softmax function to obtain a normalized association strength value. The system concatenates the calculation results of all the attention heads to form a three-dimensional attention distribution, where the first dimension corresponds to the subsequence number, the second dimension corresponds to the character position in the subsequence, and the third dimension corresponds to the attention head number. Each element represents the association strength of the corresponding position under the corresponding attention head.

[0097] S23: Process the three-dimensional attention distribution and character vector sequence, control the enhancement strength of cultural semantic components through learnable gating parameters, and generate culturally enhanced word vectors. These culturally enhanced word vectors are used to indicate the feature fusion input for multi-task joint decoding.

[0098] Specifically, after acquiring the 3D attention distribution and the character vector sequence, the system performs dimensional alignment on both. The system matches the association strength corresponding to each subsequence in the 3D attention distribution with the character vector of the corresponding subsequence in the character vector sequence to ensure accurate positional correspondence. For parts with mismatched lengths, the system uses interpolation to adjust them, ensuring that the association strength and the character vector are consistent in length.

[0099] Preferably, the system introduces a learnable gating parameter, which is learned through a training process and initialized with a randomly generated decimal. The learnable gating parameter is multiplied by the association strength in the three-dimensional attention distribution to obtain the cultural semantic enhancement coefficient for each character vector. The magnitude of the enhancement coefficient reflects the strength of the cultural semantic component enhancement; the larger the coefficient, the higher the proportion of cultural semantic components in the corresponding character vector.

[0100] The system multiplies the enhancement coefficients by the corresponding character vectors in the character vector sequence to obtain the enhanced character vectors. Then, the system arranges all the enhanced character vectors sequentially to form a preliminary sequence of culturally enhanced word vectors. The system performs feature fusion on this sequence, adjusting the vector dimensions to a preset value through linear transformation and applying an activation function for non-linear transformation to enhance the vectors' expressive power. Finally, it generates culturally enhanced word vectors, which are used to indicate the feature fusion input for multi-task joint decoding.

[0101] In one embodiment, S3 of the proprietary domain Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0102] S31: Perform parameter transfer processing on the pre-trained general Chinese word segmentation model, freeze the underlying Transformer encoder to retain general language knowledge, and generate a domain-adaptive base.

[0103] Specifically, the system acquires a pre-trained general-purpose Chinese word segmentation model. This model is trained on a general Chinese corpus covering news, encyclopedias, and novels, and includes a multi-layer Transformer encoder, a word embedding layer, and an output layer. The system analyzes the parameters of each layer of the model, identifying the boundary between the bottom-level Transformer encoder and the upper task layers. The bottom-level Transformer encoder is responsible for extracting general language features, including Chinese character structure features, basic grammatical features, and general semantic association features; the upper task layers optimize parameters for the general word segmentation task.

[0104] For example, the system loads all parameters of the pre-trained model into the local computing framework and freezes the weights of the underlying Transformer encoder through a parameter locking mechanism, retaining only the updatable parameters of the word embedding layer and the upper task layer. The freezing process is achieved by setting a gradient calculation switch for the parameters, preventing the underlying Transformer encoder from receiving gradient updates from backpropagation during subsequent training. The system performs initial adaptation of the model based on a small sample corpus from the international education domain, adjusting the word embedding layer parameters to meet the representation requirements of domain-specific characters.

[0105] During the adaptation process, the system inputs the domain corpus into the model, generates feature representations through forward computation, and then updates the word embedding layer parameters through backpropagation, enabling the vector representations of domain-specific characters to form clusters in the semantic space. After completing the above processing, the system generates a domain-adaptive base, which includes a frozen low-level encoder, an updatable top-level encoder, and an initial classifier. This base retains general language knowledge and has preliminary domain feature adaptation capabilities.

[0106] S32: Obtain target domain data of user-output text sequence, perform adversarial training processing in combination with domain adaptive base, and confuse the feature discrimination signals of source domain and target domain through gradient inversion layer and domain classifier to generate domain robust features.

[0107] Specifically, the system acquires target domain data from user-output text sequences. This target domain data comprises various texts within the international education field, including teaching plans, course descriptions, and cross-cultural teaching cases. The system preprocesses the target domain data by removing duplicate text through text comparison, removing noise data containing garbled characters and meaningless symbols through regular expression matching, and performing initial segmentation using the same word segmentation standard as general corpora, thus forming a target domain corpus composed of word sequences.

[0108] For example, the system combines a target domain corpus with a domain adaptive base for adversarial training, constructing an adversarial training framework. In this framework, the feature extraction part uses the encoder in the domain adaptive base to convert the input text into feature vectors. The domain classifier uses a multilayer perceptron structure, receiving the feature vectors as input and outputting the probability of the corresponding feature originating from either the source or target domain, used to determine the domain origin of the feature. A gradient reversal layer is inserted between the feature extraction part and the domain classifier. During forward propagation, the gradient reversal layer does not modify the feature vectors, keeping the features unchanged; during backward propagation, the gradient reversal layer multiplies the incoming gradient by a negative number, reversing the gradient direction.

[0109] During training, the system simultaneously optimizes the parameters of both the feature extraction component and the domain classifier. The feature extraction component obtains an optimization signal opposite to that of the domain classifier through a gradient inversion layer. Its optimization goal is to make it difficult for the domain classifier to distinguish the source of features, while the domain classifier tries its best to distinguish accurately through normal gradient updates, creating an adversarial relationship. Through multiple rounds of iterative training, the system continuously monitors the discrimination accuracy of the domain classifier. When the accuracy approaches a random level, training stops. At this point, the features generated by the feature extraction component have weakened the domain differences, and the system defines these features as domain-robust features. The system performs batch feature extraction, traversing all texts in the target domain corpus, converting them into corresponding domain-robust features through the feature extraction component, and storing them in the form of a feature matrix. The rows of the matrix correspond to the text index, and the columns correspond to the various dimensions of the features.

[0110] S33: Perform distribution alignment processing on the domain robust features. Based on the Wasserstein distance, minimize the feature distribution difference between the source and target domains to generate domain invariant features. The domain invariant features are used to indicate the cross-domain general representation in subsequent multi-task decoding.

[0111] Specifically, the system acquires domain-robust features and source domain features extracted from general corpora using a domain-adaptive base. It performs a dimensionality check on both types of features, ensuring complete consistency by comparing the dimensionality values ​​of the feature vectors. If discrepancies exist, the system adjusts the feature dimensions using a linear transformation, mapping the features to the same dimensional space through matrix multiplication. The system then divides the two types of features into training and validation sets, based on the data distribution characteristics, ensuring that the samples in the training and validation sets are evenly distributed in terms of vocabulary type and grammatical structure.

[0112] For example, the system introduces Wasserstein distance as a quantitative indicator of distribution difference, calculating the distance between the feature distributions of the source and target domains. Wasserstein distance, measured by optimal transport theory, measures the minimum transport cost between two distributions, capturing the overall difference in distribution patterns. The system incorporates this distance as a loss term into the training objective, co-optimizing it with the original task loss. The system updates the parameters of the domain adaptive base through backpropagation. During optimization, the model simultaneously reduces both the task loss and the Wasserstein distance loss. The task loss ensures that features retain the semantic and boundary information required for word segmentation, while the Wasserstein distance loss drives the feature distributions of the source and target domains towards a consistent direction. After multiple iterations, the difference in feature distributions between the two domains gradually narrows, generating domain-invariant features. These features exhibit a consistent distribution across different domain data, serving as a cross-domain universal representation in subsequent multi-task decoding.

[0113] In one embodiment, S4 of the proprietary domain Chinese word segmentation method based on transfer learning provided by the present invention specifically includes the following steps:

[0114] S41: Process culturally enhanced word vectors and domain-invariant features, align the feature spaces of cultural semantic representation and domain-general representation based on cross-modal attention mechanism, and fuse the two heterogeneous features through cross-attention weights to generate a fused feature tensor.

[0115] Specifically, after acquiring the culture-enhanced word vectors and domain-invariant features, the system extracts the dimensional information of the two types of features and determines whether there are differences by comparing the dimensional values. If differences exist, the system can call a linear transformation function to process the domain-invariant features. This function performs operations based on a randomly initialized matrix, and the elements in the matrix are adjusted through backpropagation during subsequent model training. During the transformation, the system multiplies each element of the domain-invariant feature with the corresponding element in the matrix, and then accumulates the results to obtain the transformed feature value, ensuring that the transformed domain-invariant features and the culture-enhanced word vectors match in dimensionality.

[0116] Preferably, the system introduces a cross-modal attention mechanism, decomposing the culture-enhanced word vectors into several sub-vectors as the query vector set; and decomposing the domain-invariant features into the same number of sub-vectors, serving as the key vector set and value vector set, respectively. The system performs a dot product operation on the query vector set and the key vector set to obtain initial association values, then adjusts the initial association values ​​through a scaling operation, and subsequently inputs the adjustment result into a softmax function to generate cross-attention weights. The system multiplies the cross-attention weights element-wise with the value vector set to obtain weighted domain-invariant feature sub-vectors, and combines these sub-vectors sequentially to form complete weighted domain-invariant features. Subsequently, the system adds the weighted domain-invariant features element-wise with the culture-enhanced word vectors to obtain preliminary fused features, and performs batch normalization on the preliminary fused features. By calculating the feature mean and variance, each feature element is standardized to stabilize the feature distribution. After processing, the system integrates the standardized features into a three-dimensional array to form a fused feature tensor, where each dimension corresponds to the text sequence length, feature dimension, and batch size, respectively.

[0117] S42: The fused feature tensor is processed by a bidirectional gated recurrent unit to capture the long-distance contextual dependencies of the text sequence. Important semantic information is preserved through the gating mechanism to generate time-aware features.

[0118] Specifically, the system reads the dimensionality information of the fused feature tensor and converts the three-dimensional tensor into a sequence form according to the input requirements of the bidirectional gated loop unit. During the conversion, the system extracts feature vectors in the order of the text sequence, maintaining the temporal relationship between the vectors. For sequences whose length does not meet the requirements, the system adds zero-value vectors to both ends of the sequence to make all sequences the same length. Preferably, the system calls the bidirectional gated loop unit, which includes two gating mechanisms: an update gate and a reset gate. The update gate determines the degree to which the state information from the previous moment is retained and the degree to which the current input information is included, while the reset gate determines the degree to which the state information from the previous moment is ignored. The system calculates the values ​​of the two gates using the sigmoid function, with values ​​ranging from zero to one. The closer the update gate value is to one, the more state information from the previous moment is retained; the closer the reset gate value is to one, the more state information from the previous moment is used in the calculation of the current state.

[0119] The system calculates the candidate state and final state at the current time step based on the gating value. The candidate state is generated using the tanh function, and the final state is obtained by combining the state from the previous time step and the current candidate state. The output states of the forward and backward gating loop units are concatenated at each time step to form the temporal-aware feature for that moment. The system concatenates the temporal-aware features from all time steps in sequence to form a complete temporal-aware feature sequence, which contains long-range contextual dependencies within the text sequence.

[0120] S43: Perform CRF decoding on the time-aware features, optimize the label transition probability through the Viterbi algorithm, select the globally optimal character position label sequence, and generate the initial word segmentation sequence.

[0121] Specifically, after acquiring time-series-aware features, the system inputs them into a Conditional Random Field (CRF) model. Each vector in the time-series-aware features is processed through a state feature function. This function combines the element values ​​of the feature vectors with preset weight parameters to calculate the score for each character position corresponding to different labels. Label types include word beginning, word middle, and word ending, used to mark the character's position within the vocabulary. It should be noted that a Conditional Random Field (CRF) is a discriminative probabilistic graphical model, primarily used to model dependencies between variables and calculate conditional probability distributions.

[0122] For example, the system invokes a dynamic semantic dictionary to extract boundary information of culturally loaded words. For the positions in the time-series-aware features corresponding to culturally loaded words, the system adjusts the initial score output by the state feature function based on the boundary information to improve the label scores that meet the boundary requirements. Subsequently, the system calculates the transition score between adjacent labels using a transition feature function. This function, based on preset transition weights and the types of the preceding and following labels, outputs a value reflecting the rationality of the label sequence.

[0123] Preferably, the system calls the Viterbi algorithm to optimize the label sequence. Starting from the first position of the sequence, for each possible label, it calculates the maximum cumulative score of the current label by combining the maximum cumulative score of the labels at the previous position and the transition score between the current label and the previous label, and records the corresponding predecessor label. The algorithm traverses the entire sequence in this way until the last position. After reaching the end of the sequence, the algorithm starts from the label corresponding to the maximum cumulative score at the last position and backtracks to the predecessor label of each position to form a complete optimal label sequence. Based on the label sequence, it determines the word segmentation boundary, combines the characters corresponding to consecutive word beginning labels and word middle labels into words, and takes the position of the word ending label as the end point of the word. After processing, the system arranges these words in order to form the initial word segmentation sequence.

[0124] In one embodiment, such as Figure 3 As shown, S5 of the domain-specific Chinese word segmentation method based on transfer learning provided by this invention specifically includes the following steps:

[0125] S51: Process the initial word segmentation sequence, construct a word position index by combining it with a dynamic semantic dictionary, and generate a word mapping table to be optimized.

[0126] Specifically, after receiving the initial word segmentation sequence, the system scans the sequence word by word, recording the start and end positions of each word (represented by character indices). The system then calls a dynamic semantic dictionary and uses a string matching algorithm to compare the words in the initial word segmentation sequence with the culturally loaded words in the dictionary, marking the successfully matched words and their position indices in the sequence. For unmatched words, the system also records their position information as a reference for subsequent contextual analysis.

[0127] The system constructs a lexical position index, stored using a key-value pair structure. The key is the lexical's sequential number in the sequence, and the value includes the lexical text, the starting character index, the ending character index, and a flag indicating whether it is a culture-loaded word. After the index is constructed, the system uses the formula I(w k )=(s k ,e k ,l k ) represents the position information of the k-th word, where I(w k ) represents the location index, sk e is the starting character index. k To terminate the character index, l k The marker is 1 for culturally loaded words and 0 for ordinary words.

[0128] The system associates the position index with the word text in the initial word segmentation sequence and arranges them in word order to form a word mapping table to be optimized. This table contains basic information and position tags of all words in the sequence, providing structured data support for subsequent extraction of context vectors and ensuring that the context range of each word can be located through the index.

[0129] S52: Process the vocabulary mapping table to be optimized, extract the local context vectors of the preset window size before and after each word, aggregate the context semantic information through average pooling operation, and generate the context semantic representation.

[0130] Specifically, after obtaining the vocabulary mapping table to be optimized, the system reads the start and end indices of each vocabulary record in the table to determine the specific position of each vocabulary in the initial text sequence. Based on a preset window size, the system determines the context range to be extracted before and after each vocabulary. The window size is set through a configuration file and applies to context extraction for all vocabulary. For vocabulary at the beginning or end of the sequence, if the window range exceeds the sequence boundary, the system extracts the actual existing context vocabulary, limited by the sequence boundary.

[0131] The system extracts words within the window range before and after each word from the initial word segmentation sequence, forming a local context word set. For each local context word set, the system calls a word vector model to generate corresponding word vectors. The word vector model is consistent with the model used in previous steps to ensure the uniformity of the vector space. The system performs average pooling on the word vectors of the local context word set, calculates the mean of all vector elements, and obtains an aggregated vector, which is the local context vector of that word.

[0132] The system associates the local context vector of each word with its own word vector to form a contextual semantic representation. This representation includes the semantic information of the word itself and the semantic information of its surrounding context, reflecting the meaning of the word in a specific context. The system performs a dimensionality check on the generated contextual semantic representations to ensure that all representations have the same vector dimension. If there are dimensional differences, they are adjusted through linear transformation.

[0133] S53: Process the contextual semantic representation, measure the word-context relevance through a bilinear transformation matrix, and generate an optimized score vector.

[0134] Specifically, after acquiring the contextual semantic representation, the system uses the contextual semantic representation vector of each word as input to prepare for relevance calculation. The system introduces a bilinear transformation matrix, which is a trainable parameter. Its initial values ​​are generated through random initialization and updated via backpropagation during model training to optimize the accuracy of relevance calculation. The dimension of the bilinear transformation matrix matches the dimension of the contextual semantic representation vector, ensuring the validity of matrix operations.

[0135] Preferably, the system processes the context semantic representation vector through a bilinear transformation matrix. The calculation process involves matrix-vector multiplication to obtain an intermediate vector. The intermediate vector is then processed by an activation function to generate a correlation value between the word and its context. The correlation value reflects the degree of semantic matching between the word and its context. The higher the value, the stronger the semantic coherence of the word in the context.

[0136] The system arranges the relevance scores of each word in the order they appear in the word mapping table to be optimized, forming a one-dimensional vector, which is the optimization score vector. Each element in the optimization score vector corresponds to the relevance score of a word, and the element's position corresponds one-to-one with the word's position in the sequence. The system normalizes the optimization score vector to ensure that the element values ​​are within a uniform range, facilitating comparisons of relevance scores between different words.

[0137] S54: Perform Viterbi path search on the optimized score vector to find the globally optimal segmentation path and generate Chinese word segmentation results.

[0138] Specifically, after obtaining the optimized score vector, the system constructs a segmentation path graph by combining the word position information in the word mapping table to be optimized. Nodes in the segmentation path graph represent the positions of words in the sequence, and edges between nodes represent the connections between words. The weight of each edge is determined by the relevance value of the corresponding word in the optimized score vector; the higher the relevance value, the greater the edge weight. When constructing the path graph, the system ensures that all possible segmentation paths are included, including paths based on the initial word segmentation sequence and possible adjustment paths.

[0139] Preferably, the system employs the Viterbi algorithm to search the segmentation path graph and find the globally optimal segmentation path. Starting from the beginning of the sequence, the algorithm calculates the maximum cumulative weight of each position leading to the current node, while simultaneously recording the optimal predecessor node for that node. During the calculation, the algorithm considers all possible predecessor nodes and selects the path with the highest cumulative weight as the optimal path for the current node.

[0140] When the algorithm reaches the end position of the sequence, it backtracks to the optimal predecessor node to obtain the globally optimal segmentation path from the start position to the end position. Based on the optimal segmentation path, the system determines the final word segmentation boundary, adjusts the segments in the initial word segmentation sequence that do not conform to the optimal path, merges or splits the corresponding words, and forms the final Chinese word segmentation result. The Chinese word segmentation result contains a list of words arranged in order, with each word accompanied by its start and end indices in the original text, as well as the corresponding relevance value.

[0141] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0142] In summary, the proprietary domain-specific Chinese word segmentation method provided in this application accurately locates the position of culturally loaded words by constructing a lexicon mapping table to be optimized, enabling targeted optimization of compound terms and solving the semantic fragmentation problem caused by boundary recognition errors of culturally loaded words in the background technology. The semantic representation method based on local context window extraction and average pooling aggregation can achieve efficient fusion of contextual information, effectively capturing the association features between terms such as "calligraphy import method" and the preceding and following semantics, overcoming the lack of semantic consistency caused by long-distance dependence in traditional models. The use of a bilinear transformation matrix to measure the word-context association degree can establish a fine-grained semantic association model, significantly improving the semantic adaptability of compound words such as "scenario creation method" to teaching scenarios, thus addressing the insufficient adaptability of static similarity calculation methods in dynamic contexts. By searching for the globally optimal segmentation path using the Viterbi algorithm, a cross-lexicon collaborative decision-making mechanism can be realized, eliminating the fragmented segmentation phenomenon of terms such as "educational involution" in the initial word segmentation, ensuring the structural integrity of culturally loaded words in complete semantic units.

[0143] Based on the same inventive concept, this application also provides a transfer learning-based proprietary domain Chinese word segmentation system for implementing the aforementioned transfer learning-based proprietary domain Chinese word segmentation method. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of the one or more transfer learning-based proprietary domain Chinese word segmentation system embodiments provided below can be found in the limitations of the transfer learning-based proprietary domain Chinese word segmentation method described above, and will not be repeated here.

[0144] Preferably, such as Figure 4 As shown, this invention provides a domain-specific Chinese word segmentation system 600 based on transfer learning, which is configured with the following modules:

[0145] The cultural load word extraction module 610 is used to extract cultural load words from the acquired international education field public corpus and cultural load word database, calculate the semantic similarity between candidate words and typical cultural context words, filter out words with cross-cultural semantic conflicts, and generate a dynamic semantic dictionary.

[0146] The cultural semantic perception module 620 is used to perform cultural semantic perception processing on the text sequence input by the user and the dynamic semantic dictionary. It generates culturally enhanced word vectors by weighting the vector representation of culturally loaded words through an attention mechanism.

[0147] The domain feature alignment module 630 is used to acquire target domain data of the user output text sequence, process it in combination with a pre-trained general Chinese word segmentation model, eliminate domain differences through adversarial training and feature distribution alignment, and generate domain-invariant features.

[0148] The initial word segmentation generation module 640 is used to process culturally enhanced word vectors and domain-invariant features. It optimizes the word segmentation boundary by fusing sequence labeling and boundary detection tasks to generate the initial word segmentation sequence.

[0149] The word segmentation result optimization module 650 is used to process the initial word segmentation sequence and dynamic semantic dictionary, calculate the semantic similarity score between cultural load words and their context, optimize the Chinese segmentation path, and generate Chinese word segmentation results. The Chinese word segmentation results are used to display the segmented text sequence.

[0150] Preferably, the cultural load word extraction module 610 provided in this application is configured with the following units:

[0151] The candidate vocabulary filtering unit is used to process the acquired international education field public corpus and culture-loaded word database, calculate the cultural context weight factor of the terms, filter high-frequency cultural terms in the corpus by combining the TF-IDF algorithm, calculate the term weights by word frequency statistics and inverse document frequency weighting, and generate a candidate vocabulary set.

[0152] The semantic similarity calculation unit is used to process the candidate word set and the cultural load word database. Based on the cultural semantic projection transformation of the word vector model, it calculates the cosine similarity between candidate words and cultural context words and generates a weighted similarity matrix.

[0153] The high-conflict word screening unit is used to perform cross-cultural conflict screening on the similarity matrix, select words with similarity higher than a preset dynamic threshold as high-conflict cultural load words, and generate a dynamic semantic dictionary. The dynamic semantic dictionary is used to indicate the recognition boundary of cultural load words.

[0154] Preferably, the cultural semantic perception module 620 provided in this application is configured with the following units:

[0155] The character vector mapping unit is used to process the text sequence input by the user. It maps characters into dense vectors through a domain-adaptive word embedding model to generate a character vector sequence.

[0156] The attention distribution calculation unit is used to process character vector sequences and dynamic semantic dictionaries, call the multi-head attention mechanism to calculate the association strength between culture-loaded words and context, and generate a three-dimensional attention distribution.

[0157] The cultural semantic enhancement unit processes the 3D attention distribution and character vector sequence. It controls the enhancement intensity of cultural semantic components through learnable gating parameters to generate culturally enhanced word vectors. These culturally enhanced word vectors are used to indicate the feature fusion input for multi-task joint decoding.

[0158] Preferably, the domain feature alignment module 630 provided in this application is configured with the following units:

[0159] The parameter transfer and base construction unit is used to perform parameter transfer processing on the pre-trained general Chinese word segmentation model, freeze the underlying Transformer encoder to retain general language knowledge, and generate a domain-adaptive base.

[0160] The adversarial training processing unit is used to acquire target domain data of user-output text sequences, perform adversarial training processing in conjunction with the domain adaptive base, and confuse the feature discrimination signals of the source domain and the target domain through a gradient inversion layer and a domain classifier to generate domain robust features.

[0161] The feature distribution alignment unit is used to perform distribution alignment processing on the domain robust features. Based on the Wasserstein distance, it minimizes the feature distribution difference between the source and target domains to generate domain invariant features. These domain invariant features are used to indicate the cross-domain general representation in subsequent multi-task decoding.

[0162] Preferably, the initial word segmentation generation module 640 provided in this application is configured with the following units:

[0163] The cross-modal feature alignment unit is used to process culturally enhanced word vectors and domain-invariant features. It aligns the feature spaces of cultural semantic representation and domain-general representation based on the cross-modal attention mechanism, and fuses the two heterogeneous features through cross-attention weights to generate a fused feature tensor.

[0164] The temporal feature capture unit is used to process the fused feature tensor through bidirectional gated recurrent units, capture the long-distance contextual dependencies of the text sequence, retain important semantic information through the gating mechanism, and generate temporal-aware features;

[0165] The word segmentation tag decoding unit is used to perform CRF decoding on time-aware features. It optimizes the tag transition probability through the Viterbi algorithm, selects the globally optimal character position tag sequence, and generates the initial word segmentation sequence.

[0166] Preferably, the word segmentation result optimization module 650 provided in this application is configured with the following units:

[0167] The vocabulary position index unit is used to process the initial word segmentation sequence, combine it with the dynamic semantic dictionary to construct the vocabulary position index, and generate a vocabulary mapping table to be optimized.

[0168] The context vector extraction unit is used to process the vocabulary mapping table to be optimized, extract the local context vectors of a preset window size before and after each word, and aggregate the context semantic information through average pooling to generate a context semantic representation.

[0169] The relevance measurement unit is used to process the context semantic representation, measure the word-context relevance through a bilinear transformation matrix, and generate an optimized score vector;

[0170] The optimal path search unit is used to perform Viterbi path search on the optimized score vector to find the globally optimal segmentation path and generate Chinese word segmentation results.

[0171] In one embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described domain-specific Chinese word segmentation method based on transfer learning.

[0172] In one embodiment, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned domain-specific Chinese word segmentation method based on transfer learning.

[0173] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0174] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0175] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A domain-specific Chinese word segmentation method based on transfer learning, characterized in that, Includes the following steps: S1: Extract culturally loaded words from the obtained international education corpus and culturally loaded word database, calculate the semantic similarity between candidate words and typical cultural context words, and filter out words with cross-cultural semantic conflicts to generate a dynamic semantic dictionary. The typical cultural context terms include benchmark terms of Eastern educational philosophies and Western teaching models, and the dynamic semantic dictionary is used to indicate the recognition boundaries of culturally loaded terms. S2: Perform cultural semantic perception processing on the text sequence input by the user and the dynamic semantic dictionary, and generate culturally enhanced word vectors by weighting the vector representation of culturally loaded words through an attention mechanism; S3: Acquire target domain data, process it in conjunction with a pre-trained general Chinese word segmentation model, eliminate domain differences through adversarial training and feature distribution alignment, and generate domain-invariant features. The target domain data includes international curriculum standards, cross-border teaching case sets, teaching plans, and course description text sequences. S4: Process the culture-enhanced word vectors and the domain-invariant features, optimize the word segmentation boundary by fusing sequence labeling and boundary detection tasks, and generate an initial word segmentation sequence; S5: Process the initial word segmentation sequence and the dynamic semantic dictionary, calculate the semantic similarity score between the cultural load words and their context, optimize the Chinese segmentation path, and generate Chinese word segmentation results. The Chinese word segmentation results are used to display the segmented text sequence.

2. The method according to claim 1, characterized in that, S1 includes: S11: Process the acquired international education field public corpus and culture-loaded word database, calculate the cultural context weight factor of the terms, combine the TF-IDF algorithm to screen high-frequency cultural terms in the corpus, calculate the term weights through word frequency statistics and inverse document frequency weighting, and generate a candidate vocabulary set; S12: Process the candidate vocabulary set and the cultural load word database, calculate the cosine similarity between candidate words and cultural context words based on the cultural semantic projection transformation of the word vector model, and generate a weighted similarity matrix. S13: Perform cross-cultural conflict screening on the similarity matrix, select words with similarity higher than a preset dynamic threshold as high-conflict cultural load words, and generate a dynamic semantic dictionary. The dynamic semantic dictionary is used to indicate the recognition boundary of cultural load words.

3. The method according to claim 2, characterized in that, The cultural context weighting factor is calculated using the following formula: ； in, As a cultural context weighting factor, The number of times term t appears in the culturally loaded word database. This represents the total number of terms in the corpus.

4. The method according to claim 1, characterized in that, S2 includes: S21: Process the text sequence input by the user, and map the characters into dense vectors through a domain-adaptive word embedding model to generate a character vector sequence; S22: Process the character vector sequence and the dynamic semantic dictionary, call the multi-head attention mechanism to calculate the association strength between culture-loaded words and context, and generate a three-dimensional attention distribution; S23: Process the three-dimensional attention distribution and the character vector sequence, control the enhancement intensity of cultural semantic components through learnable gating parameters, and generate culturally enhanced word vectors, which are used to indicate the feature fusion input for multi-task joint decoding.

5. The method according to claim 1, characterized in that, S3 includes: S31: Perform parameter transfer processing on the pre-trained general Chinese word segmentation model, freeze the underlying Transformer encoder to retain general language knowledge, and generate a domain-adaptive base; S32: Acquire target domain data, perform adversarial training processing in conjunction with the domain adaptive base, and confuse the feature discrimination signals of the source domain and the target domain through a gradient inversion layer and a domain classifier to generate domain robust features; S33: Perform distribution alignment processing on the domain robust features, and generate domain invariant features by minimizing the feature distribution difference between the source domain and the target domain based on the Wasserstein distance. The domain invariant features are used to indicate the cross-domain general representation in subsequent multi-task decoding.

6. The method according to claim 1, characterized in that, S4 includes: S41: Process the culture-enhanced word vectors and the domain-invariant features, align the feature spaces of the cultural semantic representation and the domain-general representation based on the cross-modal attention mechanism, and fuse the two heterogeneous features through cross-attention weights to generate a fused feature tensor; S42: The fused feature tensor is processed by a bidirectional gated recurrent unit to capture the long-distance contextual dependencies of the text sequence, retain important semantic information through a gating mechanism, and generate time-aware features; S43: Perform CRF decoding on the time-aware features, optimize the label transition probability using the Viterbi algorithm, select the globally optimal character position label sequence, and generate the initial word segmentation sequence.

7. The method according to any one of claims 1-6, characterized in that, S5 includes: S51: Process the initial word segmentation sequence, combine it with the dynamic semantic dictionary to construct a word position index, and generate a word mapping table to be optimized; S52: Process the word mapping table to be optimized, extract the local context vector of the preset window size before and after each word, aggregate the context semantic information through average pooling operation, and generate the context semantic representation; S53: Process the context semantic representation, measure the word-context relevance through a bilinear transformation matrix, and generate an optimized score vector; S54: Perform Viterbi path search on the optimized score vector to find the globally optimal segmentation path and generate Chinese word segmentation results.

8. A domain-specific Chinese word segmentation system based on transfer learning, characterized in that, The system includes: The cultural load word extraction module is used to extract cultural load words from the acquired international education field public corpus and cultural load word database, calculate the semantic similarity between candidate words and typical cultural context words, filter out words with cross-cultural semantic conflicts, and generate a dynamic semantic dictionary. The typical cultural context terms include benchmark terms of Eastern educational philosophies and Western teaching models, and the dynamic semantic dictionary is used to indicate the recognition boundaries of culturally loaded terms. The cultural semantic perception module is used to perform cultural semantic perception processing on the text sequence input by the user and the dynamic semantic dictionary, and generate culturally enhanced word vectors by weighting the vector representation of culturally loaded words through an attention mechanism. The domain feature alignment module is used to acquire target domain data, process it in conjunction with a pre-trained general Chinese word segmentation model, eliminate domain differences through adversarial training and feature distribution alignment, and generate domain-invariant features. The target domain data includes international curriculum standards, cross-border teaching case sets, teaching plans, and course description text sequences. The initial word segmentation generation module is used to process the culture-enhanced word vectors and the domain-invariant features, optimize the word segmentation boundary by fusing sequence labeling tasks and boundary detection tasks, and generate an initial word segmentation sequence. The word segmentation result optimization module is used to process the initial word segmentation sequence and the dynamic semantic dictionary, calculate the semantic similarity score between culturally loaded words and their context, optimize the Chinese segmentation path, and generate Chinese word segmentation results. The Chinese word segmentation results are used to display the segmented text sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images