A method for text segmentation of unearthed bamboo and wooden slips based on transfer learning

Abstract

The application discloses a kind of based on transfer learning's out of earth bamboo slips and wooden slips literature text word segmentation method, this method includes: constructing bamboo slips and wooden slips original text data set and literature word segmentation data set;Mask language model is pre-trained on bamboo slips and wooden slips original text based on differentiating mask strategy, obtain first pre-training model parameter;Sequence labeling word segmentation model is trained on literature word segmentation data, obtain second model parameter;The shared parameter of two models is weighted fusion, keeps the word segmentation task exclusive parameter unchanged;Based on the fusion parameter, retrain and construct transfer initialization word segmentation model on literature data;Finally, the final word segmentation model is obtained by fine-tuning on bamboo slips and wooden slips word segmentation annotation data.The application processes missing and uncertain characters by differentiating mask strategy, explicitly models multi-character word structure information by word fragment embedding, realizes cross-domain knowledge transfer by parameter fusion, effectively improves the accuracy and generalization ability of bamboo slips and wooden slips text word segmentation under the condition of limited annotation corpus.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a word segmentation method for unearthed bamboo and wooden slips documents based on transfer learning. Background Technology

[0002] While significant progress has been made in automatic word segmentation technology for Chinese text, word segmentation for unearthed bamboo and wooden slips still faces the following technical challenges:

[0003] The scale of annotated corpora is severely insufficient. Deep learning methods typically rely on large-scale manually annotated corpora for supervised training. However, word segmentation and annotation of unearthed bamboo and wooden slips requires the participation of professional scholars with knowledge in multiple disciplines such as paleography, classical Chinese, and history, resulting in extremely high annotation costs and long cycles. Currently, the publicly available corpora of Qin bamboo and wooden slips with professional word segmentation and annotation are extremely limited in size, making it difficult to support training deep neural network models from scratch. This data bottleneck makes it difficult to directly apply existing deep learning-based word segmentation methods in the context of unearthed documents.

[0004] The text contains significant noise. Due to long-term burial and physical weathering, unearthed bamboo and wooden slips generally have text quality problems such as missing characters, variant characters, phonetic loan characters, and interpretation discrepancies. These noise characteristics make it difficult to directly apply existing word segmentation models designed for standard modern Chinese texts.

[0005] Existing transfer learning schemes have shortcomings. Some existing studies attempt to directly transfer word segmentation models from modern Chinese to classical Chinese scenarios, but due to the significant linguistic differences between the source domain (modern Chinese) and the target domain (Qin bamboo slips), the results of direct transfer are usually unsatisfactory. Furthermore, existing schemes do not fully consider the unique noisy characteristics of unearthed documents and lack effective mechanisms for handling missing and uncertain characters. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a word segmentation method for unearthed bamboo and wooden slips based on transfer learning. This method fully considers the linguistic characteristics and data conditions of unearthed bamboo and wooden slips, and effectively improves the accuracy and generalization ability of word segmentation of bamboo and wooden slips texts under the condition of limited labeled corpus by constructing a transfer learning framework that combines pre-training of a masked language model with training of a word segmentation model.

[0007] Accordingly, this invention discloses a word segmentation method for unearthed bamboo and wooden slips documents based on transfer learning, comprising the following steps:

[0008] Construct a target domain dataset and a source domain dataset. The target domain dataset includes a raw text dataset of bamboo and wooden slips without word segmentation annotation and a bamboo and wooden slips word segmentation annotation dataset with word segmentation annotation. The source domain dataset includes a document word segmentation corpus with a similar system to the target domain dataset. Both the document word segmentation corpus and the bamboo and wooden slips word segmentation annotation dataset use the same annotation standard for character-level word segmentation annotation.

[0009] Based on the original text dataset of the bamboo and wooden slips, training samples for a masked language model are constructed using a differential masking strategy. The differential masking strategy includes executing the corresponding mask prediction participation probability according to the interpretation state of the character. The masked language model is pre-trained based on the training samples to obtain the first pre-trained model parameters. The masked language model includes an input representation layer, a context encoding layer, and a prediction layer connected in sequence.

[0010] The sequence labeling word segmentation model is trained on the source domain dataset to obtain the second model parameters. The sequence labeling word segmentation model includes an input representation layer, a context encoding layer, a label mapping layer, a sequence structure modeling layer, and a sequence decoding layer connected in sequence.

[0011] The same parameters in the first pre-trained model parameters and the second model parameters are weighted and fused, while keeping the parameters of the label mapping layer and the sequence structure modeling layer unchanged, to obtain the fused model parameters.

[0012] The sequence labeling word segmentation model is initialized based on the parameters of the fusion model, and then retrained based on the source domain dataset to obtain the transfer initialization word segmentation model;

[0013] The transfer initialization word segmentation model is trained under supervised fine-tuning based on the aforementioned bamboo and wooden slips word segmentation annotation dataset to obtain the final bamboo and wooden slips text word segmentation model.

[0014] In summary, this invention provides a word segmentation method for unearthed bamboo and wooden slips based on transfer learning. This method employs a multi-stage training strategy: pre-training a masked language model on the original bamboo and wooden slip text, training a word segmentation model on the document's word segmentation data, achieving knowledge transfer through parameter fusion, and then fine-tuning the target domain to adapt to the linguistic features of the bamboo and wooden slips. This effectively improves the accuracy and generalization ability of word segmentation on bamboo and wooden slip texts even with limited labeled corpora. This method is not only applicable to word segmentation of Qin bamboo and wooden slip texts but can also be extended to the automatic word segmentation processing of other unearthed ancient Chinese documents, providing strong technical support for digital humanities research and the digital organization of unearthed documents. Attached Figure Description

[0015] Figure 1 This is a flowchart of a text segmentation method for unearthed bamboo and wooden slips based on transfer learning, according to an embodiment of the present invention.

[0016] Figure 2 This is a flowchart illustrating the construction process of the target domain dataset and the source domain dataset in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the mask language model structure according to an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the coding unit structure according to an embodiment of the present invention.

[0019] Figure 5 This is a schematic diagram of the sequence labeling word segmentation model structure according to an embodiment of the present invention.

[0020] Figure 6 This is a schematic diagram of the migration initialization word segmentation model structure in an embodiment of the present invention.

[0021] Figure 7 This is a schematic diagram of the word segmentation model for bamboo and wooden slips according to an embodiment of the present invention.

[0022] Figure 8 This is a schematic diagram of the gated fusion module structure according to an embodiment of the present invention. Detailed Implementation

[0023] To further illustrate the technical means and effects adopted by this application to achieve its intended purpose, the specific implementation methods, structures, features, and effects according to this application are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0024] To address the shortcomings of existing technologies, this invention provides a word segmentation method for unearthed bamboo and wooden slips based on transfer learning. This method fully considers the linguistic characteristics and data conditions of the unearthed bamboo and wooden slips, and effectively improves the accuracy and generalization ability of word segmentation on bamboo and wooden slips under the condition of limited labeled corpus by constructing a transfer learning framework that combines pre-training of a masked language model with training of a word segmentation model.

[0025] Figure 1 This is a flowchart of a text segmentation method for unearthed bamboo and wooden slips based on transfer learning, according to an embodiment of the present invention.

[0026] Specifically, the technical solution of the present invention includes the following basic steps:

[0027] S101: Construction of the target domain dataset and the source domain dataset.

[0028] The target domain dataset (bamboo and wooden slips dataset) and the source domain dataset (document word segmentation dataset) are constructed. The target domain dataset includes the original text dataset of bamboo and wooden slips without word segmentation annotation and the bamboo and wooden slips word segmentation annotation dataset with manual word segmentation annotation; the source domain dataset includes a document word segmentation corpus dataset with a language system similar to that of bamboo and wooden slips text. Both the bamboo and wooden slips word segmentation annotation dataset and the document word segmentation corpus dataset are annotated with character-level word segmentation according to the BMES annotation specification to ensure consistency in annotation granularity and structural form between different data sources.

[0029] Specifically, the target domain can be understood as the unearthed bamboo and wooden slips text that actually needs to be segmented, and the source domain can be understood as the transfer object that is close to the target domain in age (and language family) and has an authoritative corpus. The substantive operation to be performed in this embodiment of the invention is to adaptively transfer the segmentation logic of the source domain to the target domain to realize the automatic segmentation function of the target domain.

[0030] This embodiment uses the bamboo slips from the Qin tomb at Shuihudi as the target domain and the documents from the reign of Duke Zhao of Zuo (541 BC to 510 BC) as the source domain data source, and details the construction process of the target domain dataset and the source domain dataset.

[0031] Figure 2 This is a flowchart illustrating the construction process of the target domain dataset and source domain dataset according to an embodiment of the present invention. The construction process of the target domain dataset and source domain dataset according to an embodiment of the present invention includes:

[0032] S201: Construction of the original text dataset of bamboo and wooden slips.

[0033] The following excerpts of authentic Qin bamboo slip text are taken from the transcription of the Qin bamboo slips from the Shuihudi Qin Tomb as examples:

[0034] "Sikong Teng wrote a letter to the official, saying that it was permissible to allow the official to file a complaint, but it was inappropriate."

[0035] The above text is serialized at the character level, with each Chinese character treated as an independent record and arranged in the original text order.

[0036] The basic format of the dataset is one character per line, with blank lines separating adjacent sentences. Accordingly, the target domain dataset of this embodiment contains 17 Chinese characters. The text in the original bamboo and wooden slips text dataset is not labeled with any word segmentation or part-of-speech tags and is used for the pre-training of the subsequent masked language model.

[0037] S202: Construction of Bamboo and Wooden Slips Word Segmentation and Annotation Dataset.

[0038] On the basis of the aforementioned original text dataset of bamboo and wooden slips, ancient Chinese domain experts perform manual word segmentation and annotation with reference to the word segmentation specifications of the part-of-speech tagging set of Peking University. The basic principle of word segmentation is to split the continuous character sequence into the minimum linguistic units with independent semantics on the premise of respecting the grammatical structure and semantic integrity of ancient Chinese.

[0039] The word segmentation result of the sample text is: "Sikong / Tengshu / all / reply / say / allow / permit / sit / official / inform / judge / should not", the annotated dataset adopts the same character-level organization form as the original text dataset, each record contains a character and the corresponding BMES tag, and sentences are separated by blank lines.

[0040] In accordance with the BMES annotation specification, the annotation tag corresponding to each character is shown in Table 1.

[0041] Table 1

[0042] S203: Constructing a literature word segmentation dataset.

[0043] For example, the following literature fragment from *Zuo Zhuan · The First Year of Duke Zhao* is selected as the source domain training corpus: "The Marquis of Jin sought a doctor in Qin, and the Earl of Qin sent Doctor He to examine him".

[0044] The word segmentation and annotation result of this fragment is: Marquis of Jin / seek / doctor / in / Qin / Earl of Qin / send / Doctor He / examine / him.

[0045] In accordance with the BMES annotation specification, the annotation tag corresponding to each character is shown in Table 2.

[0046]

[0047] Table 2

[0048] In the process of constructing the aforementioned dataset, preprocessing operations need to be performed on the original text, and the preprocessing operations include:

[0049] Whitespace character cleaning: delete redundant spaces, tabs and line breaks generated during the optical character recognition process, and normalize the text into a continuous character stream;

[0050] Variant character normalization: establish a variant character mapping table to uniformly map the non-standard writing forms in Qin bamboo slips to standard glyph encodings. For example, normalize "羣" to "群", and map some special shorthand variants of the character "大" to the standard encoding of the character "大";

[0051] Uncertain character processing: for characters with uncertain interpretation (such as completely missing characters marked with "□" or doubtful characters marked with superscript "?"), retain their marking symbols for subsequent differentiated mask processing;

[0052] Low-quality segment filtering: calculating the proportion of uncertain characters in each text segment , where is the number of uncertain characters, is the total number of characters. When (in this embodiment, take ), delete this segment;

[0053] Duplicate segment deduplication: using a hash algorithm to detect and delete duplicate text segments in the dataset, so as to avoid deviations in model training caused by duplicate samples.

[0054] The role of the preprocessing operation is to reduce the impact of noisy data on the model training effect.

[0055] S102: Training a masked language model based on a differentiated masking strategy.

[0056] Figure 3 is a schematic structural diagram of the masked language model according to the embodiment of the present invention.

[0057] Based on the original bamboo slip text dataset, training samples for the masked language model are constructed through the differentiated masking strategy. The data in the original bamboo slip text dataset is organized into structured data files by the masked language model for batch loading and training, where each sample at least includes character sequence information, masked position indices, and mask prediction label information, to support efficient subsequent training of the model.

[0058] The main content of this differentiated masking strategy is to implement different participation probabilities for mask prediction on characters in different interpretation states (determined characters, uncertain characters, and missing characters), so as to reduce the interference of uncertain annotations on model training, thereby generating training samples that are beneficial to improving the quality of the masked language model; then the training samples are used to pre-train the masked language model including an input representation layer, a context encoding layer and a prediction layer, to obtain first pre-trained model parameters.

[0059] Specifically, in order to standardize the content of the input data, standardized processing needs to be performed on the input data before the pre-training of the masked language model. The standardized processing means adopted in the embodiment of the present invention is character set serialization processing, that is, a sentence in the original bamboo slip text dataset is split into character levels according to rules before training the masked language model.

[0060] The text segment of Qin bamboo slips in Embodiment 1 is taken as an example to perform character-level serialization processing. Let the input character sequence be:

[0061] , which respectively represent [Si, Kong, Teng, Shu, Jie, Zeng, Bao, Yue, Ke, Xu, Yi, Zuo, Guan, Gao, Lun, Bu, Dang].

[0062] Insert special tokens at the start position and end position of the sequence respectively: , randomly select part of the character positions from the character-level sequence as mask prediction targets, and the mask positions are limited between <[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]> and [SEP], so as to avoid interfering with the semantics of the sequence boundary tokens.

[0063] In the embodiment of the present invention, a maximum of two character positions are selected as mask prediction targets for each input character sequence. When the length of the character sequence is insufficient to support the said number, the number of masked characters can be reduced correspondingly to ensure the effectiveness and stability of the mask sample construction process. For the selected masked character positions, if the corresponding character is a clearly interpreted character, the character is always used as a mask prediction target to participate in training; if the corresponding character is an uncertainly interpreted or damaged character, after it is selected as a mask position, the character is used as a mask prediction target to participate in training with a 50% probability, and in the remaining 50% of cases, the character is only used as a context input without participating in the prediction loss calculation, so as to reduce the interference of uncertain character annotations on the parameter learning process of the mask language model while maintaining the context semantic modeling ability, and improve the modeling effect of the mask language model on the context semantic information of Qin bamboo slip texts.

[0064] Specifically, set the preset maximum sequence length , the current sequence length is 19, which is less than the maximum length, so no truncation is required. For sequences with a length less than 128, pad the [PAD] token at the end until the length reaches 128.

[0065] Specifically, in this step, it is assumed that there are damaged characters in the unearthed bamboo slip fragments, such as: "司空□书皆赠报曰可许", where "□" represents a damaged character.

[0066] Correspondingly, when performing masking processing on this sequence, randomly select mask positions from the sequence to form a set , in this embodiment, it is set that each sequence can select at most mask positions. Assume that the randomly selected mask position set is , where position 3 corresponds to the character "□" (damaged character, ), position 9 corresponds to the character "可" (clearly interpreted character, ).

[0067] According to the differentiated masking strategy, for position 3 (damaged character), it is determined with a 50% probability whether to take it as a prediction target. If the random sampling result is yes, this position participates in the prediction loss calculation; if the sampling result is no, it is only used as a context input; for position 9 (determined character), it is always taken as a prediction target to participate in training.

[0068] Assume that in this masking process, the damaged character position 3 does not participate in prediction (serves only as context), and it is determined that character position 9 participates in prediction, then the masked input sequence is expressed as ([MASK] denotes the masked position):

[0069] Correspondingly, the prediction target label is: the real character "ke" at position 9. In specific implementation, on the basis of generating masked samples, continuous character combinations in the character sequence are analyzed to generate sub-word units. Sub-words can be divided based on high-frequency N-grams in the training corpus or fixed-length character combinations, each sub-word is assigned a unique identification ID, and the start position, end position and length information are recorded. The sub-word features are aligned with the corresponding character-level sequence to form an auxiliary feature tensor, which is encoded together with the character-level tensor and the mask position tensor, so as to obtain a joint representation containing both character information and potential word structure information, for enhancing the model's perception ability of multi-character word boundaries and combination relationships.

[0070] This embodiment adopts a dictionary-based maximum matching algorithm to identify sub-words. Specifically, for the example text, assume that the corresponding sub-words have been included in the dictionary, and the dictionary is used to process the character sequence to perform sub-word analysis, and the identification result is shown in the recorded content of Table 3.

[0071]

[0072] Table 3

[0073] After obtaining the sub-word identification result, a unique identification ID is assigned to each sub-word, and its position information in the sequence is recorded. For characters that belong to multiple sub-words at the same time (such as in the case of overlap), all sub-word embedding vectors they belong to are accumulated.

[0074] Finally, the processed character sequence, mask position information and sub-word structured features are organized and encoded to form a data representation structure suitable for masked language model training.

[0075] The encoded representation includes the order of each character in the sequence, the mask prediction target position, as well as the identification information, position and length information related to multi-character sub-words. In the actual training phase, these encoded information will be further converted into an input format acceptable to the model to support the training of the masked language model task. In this embodiment, the training samples are stored and managed in the form of structured data.

[0076] Specifically, in the embodiment of the present invention, the training samples are organized and stored in JSON format, and each sample independently records the corresponding character sequence information, mask prediction information and sub-word feature information.

[0077] Each sample contains the following information:

[0078] Character sequences (including special markers [CLS], [SEP], [PAD], and [MASK]);

[0079] Mask position index set;

[0080] The actual character label corresponding to the mask position;

[0081] Word fragment identifier, start position, and end position information;

[0082] Valid sequence length;

[0083] Decipher the state marker sequence.

[0084] The encoded samples are stored in JSON format, with the following example structure:

[0085] {

[0086] "token_ids":[101,2345,1890,103,4567,...],

[0087] "mask_positions":[9],

[0088] "mask_labels":

[3456] ,

[0089] "word_pieces":[

[0090] {"id":12,"start":1,"end":2},

[0091] {"id":45,"start":3,"end":4}

[0092] ],

[0093] "sequence_length":19,

[0094] "reading_states":[0,0,2,0,0,0,0,0,0,0,...]

[0095] }

[0096] It should be noted that existing masked language models typically employ a uniform random masking strategy during training, using the same method to train characters or words in the input sequence for masked prediction. This strategy is primarily designed for modern Chinese or texts with relatively standardized corpora, and does not fully consider the unique linguistic phenomena commonly found in ancient documents, especially unearthed bamboo and wooden slips, such as missing characters, variant forms, and uncertain interpretations. In unearthed bamboo and wooden slips, uncertain characters often lack stable semantic meaning. Directly equating them with definite characters for prediction training can easily introduce noise into the model parameter learning process, thereby affecting the language model's ability to model true semantic context relationships.

[0097] Therefore, in constructing training samples for the masked language model, this embodiment of the invention adopts a differentiated processing method for characters with uncertainty in the unearthed bamboo and wooden slips. This allows the masked language model to focus more on the contextual relationships between semantically stable characters during the pre-training stage, while weakening the interference of uncertain characters on the prediction target. This guides the masked language model to learn character representations that are more consistent with the linguistic structural features of the unearthed bamboo and wooden slips.

[0098] Specifically, the network structure of the masked language model in this embodiment of the invention includes an input representation layer, a context encoding layer, and a prediction layer. For differentiation, the masked language model in this embodiment includes a first input representation layer, a first context encoding layer, and a first prediction layer.

[0099] The input representation layer consists of three embedding modules: a character embedding module, a word fragment embedding module, and a position embedding module.

[0100] Specifically, assume the input character sequence is Each character First, the character embedding module generates a vector representation. ,in For the embedding dimension, the character embedding matrix is ​​randomly initialized and updated during pre-training to capture the semantic information of each character.

[0101] At the same time, word fragment units are generated from consecutive character combinations in the character sequence. Each word fragment Each element corresponds to a unique identifier (ID) and is mapped to a vector by the word fragment embedding module. Word fragment embedding can capture the boundary information and potential semantic relationships of multi-character combinations, thereby enhancing the model's ability to perceive multi-character words.

[0102] To preserve sequence information, each character position in the input sequence is further processed by a position embedding module to generate a position vector. The input is then added element-wise to the character embedding to obtain the final input representation:

[0103]

[0104] wherein the summation operation represents the superposition of all word piece embeddings where the character is located, and this design ensures that the masked language model can simultaneously perceive the character's own information, word piece structure information and sequence position information.

[0105] In the embodiment of the present invention, assume that the input character sequence is , and the embedding dimension is .

[0106] For the character (the 0th position is a special token, the character "Kong" is located at the 2nd position, belonging to the word piece "Sikong"), the processing steps include:

[0107] Character embedding: map "Kong" to a vector through the character embedding module . The character embedding matrix , wherein is the size of the character vocabulary (in this embodiment, , covering common Chinese characters, special characters on bamboo slips and special tokens).

[0108] Word piece embedding: the word piece "Sikong" is assigned an identifier , which is mapped by the word piece embedding module to , and the word piece embedding matrix , wherein is the size of the word piece vocabulary.

[0109] Position embedding: the position embedding vector at the 2nd position is , and the position embedding matrix is a learnable parameter.

[0110] Finally, the output representation of the input representation layer for the character "Kong" is the sum of the character embedding, the word piece embedding and the position embedding: , this design ensures that the masked language model can simultaneously perceive the character's own information, word piece structure information and sequence position information.

[0111] Figure 4 is a schematic structural diagram of the encoding unit in the embodiment of the present invention.

[0112] Specifically, the context encoding layer includes a sequence modeling module composed of multiple layers of encoding units, and each layer of encoding unit at least includes a multi-head self-attention module, a feedforward neural network module, as well as a residual connection and layer normalization module, which is used to model the global dependency relationship of the input sequence and the interaction information between characters and word pieces.

[0113] Specifically, the input representation of the context encoding layer is ,in Indicates the first A vector representing the input of 1 character. For sequence length, For the embedding dimension, vector It is obtained by superimposing character embedding, word fragment embedding and position embedding.

[0114] Input representation go through The process is performed by a layer-by-layer joint sequence coding unit. Each coding unit consists of three modules, among which the multi-head self-attention module processes the first... Layer input ( The input (from the first layer) is processed to generate a Query, Key, and Value matrix:

[0115]

[0116] in, , For trainable weight matrix, , For the number of attention heads.

[0117] The formula for calculating attention is:

[0118] .

[0119] Specifically, in the context encoding layer, after the input sequence undergoes multi-head self-attention computation, each character position (or word segment position) receives a context-aggregated feature vector. , dimension (i.e., embedding dimension) This operation yields a weighted combination of each character vector with other characters and their covered word segments, capturing dependencies across characters and word segments.

[0120] The feedforward neural module for each vector Independent nonlinear mapping is performed to enhance vector expressive power, fusing contextual information from characters and word fragments, and is represented as follows:

[0121]

[0122] in, , For trainable weight matrix, , For bias, The activation function is ReLU. This represents the intermediate layer dimension of the feedforward network.

[0123] The residual connection and layer normalization module is stacked through multiple layers of joint sequence coding units, and the final output is... Each vector represents the joint representation of a character sequence under global context and word fragment information. It simultaneously encodes the character's own information, the surrounding character context, and the semantic information of potential word fragments, providing input for the mask prediction task.

[0124] Specifically, the output of the residual connection and layer normalization module is:

[0125]

[0126] Wherein, LayerNorm applies each vector along its feature dimension Normalization is performed to ensure training stability, and residual connections enable the original input information to be effectively combined with the attention and FFN output information.

[0127] The prediction layer includes a feature mapping module and a probability normalization module. It maps the feature representation output by the context encoding layer to the prediction result corresponding to the masked position, i.e., the probability distribution of the masked position in the vocabulary, used to predict what the originally masked character is. Specifically, the feature mapping module selects feature vectors corresponding to the positions of the characters involved in the prediction from the sequence features output by the context encoding layer, and projects these feature vectors onto the vocabulary space through a linear mapping operation to obtain the prediction score for each candidate character. Subsequently, the probability normalization module converts the prediction score into a probability distribution, and calculates the training loss based on the probability distribution and the corresponding real character label, thereby optimizing and updating the model parameters.

[0128] In this embodiment of the invention, the context coding layer is composed of... The layers are stacked, and each layer contains a multi-head self-attention module, a feedforward neural network module, and a residual connection and layer normalization module.

[0129] With the first Taking the context coding layer as an example, the data processing procedure of the context coding layer includes:

[0130] Multi-head self-attention computation: Input .

[0131] Let the number of attention heads be Then the dimension of each head .

[0132] Generate query, key, and value matrix:

[0133]

[0134] in, .

[0135] Calculate the scaled dot product attention (considering the padding mask and the predicted mask):

[0136]

[0137] in Mask padding location [PAD], Mask the location of the mask to be predicted.

[0138] Feedforward neural network processing: Residual connections and layer normalization are performed on the attention output.

[0139]

[0140] Then through a feedforward neural network (intermediate layer dimension) ):

[0141]

[0142] in For activation function, , .

[0143] Perform residual connections and layer normalization again:

[0144]

[0145] After 12 layers of coding units, the final context-coded representation is obtained. .

[0146] The prediction layer selects the encoding vector corresponding to the mask position. Projected onto the vocabulary space via a linear mapping:

[0147]

[0148] in , .

[0149] The predicted score is converted into a probability distribution using the Softmax function:

[0150]

[0151] The training objective is to minimize the cross-entropy loss:

[0152]

[0153] in This is the set of mask positions that actually participate in the loss prediction calculation (excluding uncertain character positions that are not selected with a 50% probability).

[0154] Finally, after the masked language model is trained, the first pre-trained model parameters are obtained. It is important to note that in the subsequent fusion steps, the parameters of the input representation layer and the context encoding layer are mainly fused; the data from the prediction layer is only used internally by the masked language model. S103: Train a sequence labeling word segmentation model based on source domain data.

[0155] Figure 5 This is a schematic diagram of the sequence labeling word segmentation model structure according to an embodiment of the present invention.

[0156] The second model parameters are obtained by training a word segmentation task based on a sequence labeling word segmentation model combined with a source domain dataset.

[0157] The sequence labeling and word segmentation model includes an input representation layer, a context encoding layer, a label mapping layer, a sequence structure modeling layer based on conditional random fields, and a sequence decoding layer based on the Viterbi algorithm.

[0158] Specifically, the structure of the input representation layer and context encoding layer of the word segmentation model is the same as that of the masked language model. To distinguish them, the input representation layer in the sequence labeling word segmentation model is named the second input representation layer, and the context encoding layer is named the second context encoding layer. To ensure consistency, the label mapping layer, sequence structure modeling layer, and sequence decoding layer are also prefixed with "second".

[0159] Furthermore, the word segmentation model introduces a functional layer for sequence labeling tasks after the masked language model's encoding structure.

[0160] Specifically, the label mapping layer is used to map the continuous feature representation of each character position output by the context coding layer to the emission score corresponding to the word segmentation label set. This process is implemented by linear mapping to obtain the score representation of each character position corresponding to different word segmentation labels.

[0161] The sequence structure modeling layer adopts a conditional random field model to model the transition relationship between adjacent tags in the character sequence, so as to explicitly introduce global structural constraints between word segmentation tags.

[0162] The sequence decoding layer is used to decode the label path with the highest total score from all possible label sequences under the conditional random field model. Specifically, the sequence decoding layer uses the Viterbi algorithm to perform optimal path search on the conditional random field model.

[0163] Specifically, the data processing procedure in the functional layer is as follows:

[0164] Tag mapping layer: Represents the position of each character output by the context encoding layer. Mapped to emission fractions in the BMES label space:

[0165]

[0166] in , (BMES has 4 tags in total) For position The emission fraction vectors corresponding to the four labels.

[0167] Sequence structure modeling layer based on conditional random fields: defining the label transition score matrix. ,in Indicates from the label Transferred to The score.

[0168] For the input sequence and label sequence Its conditional random field score is:

[0169]

[0170] Label transition matrix During training, it learns legal transition patterns between labels automatically. For example, the model learns that a B label is usually followed by an M or E label (rather than an S label), and an E label is usually followed by a B or S label, etc.

[0171] Sequence decoding layer based on Viterbi algorithm: In the inference phase, the Viterbi algorithm is used to search for the optimal label sequence.

[0172]

[0173] The Viterbi algorithm efficiently computes the optimal path through dynamic programming, avoiding the exponential complexity of exhaustively searching all possible label sequences.

[0174] Specifically, since the sequence labeling word segmentation model and the masked language model share the basic structure of the input representation layer and the context encoding layer, the input representation layer and the context encoding layer process the data in the same way during the training process of the sequence labeling word segmentation model. Throughout the training process, the word segmentation model is trained on the literature word segmentation dataset, and the training objective is to maximize the conditional log-likelihood.

[0175]

[0176] in The number of training samples, Given the set of model parameters, the conditional probabilities are calculated using the forward-backward algorithm of the conditional random field:

[0177]

[0178] Taking the excerpt from the Zuo Zhuan mentioned in the previous steps as an example:

[0179] Input sequence:

[0180] Real label sequence:

[0181] The model calculates the emission fraction vector for each location. Simultaneously learn the transition matrix During training, the model parameters are updated using the backpropagation algorithm, thus adjusting the conditional probability of the true label sequence. maximize. .

[0182] S104: Model parameter fusion.

[0183] This step mainly provides a method for fusing masked language model parameters and word segmentation model parameters. Through this method, a new fusion model is obtained, which is used to improve the representation ability and generalization performance of the word segmentation model by introducing the general language knowledge learned by the masked language model while retaining the structure of the word segmentation task.

[0184] Specifically, the first pre-trained model parameters obtained from the mask language model training and the second model parameters obtained from the word segmentation model training are acquired in the aforementioned steps. Then, the model structure corresponding to the word segmentation model is constructed according to the word segmentation model's configuration file, and the model is initialized without loading any pre-trained weights.

[0185] Specifically, firstly, the parameters of the first pre-trained model obtained from the training model in the aforementioned steps are acquired. Second model parameters .

[0186] During the model parameter fusion process, the parameter names in the word segmentation model are first standardized to ensure that their naming is consistent with that of the mask language model parameters in the encoding and embedding layers. For word-level embedding parameters in the word segmentation model, when the vocabulary size is smaller than the target Qin Jian vocabulary size, the word-level embedding matrix is ​​expanded, and the original embedding parameters are copied to the corresponding positions in the expanded embedding matrix. The remaining newly added positions are generated using a random initialization method.

[0187] Furthermore, the parameters of the word segmentation model and the masking language model are fused parameter by parameter. When a parameter belongs to the embedding layer or the context encoding layer, a weighted method is used to fuse the corresponding parameters, and the fusion result is determined by the following formula:

[0188]

[0189] in, This represents the corresponding model parameters in the masked language model. This represents the corresponding model parameters in the word segmentation model. To incorporate the weighting coefficients, in this embodiment, the value is taken as follows: .

[0190] It should be noted that when the dimensions of the corresponding parameters are not completely consistent, only the parameter submatrices within the common dimension range are weighted and fused, while the remaining parameters retain their original values ​​from the word segmentation model.

[0191] For parameter layers that are only related to the word segmentation task, including the label mapping layer and the conditional random field sequence modeling layer, the parameters are not included in the weighted calculation during the parameter fusion process. Instead, the original parameters obtained from the word segmentation model training are directly retained to ensure the structural integrity and prediction consistency of the word segmentation task.

[0192] In this embodiment of the invention, the shared parameter set corresponding to the semantic functions in the two models is first identified. Shared parameter set The results are summarized in Table 4.

[0193]

[0194] Table 4

[0195] For each shared parameter, perform a weighted fusion:

[0196]

[0197] In this embodiment, the fusion weight coefficient is taken. That is, the masking language model parameters account for 70% of the weight, and the word segmentation model parameters account for 30% of the weight, which is higher. The value helps preserve the domain-specific representations learned by the masked language model on the original text of the bamboo slips.

[0198] Taking a character embedding matrix as an example:

[0199]

[0200] In practice, when and When the dimensions are not completely consistent (e.g., different character vocabulary sizes), weighted fusion is performed only on the parameter submatrices within the common dimension range. Assume... ,but:

[0201]

[0202] Parts beyond the public dimension (such as) to (line) keep The original value is generated using a random initialization method.

[0203] Furthermore, the parameters of the word segmentation task-specific layer are not fused; instead, the original values ​​of the second model parameters are directly retained to ensure the structural integrity and prediction consistency of the word segmentation task. For example:

[0204] Tag mapping layer parameters: ,

[0205] Conditional random field transition matrix: .

[0206] S105: Transfer initialization model retraining.

[0207] The sequence labeling word segmentation model is initialized based on the parameters of the fusion model, and then retrained using the source domain dataset to adapt the fused parameters to the target distribution of the word segmentation task, thus constructing a transfer-initialized word segmentation model.

[0208] Figure 6 This is a schematic diagram of the migration initialization word segmentation model structure according to an embodiment of the present invention. Accordingly, the migration initialization word segmentation model structure of this embodiment is the same as that of the sequence labeling word segmentation model, except that the configuration parameters have changed. For easy distinction, the migration initialization word segmentation model of this embodiment includes a third input representation layer, a third context encoding layer, a third label mapping layer, a third sequence structure modeling layer, and a third sequence decoding layer connected in sequence.

[0209] Specifically, after parameter fusion is completed, this embodiment of the invention loads the fused model parameters into the word segmentation model structure and retrains it based on the source domain dataset. This retraining process uses the same training configuration and optimization objective as the word segmentation model training. Through iterative training, the fused parameters are made to converge again on the word segmentation task, thereby restoring and strengthening the model's ability to recognize the distribution of word segmentation tags and sequence structure features, and finally constructing the transfer-initialized word segmentation model. S106: Target domain supervised fine-tuning of the transfer-initialized word segmentation model to obtain the bamboo and wooden slips text word segmentation model.

[0210] Figure 7 This is a schematic diagram of the word segmentation model for bamboo and wooden slips according to an embodiment of the present invention.

[0211] Based on the transfer initialization word segmentation model, supervised fine-tuning training is performed using a bamboo and wooden slips word segmentation annotation dataset to gradually adapt the model to the linguistic features of the bamboo and wooden slips text, resulting in the final bamboo and wooden slips text word segmentation model.

[0212] In an optional implementation, the target domain supervised fine-tuning is achieved through a gated fusion module. Accordingly, the structure of the bamboo and wooden slip text segmentation model in this embodiment of the invention needs to be based on the transfer initialization segmentation model. A gated fusion module is added between the third context encoding layer and the third label mapping layer. Due to the data source requirements of the gated fusion module, a bamboo and wooden slip feature extractor also needs to be connected to the gated fusion module. The bamboo and wooden slip feature extractor consists of a first input representation layer and a first context encoding layer.

[0213] Figure 8 This is a schematic diagram of the gated fusion module structure according to an embodiment of the present invention.

[0214] Specifically, the gated fusion module in this embodiment consists of three core sub-modules: an external feature injection interface, an adaptive gated unit, and a residual fusion layer. Its function is to dynamically inject deep semantic features from the bamboo and wooden slips domain into the general semantic features of the word segmentation base model, thereby achieving dynamic proportional weighted fusion at the feature level.

[0215] In actual operation, the gating fusion module receives two input features:

[0216] Main task characteristics The output originates from the real-time forward propagation of the aforementioned transfer-initialized word segmentation model (i.e., the word segmentation base model). Specifically, the input text sequence sequentially passes through the third input representation layer and the third context encoding layer. After processing by the layer coding unit, the hidden layer state sequence output by the last layer coding unit is extracted and denoted as... ,in The effective length of the current input sequence. For the embedded dimension.

[0217] auxiliary features : Derived from the static auxiliary feature matrix of the bamboo and wooden slip feature extractor Index reading. Before fine-tuning training, all text sequences from the bamboo and wooden slips word segmentation and annotation dataset are input into the frozen model one by one, offline forward inference is performed, the hidden layer state sequence of the encoder's last layer output is extracted, and the samples are stacked according to their index order in the dataset to construct a shape of The three-dimensional static tensor is denoted as the auxiliary feature matrix. ,in The total number of samples, This represents the maximum length of the sequence. During the gating fusion module's operation, the external feature injection interface, based on the sample index of the current input text sequence, extracts features from... Read the corresponding hidden layer state slices to obtain auxiliary features. .

[0218] The calculation process of the gating fusion module includes the following steps:

[0219] Feature concatenation, combining the main task features with auxiliary features Concatenate the features along the feature dimension to generate a joint feature representation:

[0220]

[0221] Gating coefficient generation: The adaptive gating unit performs linear mapping and activation on the joint features to generate a dynamic gating coefficient matrix. :

[0222]

[0223] in, The weight matrix is ​​a learnable linear mapping. For bias vectors, The sigmoid activation function is used to compress the values ​​of the gating coefficients to a minimum. Interval. Each element in the coefficient matrix. Each calculation is performed independently based on the local context state of the current input sequence, enabling adaptive adjustment for different character positions and feature dimensions.

[0224] Residual weighted fusion, the residual fusion layer is based on the main task characteristics As a baseline flow, with auxiliary features As a correction stream, through the gating coefficient The modified stream is element-wise weighted and injected into the reference stream using a residual connection method to obtain the fused features:

[0225]

[0226] in, This represents the Hadamard element-wise product. This calculation guarantees that when... Approaching At that time, the fused features degenerate into the general semantic features of the base model. ;when Approaching At that time, the characteristics of the bamboo and wooden slips domain participate in the correction with the greatest intensity, thereby achieving dynamic adaptive control of the degree of integration.

[0227] The output of the gated fusion module is a fusion feature matrix. This fusion feature simultaneously encodes the general ancient Chinese syntax structure and the domain-specific semantic information of bamboo and wooden slips, and preserves the integrity of the original main task features through residual connections, effectively avoiding the degradation of the general capabilities of the base model caused by the forced coverage of domain features.

[0228] Additionally, it should be noted that in the final bamboo and wooden slip text segmentation model, from the structural analysis of the model itself, the third input representation layer and the third context encoding layer are actually trained by fusing the parameters of the second input representation layer and the second context encoding layer, as well as the parameters of the first input representation layer and the first context encoding layer. The gating fusion module collects the outputs of the first context encoding layer and the third context encoding layer respectively and fuses them to obtain the fusion feature matrix. This involves a dual weighted fusion strategy for source domain data and target domain data. The first layer is parameter-level fixed-ratio weighted fusion, which enables the transfer initialization segmentation model to retain both the language features of the bamboo and wooden slip domain and the features of the general segmentation task. The second layer is feature-level dynamic ratio weighted fusion, where the gating mechanism calculates and generates fusion weights in real time according to the context state of the current input sequence. The two-stage weighted fusion forms a collaborative mechanism of static prior solidification and dynamic context correction. The parameter-level fixed weighting solves the cross-domain adaptation problem of the model's basic representation, while the feature-level dynamic weighting solves the fine-grained segmentation decision optimization problem under specific inputs.

[0229] Fusion features The data is then fed into the label mapping layer, projected onto the BMES label space through the linear mapping module, and finally decoded through the sequence structure modeling layer (conditional random field) and the sequence decoding layer (Viterbi algorithm).

[0230] Specifically, in this step, the fine-tuning training of the bamboo and wooden slip text segmentation model adopts the same training objective as step S103 (conditional random field negative log-likelihood loss), but uses bamboo and wooden slip segmentation annotation data instead of transmitted literature data. Through this stage of training, the model gradually adapts to the unique linguistic features of bamboo and wooden slip texts in terms of word usage habits, grammatical structure, and writing style. It should be noted that the implementation of the gated fusion module is an optional implementation method. When the bamboo and wooden slip text segmentation model does not have the gated fusion module, since a fixed proportion of parameter fusion calculation is performed during the composition process of the bamboo and wooden slip text segmentation model, the model can initially learn the relevant features of the source and target domains. Through fine-tuning training, the final bamboo and wooden slip text segmentation model can achieve relatively excellent segmentation performance in the target domain.

[0231] After the above steps, a final word segmentation model for bamboo and wooden slips text can be obtained.

[0232] S107: Word segmentation of bamboo and wooden slip texts based on a bamboo and wooden slip text segmentation model.

[0233] To facilitate understanding, a Qin Dynasty bamboo slip text that did not appear in the training set was used as test input to demonstrate the complete reasoning process of the final word segmentation model.

[0234] The content of the test input text is: "Ling Teng bu sheng zhu shang fu gan bi", this sentence is from *Yu Shu* (Statements) in the Shuihudi Qin bamboo slips, which means that the county magistrate Teng is not competent (for his duties), and does not dare to evade punishment and reward. The specific word segmentation process is as follows:

[0235] S301: Character-level serialization.

[0236] Let the input character sequence be , which respectively represent [xian, ling, Teng, bu, sheng, zhu, shang, fu, gan, bi]. Special tokens <[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]> and [SEP] are introduced at the start position and end position of the sequence respectively to obtain the complete input sequence: .

[0237] The current sequence length is 12, which is less than the maximum length of 128, so no truncation is required. For sequences with a length less than 128, [PAD] tokens are appended at the end to fill the sequence to a length of 128.

[0238] S302: Input representation calculation.

[0239] The input representation layer processes each character and word fragment in . Assume that word fragments such as "county magistrate", "incompetent", "punishment and reward", and "dare to evade" have been included in the dictionary, then the characters at the corresponding positions will obtain the contribution of the word fragment embeddings. Taking the character (ling, located at the 2nd position of the original sequence, belonging to the word fragment "county magistrate") as an example, its processing steps include:

[0240] Character embedding: ;

[0241] Word fragment embedding: ;

[0242] Position embedding: ;

[0243] Final input representation: .

[0244] This design ensures that the model can simultaneously perceive the character's own information, word fragment structure information and sequence position information.

[0245] S303: Context encoding.

[0246] The input representation is processed by layer encoding units. During the encoding process, the model captures the following semantic relationships through the self-attention mechanism:

[0247] The strong dependency relationship between "ling" and "xian" (forming the official position name "county magistrate");

[0248] The collocation relationship between "bu" and "sheng" ("busheng" is a fixed collocation, meaning "unable to bear / be competent for");

[0249] The coordinate relationship between "zhu" and "shang" ("zhushang" is a coordinate structure, meaning "punishment and reward");

[0250] The verb-object relationship between "gan" and "bi" ("ganbi" means "dare to evade").

[0251] After layers of context encoding, the hidden state sequence of the last layer is output . The hidden state of the last layer is the main task feature, denoted as , which simultaneously encodes the character's own information, the context of surrounding characters, and the semantic information of potential word segments, and provides input for subsequent feature fusion and word segmentation label prediction.

[0252] S304: Gated fusion calculation.

[0253] In the case of adopting the residual gated fusion word segmentation model, the external feature injection interface reads the corresponding auxiliary features in the bamboo slip domain from the auxiliary feature matrix according to the index of the current test sample in the dataset . The adaptive gating unit processes and to perform feature concatenation and linear mapping to generate gating coefficients :

[0254]

[0255] Herein, is a learnable linear mapping weight matrix, is a bias vector, is the Sigmoid activation function.

[0256] Taking each character in the test input as an example, the representative values of its gating coefficients in the key semantic dimension are shown in Table 5.

[0257]

[0258] Table 5

[0259] The residual fusion layer performs element-wise weighting and residual injection:

[0260]

[0261] Herein, ​denotes the Hadamard element-wise product. This calculation significantly enhances the fused features of characters such as "county", "decree", "not", "surpass", "punish", "reward", "dare", and "exile" on specific semantic dimensions such as legal terms on bamboo slips and official position systems; while for characters with strong versatility such as "teng" and "fu", the fused features mainly retain the general semantic representation of the base model, avoiding the introduction of unnecessary domain noise. After gated fusion, the fused features simultaneously encode the general ancient Chinese syntactic structure and the domain-specific semantic information of bamboo slips, effectively preserve the integrity of the original main task features through residual connection, and provide a richer contextual discriminant basis for subsequent label mapping.

[0262] S305: Label mapping.

[0263] The label mapping layer maps the fused features to the emission scores in the BMES label space:

[0264]

[0265] wherein, , , is the size of the BMES label set. Assume that the emission score matrix calculated by the model (each row corresponds to a character, and each column corresponds to four labels B / M / E / S) is shown in Table 6.

[0266]

[0267] Table 6

[0268] S306: Conditional random field decoding.

[0269] The sequence structure modeling layer models the label transition relationship through a conditional random field, and its transition matrix constrains the validity of the label sequence. The Viterbi algorithm searches for the optimal label path under the joint action of the above emission scores and transition scores.

[0270] Assume that the learned transition matrix makes the model prefer the following transition patterns:

[0271] After B, it tends to transition to E (for two-character words) or M (for multi-character words), after E, it tends to transition to B (start of a new word) or S (single-character word), and after S, it tends to transition to B or S.

[0272] Optimal label sequence and word segmentation result

[0273] The optimal label sequence decoded by the Viterbi algorithm is shown in Table 7.

[0274]

[0275] Table 7

[0276] S307: Word segmentation.

[0277] The final word segmentation result is: 县令 / 腾 / 不胜 / 诛赏 / 弗 / 敢辟.

[0278] Among them, "Xian Ling" (county-level administrative chief), "Bu Sheng" (incompetent), and "Zhu Shang" (punishment and reward) are common vocabularies in bamboo slip documents, and "Gan Bi" (dare to evade / avoid) is a fixed collocation in ancient Chinese. This word segmentation result conforms to the grammatical rules of ancient Chinese and the language habits of bamboo slip documents.

[0279] In summary, the present invention provides a text word segmentation method for unearthed bamboo slip documents based on transfer learning. This method adopts a multi-stage training strategy that performs masked language model pre-training on the original bamboo slip text, trains the word segmentation model on document word segmentation data, realizes knowledge transfer through parameter fusion, and then undergoes target domain fine-tuning to adapt to the linguistic features of bamboo slips, which effectively improves the accuracy and generalization ability of bamboo slip text word segmentation under the condition of limited labeled corpus. This method is not only applicable to the word segmentation of Qin bamboo slip texts, but also can be extended to the automatic word segmentation processing of other unearthed ancient Chinese documents, providing strong technical support for digital humanities research and digital sorting of unearthed documents.

[0280] The above descriptions are merely preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present application shall all be included in the protection scope of the present application.

Claims

1. A word segmentation method for unearthed bamboo and wooden slips documents based on transfer learning, characterized in that, Includes the following steps: Construct a target domain dataset and a source domain dataset. The target domain dataset includes an un-tagged original text dataset of bamboo and wooden slips and a tagged dataset of bamboo and wooden slips with segmentation annotation. The source domain dataset includes a literature segmentation corpus dataset. Both the literature segmentation corpus dataset and the bamboo and wooden slips tagged dataset use the same annotation standard for character-level segmentation annotation. Based on the original text dataset of the bamboo and wooden slips, training samples for a masked language model are constructed using a differential masking strategy. The differential masking strategy includes assigning a mask prediction participation probability to the character according to the character's interpretation state. The mask language model is pre-trained based on the training samples of the mask language model to obtain the first pre-trained model parameters. The mask language model includes a first input representation layer, a first context encoding layer and a first prediction layer connected in sequence. The sequence labeling word segmentation model is trained on the document word segmentation corpus dataset to obtain the second model parameters. The sequence labeling word segmentation model includes an input representation layer, a context encoding layer, a label mapping layer, a sequence structure modeling layer, and a sequence decoding layer connected in sequence. The shared parameters in the first pre-trained model parameters and the second model parameters are weighted and fused, while keeping the parameters of the label mapping layer and the sequence structure modeling layer unchanged, to obtain the fused model parameters. The sequence labeling word segmentation model is initialized based on the parameters of the fusion model, and then retrained based on the source domain dataset to obtain the transfer initialization word segmentation model; The transfer initialization word segmentation model was trained under supervised fine-tuning based on the aforementioned bamboo and wooden slips word segmentation annotation dataset to obtain the final bamboo and wooden slips text word segmentation model; The step of constructing training samples for the masked language model using a differential masking strategy includes: performing character-level serialization processing on the original text of the bamboo slips, wherein the character-level serialization processing includes treating each Chinese character as the smallest processing unit and constructing a character sequence according to the original text order. ,in, The length is the sequence length; start markers are inserted at the beginning and end positions of the character sequence, respectively. and end marker and the length exceeds the preset maximum length The sequence is truncated, and sequences that are not long enough are padded; each character in the character sequence is processed. Identify its interpretation status ,in, This indicates that the character is being interpreted. Indicates the interpretation of undefined characters. Indicates missing characters; randomly selects a set of mask positions from the character sequence. For each selected mask position ,like Then that position will always be used as the prediction target in training, if Then, based on probability Use it as the prediction target in training, with probability It is used as context input and not involved in the prediction loss calculation; the mask position set, the corresponding real character label and the context character sequence are jointly encoded to construct the mask language model training sample.

2. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 1, characterized in that, The construction process of the original text dataset and the word segmentation and annotation dataset of the bamboo and wooden slips includes: Extracting authentic text from bamboo and wooden slips that have been deciphered; The real text is preprocessed; The preprocessed real text is organized into a character-level sequence format, with each Chinese character occupying one data record and adjacent sentences separated by blank lines, thus constructing the original text dataset of the bamboo and wooden slips. The text in the original bamboo and wooden slips text dataset was manually segmented and labeled.

3. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 1, characterized in that, The input representation layer includes a character embedding module, a word fragment embedding module, and a position embedding module. The processing of the input sequence by the input representation layer includes: For each character in the input sequence The character embedding module maps it to a character embedding vector. ,in, For the embedded dimension; Identify multi-character word fragments in the character sequence Each word fragment Each segment is assigned a unique identifier, which is then mapped to a segment embedding vector by the segment embedding module. ; The position embedding module is used for each character position Generate position embedding vectors ; By superimposing the above embedding vectors, we obtain the character. The final input is represented as follows: ； in, Indicates the character The embedding vectors of all word segments are summed.

4. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 3, characterized in that, The context coding layer includes a composition of The sequence modeling module is composed of stacked layer coding units. Each layer coding unit includes a multi-head self-attention module, a feedforward neural network module, and a residual connection and layer normalization module. The calculation process of the multi-head self-attention module is as follows: For the Layer input Generate the query matrix, key matrix, and value matrix: ； in, The weight matrix is ​​a learnable matrix. , For the number of heads; Calculate the scaled dot product attention output: ； in, This is a fill mask matrix used to mask the fill positions; This is the prediction mask matrix for the masked language model, used to mask the positions to be predicted; The feedforward neural network module performs an independent nonlinear transformation on the vector at each position: ； in, , , , , For activation function, For the intermediate layer dimensions of the feedforward network; The output of the residual connection and layer normalization module is: ； go through After processing by the layer coding unit, the output is... As a context-encoded representation.

5. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 4, characterized in that, The working process of the label mapping layer, sequence structure modeling layer, and sequence decoding layer of the sequence labeling word segmentation model is as follows: The label mapping layer transforms the representation vector of each character position output by the context encoding layer through a linear transformation. Mapped to emission score vectors in the word segmentation tag space: ； in, , For learnable parameters, The size of the BMES tag set; The sequence structure modeling layer adopts a conditional random field model, defining a given input sequence. Conditional label sequence The score is: ； in, The label transition score matrix, Indicates from the label Move to label The score; The sequence decoding layer employs the Viterbi algorithm to search for the optimal label path that maximizes the sequence score under a conditional random field model. 。 6. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 1, characterized in that, The process of weighted fusion of shared parameters includes: Identify the shared parameter set corresponding to the parameter names and semantic functions in the masked language model and the sequence labeling word segmentation model. The shared parameters include character embedding parameters, word fragment embedding parameters, position embedding parameters, weight matrices and bias parameters of each coding unit in the context coding layer; For each shared parameter The weighted fusion is performed according to the following formula: ； in, These are the corresponding parameters in the parameters of the first pre-trained model. These are the corresponding parameters in the second model parameters. For fusion weighting coefficients.

7. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 6, characterized in that, The optimization strategies employed during the supervised fine-tuning training process include: Employ a linear decaying learning rate scheduling strategy with warmup: ； in, Peak learning rate, For linear preheating steps, Total training steps; Dropout regularization is added to the multi-head self-attention module and feedforward neural network module of the context encoding layer to increase probability. Randomly discard neuron outputs; A weight decay strategy is adopted, and an L2 regularization term is introduced into the loss function. ,in, For the set of model parameters, is the regularization coefficient.

8. The text segmentation method for unearthed bamboo and wooden slips based on transfer learning according to claim 7, characterized in that, The sequence labeling word segmentation model is retrained, and the transfer-initialized word segmentation model is subjected to supervised fine-tuning training. The training objective of the word segmentation model is to minimize the negative log-likelihood loss function. ； in, The number of training samples. For the first A sequence of input characters, For the corresponding real label sequence, For model parameters, conditional probabilities The following was calculated using the forward-backward algorithm of the conditional random field model: ； in, Represents a given input sequence The set of all possible label sequences under the given conditions.

Images