A method, apparatus and storage medium for screening meaning-shift subwords

Abstract

This invention discloses a method, apparatus, and storage medium for filtering meaning-shifted sub-words. The method involves statistically identifying source and target domain word groups based on shared sub-words, comparing these source and target domain word groups to determine if the meaning of the shared sub-words has shifted, using a pre-trained language model to comprehensively represent and encode the source word groups, and utilizing cosine similarity to quantify the similarity between source word groups. This effectively filters out shared sub-words with meaning shifts, optimizes the target domain word representation, and thus improves the performance of the target domain model.

Description

Technical Field

[0001] This invention relates to the field of machine translation technology in natural language processing, and in particular to a method, apparatus, and storage medium for filtering semantically offset subwords. Background Technology

[0002] In recent years, with the rapid development of computer and machine learning technologies, deep learning technology has been widely applied in various fields, playing a significant role, including machine translation. Deep learning models can learn sufficient data features from high-resource domains, thus exhibiting excellent performance. However, in contrast, deep learning models often struggle to achieve comparable performance in low-resource domains, and therefore cannot generalize well to languages ​​and domains with limited training data. To enable machines to better understand natural sentences in low-resource languages ​​and domains, it is necessary to overcome the challenge of data scarcity. Therefore, cross-domain transfer learning methods have been proposed, which learn task knowledge from large training samples in high-resource domains and transfer it to low-resource domains, thereby enhancing the training in low-resource domains with data from other domains. Typically, the domain from which the transferred information is obtained is defined as the source domain, and the domain from which the transferred information is obtained is defined as the target domain.

[0003] Existing cross-domain transfer learning methods typically employ a pre-training-fine-tuning paradigm, see appendix. Figure 1 As shown, the source and target domains share the same segmentation model and sub-vocabulary. Both the segmentation model and sub-vocabulary are derived from data in the source domain. The segmentation model is first pre-trained on the source domain data to obtain the original domain segmentation data and sub-vocabulary. Then, the segmentation model is directly trained on the target domain data to obtain the target domain segmentation data. However, the distribution of data differs across domains, and the appropriate segmentation models for each domain are also different. In the pre-training-fine-tuning paradigm, the source and target domain models share the same source-domain-based segmentation model. This leads to inappropriate segmentation of the target domain data, resulting in a mismatch between the source and target domain segmentations and limiting the fitting of the target domain model to the target domain data.

[0004] Another cross-domain transfer learning method employs a domain-specific segmentation paradigm. This paradigm addresses the segmentation mismatch between the source and target domains. It preserves the target domain's segmentation and sub-vocabulary, obtaining segmentation models and sub-vocabularies for each domain based on the source and target domain data, respectively. (See appendix) Figure 2As shown, the domain-specific segmentation paradigm first trains a source domain model based on source domain data, obtaining a source domain sub-vocabulary list through the source segmentation model. Simultaneously, it trains a target domain model based on target domain data, obtaining a target domain sub-vocabulary list through the target segmentation model. Then, using shared sub-vocabularies between the target and source domain sub-vocabularies as an intermediary bridge, it initializes the target domain sub-vocabulary representations based on the source domain sub-vocabulary representations using mapping and reconstruction methods. This completes the transfer of sub-vocabulary vector information from the source domain to the target domain. Since the model structure remains unchanged, the target domain model parameters can be directly initialized using the source domain model parameters, and then the model is trained on the target domain data based on the initialized target domain model. However, this paradigm assumes that shared sub-vocabularies have the same meaning in different domains. In reality, because the source and target domains use their respective segmentation models, the meanings of shared sub-vocabularies may differ significantly between the two domains. Shared sub-vocabularies with significant meaning differences are called meaning-shifted sub-vocabularies. When these meaning-shifted sub-vocabularies participate in the initialization of target domain sub-vocabularies as an intermediary bridge, they affect the initialization effect and consequently the model's performance in the target domain.

[0005] Therefore, a method for filtering semantically shifted sub-words is needed to eliminate the influence of shared sub-words with semantic shifts in domain-specific segmentation paradigms on the initialization of target domain sub-word representations. Summary of the Invention

[0006] To overcome the above-mentioned shortcomings, the purpose of this invention is to provide a method for filtering meaning-shifting sub-words, which can filter meaning-shifting sub-words based on shared words, thereby improving the performance of the model in the target domain.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a method for filtering meaning-shifting subwords, the method comprising:

[0008] Based on source domain data and target domain data, respectively train source domain BPE segmentation models and target domain BPE segmentation models corresponding to their respective domains. The data of each domain is segmented by the BPE segmentation models of their respective domains to obtain source domain model sub-vocabularies and target domain sub-vocabularies respectively.

[0009] Select sub-words that appear in both the source domain sub-word list and the target domain sub-word list to form a shared sub-word list containing multiple sub-words;

[0010] The source words of each shared sub-word in the shared sub-word table in the source domain data are counted to form a source domain source word group; the source words of each shared sub-word in the shared sub-word table in the target domain data are counted to form a target domain source word group.

[0011] Based on a pre-trained language model, each source word in the source domain and the target domain source word are encoded. Then, the encoded representations are weighted and averaged based on the frequency of occurrence of each source word to obtain a comprehensive encoded representation of the shared subword in the source domain and the target domain source word.

[0012] For each shared subword, calculate the cosine similarity between the comprehensive encoded representation of its source domain term and the comprehensive encoded representation of its target domain term, and compare the cosine similarity with a set threshold. When a cosine similarity is less than the set threshold, the shared subword is identified as a meaning-shifted subword.

[0013] The beneficial effects of this invention are as follows: Addressing the issue that shared words with shifted meanings can affect the initialization of target domain word representations, this invention provides a method for filtering words with shifted meanings based on source phrase similarity. This method compares source phrases from the source and target domains to determine whether the meaning of shared words has shifted. A pre-trained language model is used to comprehensively encode the source phrases, and cosine similarity is used to quantify the similarity between source phrases. This effectively filters out shared words with shifted meanings, optimizes the target domain word representation, and thus improves the performance of the target domain model.

[0014] Furthermore, the step of segmenting data in their respective domains using BPE segmentation models to obtain source domain model sub-vocabularies and target domain sub-vocabularies specifically includes:

[0015] The BPE segmentation model counts the occurrences of word pairs in the corresponding domain dataset, and the word pairs that occur most frequently are recorded and merged into a table.

[0016] The BPE segmentation model segments words one by one in the corresponding domain dataset. When segmenting a word, it first converts the word into a character sequence, merges character pairs that conform to the rules in the merging table, until no further merging is possible and a sub-word sequence containing multiple sub-words is formed. At this point, the segmentation of the word is complete.

[0017] All words in the source domain data and the target domain data are segmented to form the source domain model sub-vocabulary and the target domain sub-vocabulary, respectively.

[0018] Furthermore, the source domain source phrases and target domain source phrases record the corresponding domain source words and their frequency of occurrence. The statistical methods for the source domain source phrases and target domain source phrases include:

[0019] The statistical source domain data and the target domain data show each word and its frequency of occurrence;

[0020] Traverse the word lists of the source domain data and the target domain data. If the sub-word sequence after the current word is segmented contains the shared sub-word, it means that the word is a source word of the shared sub-word, and the frequency of the source word is the frequency of the word.

[0021] Furthermore, the pre-trained language model is the BERT model. The BERT model can segment each input source word into B BERT subwords. When synthesizing source domain word groups, the source words from the source domain word groups are input into the BERT model; when synthesizing target domain word groups, the source words from the target domain word groups are input into the BERT model. Each subword returns a subword vector e. i， Of course, other pre-trained language models can also be used.

[0022] Furthermore, the smaller the cosine similarity of a shared subword, the greater the semantic shift that occurs in the shared subword.

[0023] Furthermore, the set threshold is greater than or equal to 0.9 and less than or equal to 1. A threshold greater than or equal to 0.9 satisfies the filtering requirements.

[0024] Furthermore, when a cosine similarity is greater than or equal to a set threshold, the shared subword is retained as a shared subword; the larger the set threshold is, the fewer the number of shared subwords are retained, but the smaller the semantic shift of the retained shared subwords.

[0025] Furthermore, after filtering out the meaning-shifted sub-words, the target domain sub-word table is initialized, and the meaning-shifted sub-words are treated as unknown sub-words in the target domain during initialization.

[0026] The present invention also discloses a device for filtering semantically offset subwords, which can filter shared subwords and select semantically offset subwords, including:

[0027] The preprocessing module is used to train source domain BPE segmentation models and target domain BPE segmentation models corresponding to their respective domains based on source domain data and target domain data. Then, the data of each domain is segmented by the BPE segmentation models of each domain to obtain source domain model sub-vocabularies and target domain sub-vocabularies respectively.

[0028] The filtering module filters out sub-words that appear in both the source domain sub-word list and the target domain sub-word list to form a shared sub-word list. It also counts the source words of each shared sub-word in the shared sub-word list in the source domain data and the target domain data to form the source word groups of the corresponding domains, thus forming source word groups of the source domain and source word groups of the target domain.

[0029] The encoding module stores a pre-trained language model. Based on the pre-trained language model, it encodes each source word in the source domain and the target domain source word, respectively. Then, it performs a weighted average of the encodings based on the frequency of occurrence of each source word to obtain a comprehensive encoding representation of the shared subword in the source domain and the target domain source word.

[0030] The filtering module is used to calculate the cosine similarity between the comprehensive encoded representation of the source domain terminology and the comprehensive encoded representation of the target domain terminology for each shared subword, and compare the cosine similarity with a set threshold. When a cosine similarity is less than the set threshold, the shared subword is identified as a meaning-shifted subword.

[0031] The present invention also discloses a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method for filtering meaning-shifted subwords. Attached Figure Description

[0032] Figure 1 This is a flowchart of the pre-training-fine-tuning paradigm in the existing technology;

[0033] Figure 2 A flowchart illustrating a domain-specific segmentation paradigm in existing technologies;

[0034] Figure 3 This is a flowchart of a method according to an embodiment of the present invention;

[0035] Figure 4 This is a structural block diagram of a device according to an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram illustrating the word segmentation using the BPE segmentation model according to an embodiment of the present invention. Detailed Implementation

[0037] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0038] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0039] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0040] The present invention provides a method for filtering meaning-shifted sub-words, which can filter shared sub-words and select meaning-shifted sub-words. In the subsequent step of initializing the target domain sub-word representation, these shared sub-words with meaning shifts are treated as unknown sub-words in the target domain, thereby improving the performance of the obtained target domain model.

[0041] See appendix Figure 3 As shown, the method includes the following steps:

[0042] Step 1: Train source domain BPE segmentation models and target domain BPE segmentation models corresponding to their respective domains based on source domain data and target domain data, respectively. Then, segment the data of each domain using the BPE segmentation models of their respective domains to obtain source domain model sub-vocabularies and target domain sub-vocabularies, respectively.

[0043] The source domain and target domain are two datasets from different domains. The source domain is the domain from which the transferred information is obtained, and the target domain is the domain from which the transferred information is acquired. The sub-vocabulary contains multiple segmented sub-word fragments.

[0044] Word segmentation is the most fundamental step in natural language processing tasks. It involves segmenting words into sub-words using a word segmentation algorithm, and then using these sub-words as the basic unit of processing before inputting them into the model. Currently, the most widely used word segmentation algorithm is Byte-Pair Encoder (BPE). Therefore, in this embodiment, a BPE segmentation model is obtained based on the BPE algorithm. The BPE algorithm segments words according to the frequency of sub-word pairs in the training data. Due to the data-driven nature of the BPE algorithm, the resulting BPE segmentation model will differ for different domain datasets, leading to different sub-vocabularies across different domains.

[0045] In this embodiment, the data of each domain is segmented using the BPE segmentation model of their respective domains, and source domain model sub-vocabularies and target domain sub-vocabularies are obtained respectively, specifically including:

[0046] The S11 and BPE segmentation models count the occurrences of word pairs in the corresponding domain dataset, and the word pairs that occur most frequently are recorded and merged into a table.

[0047] The S12 and BPE segmentation models segment words one by one in the corresponding domain dataset. When segmenting a word, the word is first converted into a character sequence. Character pairs that conform to the rules in the merging table are merged until they can no longer be merged, forming a sub-word sequence containing multiple sub-words. At this point, the segmentation of the word is complete.

[0048] As shown in Figure 5, taking the word "apple" as an example, when the BPE segmentation model segments "apple", it first segments "apple" into five characters: a, p, p, l, and e, forming a character sequence. Then, it compares the sequence with the rules in the merging table and merges the character pairs that match the rules in the merging table until no further merging is possible, resulting in the two sub-words "app" and "le".

[0049] S13. Segment all words in the source domain data and the target domain data to form the source domain model sub-vocabulary and the target domain sub-vocabulary, respectively.

[0050] Step 2: Select sub-words that appear in both the source domain sub-word list and the target domain sub-word list to form a shared sub-word list, which contains multiple sub-words.

[0051] Step 3: Calculate the source words of each shared sub-word in the shared sub-word table in the source domain data and target domain data to form corresponding domain source word groups, i.e., source domain source word groups and target domain source word groups. Each source word group records the corresponding domain source word and its frequency of occurrence.

[0052] The source word refers to the word that is contained in the sequence of subwords after a complete word is segmented. For example, if the word "direction" is segmented into the three subword sequences "dir@@ ec@@ tion" by the BPE segmentation model, then the word "direction" is the source word of the three subwords "dir@@", "ec@@", and "tion" (the symbol "@@" indicates that the subword is not the last word of the word).

[0053] Step three specifically includes:

[0054] S31. Statistical source domain data and target domain data for each word and its frequency of occurrence;

[0055] S32. Traverse the word lists of the source domain data and the target domain data. If the sub-word sequence after the current word is segmented contains the shared sub-word, it indicates that the word is a source word of the shared sub-word, and the frequency of the source word is the frequency of the word.

[0056] S33. Record all source words that form the source word group of this shared subword.

[0057] Step 4: Based on a pre-trained language model, encode each source word in the source domain and the target domain source word groups respectively. Then, based on the frequency of occurrence of each source word, perform a weighted average of the encoded representations to obtain a comprehensive encoded representation of the shared subword in the source domain and the target domain source word groups respectively.

[0058] In one embodiment, the pre-trained language model uses the BERT (Bidirectional Encoder Representations from Transformer) model. The BERT model can segment each input source word into B BERT subwords. When synthesizing source words from the source domain, the source words from the source domain are input into the BERT model; when synthesizing target domain source words, the source words from the target domain are input into the BERT model. Each subword returns a subword vector e. i Then the encoding of each source word represents Of course, other pre-trained language models can also be used, such as the Roberta model, as long as it is a pre-trained model.

[0059] The i-th shared subword t i， The comprehensive encoding of source phrases in the source domain is represented as follows: = * ,in = c Si For this purpose, the frequency of occurrence of a source word in the source domain source phrase is given, where m is the number of corresponding source word types in the source domain source phrase. This is the encoded representation of each source word in the source domain source phrase.

[0060] The i-th shared subword t i， The comprehensive encoding of source phrases in the source domain is represented as follows: = * ,in = c Ti Therefore, the frequency of occurrence of a source word in the target domain source word group for sharing sub-words, where k is the number of corresponding source word types in the target domain source word group. This is the encoded representation of each source word in the target domain source phrase.

[0061] Step 5: For each shared subword, calculate the cosine similarity between the comprehensive encoding representation of its source domain term and the comprehensive encoding representation of its target domain term, and compare the cosine similarity with a set threshold. When a cosine similarity is less than the set threshold, the shared subword is identified as a meaning-shifted subword.

[0062] Among them, the i-th shared sub-word t i cosine similarity .

[0063] The lower the cosine similarity, the greater the semantic shift of the shared sub-word. A threshold between 0.0 and 1.0 is set. The selection of this threshold requires considering both the quality and quantity of the sub-words to be retained. A higher threshold results in smaller semantic shifts and higher quality shared sub-words, but also a smaller number of retained sub-words.

[0064] In one embodiment, a threshold of 0.9 or higher is set, which satisfies the filtering requirements. In the subsequent initialization step of the target domain sub-word representation, these sub-words identified as having meaning offsets are treated as unknown sub-words in the target domain. When the cosine similarity is greater than the set threshold, this shared sub-word is retained in the shared sub-word table.

[0065] In one embodiment, the source domain can be selected as the general news domain, the target domain as the specific medical domain, and the text language of the model input can be English. This will result in a news BPE segmentation model and a medical BPE segmentation model. The news BPE segmentation model and the medical BPE segmentation model are used to segment the data in the general news domain and the specific medical domain, respectively, to obtain the news domain sub-vocabulary and the medical domain sub-vocabulary.

[0066] Subwords that appear in both the news and medical sub-vocabularies are selected to form a shared sub-vocabulary. This shared sub-vocabulary contains n subwords and is represented as follows: .

[0067] For each shared term in the shared terminology list, we calculate the frequency of its source phrases and the occurrence frequency of each source word in both the general news domain and the specific medical domain. Let the i-th shared term be t. i The source phrase in the general news field is: The source phrase in a specific medical field is In this context, S and T represent the general news domain and the specific medical domain, respectively; w represents the source term; c represents the frequency of occurrence in the training set for their respective domains; and m represents the frequency of occurrence. ti and k ti Representing shared subwords t i The number of source terms in general news and specific medical fields.

[0068] Next, we compute a comprehensive encoding representation for a shared subword in both general news domain source phrases and specific medical domain source phrases, based on the BERT multilingual pre-trained language model. First, the BERT model segments the input source word into B BERT subword fragments (b1, b2...b...). BThese BERT sub-word fragments are input into the BERT model to obtain the BERT sub-word representation of the source word, and then the sub-word representations are averaged to obtain the word representation of the source word. Then, the encoding is based on the frequency of occurrence of each source word. Weighted averages are performed to represent the comprehensive encoding of the shared sub-word in the source phrase of the news domain. Comprehensive encoding of phrases from specific medical fields .

[0069] Finally, based on the comprehensive coding of source phrases in the news field... Comprehensive encoding of phrases from specific medical fields Calculate the i-th shared subword t i cosine similarity Cosine similarity Compared with a set threshold θ (θ=0.9), shared subwords with similarity less than θ are used as meaning-shifted subwords.

[0070] See appendix Figure 4 As shown, the present invention also discloses a device for filtering meaning-shifted subwords, which can filter shared subwords and filter out meaning-shifted subwords, including a preprocessing module, a filtering module, an encoding module and a filtering module.

[0071] The preprocessing module is used to train source domain BPE segmentation models and target domain BPE segmentation models corresponding to their respective domains based on source domain data and target domain data. Then, the data of each domain is segmented by the BPE segmentation models of each domain to obtain source domain model sub-vocabularies and target domain sub-vocabularies respectively.

[0072] The filtering module selects subwords that appear in both the source and target domain subword lists to form a shared subword list. It then calculates the source words for each shared subword in the source and target domain data, forming corresponding source word groups. Each source word group records the source words in its corresponding domain and their frequency of occurrence.

[0073] The encoding module stores a pre-trained language model. Based on the pre-trained language model, it encodes each source word in the source domain and the target domain source word groups respectively. Then, it performs a weighted average of the encodings based on the frequency of occurrence of each source word to obtain a comprehensive encoding representation of the shared subword in the source domain and the target domain source word groups.

[0074] The filtering module is used to calculate the cosine similarity between the comprehensive encoded representation of the source domain terminology and the comprehensive encoded representation of the target domain terminology for each shared subword, and compare the cosine similarity with a set threshold. When a cosine similarity is less than the set threshold, the shared subword is identified as a meaning-shifted subword.

[0075] It should be noted that the above embodiments only illustrate the division of the above functional modules when implementing the device. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0076] This disclosure also provides a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the methods in the various method embodiments described above.

[0077] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0078] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and portable compact disc read-only memory (CD). ROM, digital multifunction disc (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or recessed protrusion structure storing instructions thereon, and any suitable combination thereof. The computer-readable storage medium used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0079] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0080] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method of filtering meaning-shift subwords, the method comprising: The method includes Based on source domain data and target domain data, respectively train source domain BPE segmentation models and target domain BPE segmentation models corresponding to their respective domains. The data of each domain is segmented by the BPE segmentation models of their respective domains to obtain source domain model sub-vocabularies and target domain sub-vocabularies respectively. Select sub-words that appear in both the source domain sub-word list and the target domain sub-word list to form a shared sub-word list containing multiple sub-words; The source words of each shared sub-word in the shared sub-word table in the source domain data are counted to form a source domain source word group; the source words of each shared sub-word in the shared sub-word table in the target domain data are counted to form a target domain source word group. Based on a pre-trained language model, each source word in the source domain and the target domain source word are encoded and represented. Then, the encoded representations are weighted and averaged based on the frequency of occurrence of each source word to obtain the comprehensive encoded representation of the shared subword in the source domain and the target domain source word. For each shared subword, calculate the cosine similarity between the comprehensive encoded representation of its source domain term and the comprehensive encoded representation of its target domain term, and compare the cosine similarity with a set threshold. When a cosine similarity is less than the set threshold, the shared subword is identified as a meaning-shifted subword. Specifically, the process of forming source domain source word groups by statistically analyzing the source words of each shared word in the shared sub-word table in the source domain data, and forming target domain source word groups by statistically analyzing the source words of each shared word in the shared sub-word table in the target domain data, includes: The statistical source domain data and the target domain data show each word and its frequency of occurrence; Traverse the word lists of the source domain data and the target domain data. If the sub-word sequence after the current word is segmented contains the shared sub-word, it means that the word is a source word of the shared sub-word, and the frequency of the source word is the frequency of the word. Record all source words that form the source word group for this shared subword.

2. The method of claim 1, wherein the subword is a prefix. The process of segmenting data in their respective domains using BPE segmentation models and obtaining source domain model sub-vocabularies and target domain sub-vocabularies specifically includes: The BPE segmentation model counts the occurrences of word pairs in the corresponding domain dataset, and the word pairs that occur most frequently are recorded and merged into a table. The BPE segmentation model segments words one by one in the corresponding domain dataset. When segmenting a word, it first converts the word into a character sequence, merges character pairs that conform to the rules in the merging table, until no further merging is possible and a sub-word sequence containing multiple sub-words is formed. At this point, the segmentation of the word is complete. All words in the source domain data and the target domain data are segmented to form the source domain model sub-vocabulary and the target domain sub-vocabulary, respectively.

3. The method for filtering meaning-shifting sub-words according to claim 1, characterized in that: The source domain source phrases and target domain source phrases record the source words of the corresponding domains and the frequency of occurrence of those source words.

4. The method for filtering meaning-shifting sub-words according to claim 1, characterized in that: The pre-trained language model is either a BERT model or a Roberta model.

5. The method for filtering meaning-shifting sub-words according to claim 1, characterized in that: The smaller the cosine similarity of a shared subword, the greater the semantic shift that the shared subword will experience.

6. The method for filtering meaning-shifting sub-words according to claim 1, characterized in that: The set threshold is greater than or equal to 0.9 and less than or equal to 1.

7. The method for filtering meaning-shifting sub-words according to claim 6, characterized in that: When a cosine similarity is greater than or equal to a set threshold, this shared subword is retained as a shared subword; The higher the threshold value is set, the fewer shared sub-words are retained, but the smaller the semantic shift of the retained shared sub-words.

8. The method for filtering meaning-shifting sub-words according to claim 1, characterized in that: After filtering out the meaning-off sub-words, the target domain sub-word table is initialized, and the meaning-off sub-words are treated as unknown sub-words in the target domain during initialization.

9. An apparatus for filtering semantically offset sub-words, characterized in that: include The preprocessing module is used to train source domain BPE segmentation models and target domain BPE segmentation models corresponding to their respective domains based on source domain data and target domain data. Then, the data of each domain is segmented by the BPE segmentation models of each domain to obtain source domain model sub-vocabularies and target domain sub-vocabularies respectively. The filtering module selects sub-words that appear in both the source and target domain sub-word lists to form a shared sub-word list. It then calculates the source words for each shared sub-word in the source and target domain data to form corresponding domain source word groups, thus creating source domain source word groups and target domain source word groups. Specifically, this includes: The statistical source domain data and the target domain data show each word and its frequency of occurrence; Traverse the word lists of the source domain data and the target domain data. If the sub-word sequence after the current word is segmented contains the shared sub-word, it means that the word is a source word of the shared sub-word, and the frequency of the source word is the frequency of the word. Record all source words that form the source word group for this shared subword; The encoding module stores a pre-trained language model. Based on the pre-trained language model, it encodes each source word in the source domain and the target domain source word, respectively. Then, it performs a weighted average of the encodings based on the frequency of occurrence of each source word to obtain a comprehensive encoding representation of the shared subword in the source domain and the target domain source word. The filtering module is used to calculate the cosine similarity between the comprehensive encoded representation of the source domain terminology and the comprehensive encoded representation of the target domain terminology for each shared subword, and compare the cosine similarity with a set threshold. When a cosine similarity is less than the set threshold, the shared subword is identified as a meaning-shifted subword.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

Images