Parallel sentence pair extraction method and device based on language similarity enhancement

Abstract

The present invention relates to a method and device for extracting parallel sentence pairs based on language similarity enhancement, and belongs to the field of natural language processing technology. In view of the problem that when the existing parallel sentence pair extraction method is applied to Chinese-Thai and Chinese-Lao tasks, the scarcity of Thai and Lao training data leads to the weak representation ability of the model for Thai and Lao and the poor extraction effect, the present invention is proposed. The present invention mainly includes three parts: data preprocessing integrating language enhancement features, training a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement, and extraction of Chinese-Lao-Thai multilingual parallel sentence pairs. According to these three functional modules, a parallel sentence pair extraction device based on language similarity enhancement is made, and parallel sentence pairs are extracted from Chinese-Thai and Chinese-Lao comparable corpora. The present invention effectively improves the extraction effect of Chinese-Thai and Chinese-Lao parallel sentence pairs.

Description

Technical Field

[0001] The invention relates to a method and a device for extracting parallel sentence pairs based on language similarity enhancement, and belongs to the technical field of natural language processing. Background Art

[0002] Neural machine translation is data-driven, and its performance is affected by the size of the corpus. This makes the machine translation from Lao and Thai to Chinese ineffective due to the lack of bilingual parallel corpora. Currently, there is a lack of Lao-Chinese and Thai-Chinese bilingual parallel sentence pairs on the Internet, and they mainly exist in the form of comparable corpora. Crawling comparable corpora from the Internet and annotating bilingual parallel sentence pairs is a feasible method to improve the quality and richness of bilingual data. However, the use of manual methods to annotate Chinese-Lao and Thai parallel corpora from comparable corpora faces problems such as high annotation costs and difficulty in annotation. Therefore, it is of great significance to study parallel sentence pair extraction methods that can achieve automatic annotation. The current parallel sentence pair extraction method has a good performance on rich resource language pairs (such as English-French, English-German, etc.), because a large amount of data can provide sufficient generalization ability for the parallel sentence pair extraction model, but on low resource language pairs, due to the scarcity of training data, the parallel sentence pair extraction model is difficult to effectively represent sentences, which makes it difficult for the existing parallel sentence pair extraction model to achieve good results when directly applied to Thai and Lao.

[0003] Thai and Lao, as typical scarce resource languages ​​in Southeast Asia, have their own special language characteristics. Although Thai and Lao have different character forms, they have high similarity in pronunciation and grammatical structure. In addition, Thai and Lao are both analytical languages, follow the word order rule of "Subject-Verb-Object, SVO" and post-adjective, have similar syntactic rules and the same part of speech, and have high part of speech and syntax similarity. Therefore, based on these language similarity features, data enhancement and knowledge sharing can be effectively performed, thereby improving the learning ability of the Chinese-Southeast Asian language sentence pair extraction model. Based on the above analysis, the present invention proposes a parallel sentence pair extraction method based on language similarity enhancement. Summary of the invention

[0004] The technical problem to be solved by the present invention is: the present invention provides a method and device for extracting parallel sentence pairs based on language similarity enhancement to alleviate the problems of poor representation ability and poor extraction effect of existing methods in low-resource parallel sentence pair extraction tasks.

[0005] The technical solution of the present invention is: in the first aspect, the present invention provides a parallel sentence pair extraction method based on language similarity enhancement, and the specific steps of the method are as follows:

[0006] Step 1: Data preprocessing for integrating language enhancement features: Obtain basic data of Chinese-Thai and Chinese-Lao bilingual parallel sentence pairs, use a data enhancement method based on phonetic dictionary word replacement, based on the basic data, take advantage of the language similarities between Thai and Lao in syntax, vocabulary and subwords, perform random word replacement expansion and subword pronunciation segment addition, and construct training corpus for integrating language enhancement features;

[0007] Step 2: Train a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement: First, encode the text through the E5 pre-trained language model to obtain a robust semantic representation; then introduce a multilingual joint training strategy to achieve knowledge sharing between Thai and Lao; finally, add a model optimization method based on attention representation alignment to enhance the representation consistency of the model in Chinese, Thai and Lao;

[0008] Step 3: Extraction of Chinese-Lao-Thai multilingual parallel sentence pairs: Deploy the trained Chinese-Lao-Thai multilingual parallel sentence pair extraction model as a parallel sentence pair extraction system to extract parallel sentence pairs from Chinese-Lao and Chinese-Thai comparable corpora.

[0009] As a further solution of the present invention, the specific steps of step 1 are:

[0010] Step 1.1, basic data construction: First, we used crawler technology to obtain Chinese-Thai and Chinese-Lao parallel corpora from OPUS and Asian language treebanks, and used the similarity calculation method of the E5 pre-trained language model to perform preliminary corpus screening. Combined with English entity word matching and sentence length screening methods, we constructed Chinese-Thai and Chinese-Lao bilingual parallel sentence pair datasets, and divided them into training sets, validation sets, and test sets.

[0011] Step 1.2: Perform data enhancement based on word replacement in the phonetic dictionary: Use the constructed bilingual phonetic dictionary and take advantage of the language similarities between Thai and Lao in syntax, vocabulary, and subwords to perform random word replacement expansion and subword pronunciation segment addition to construct a training corpus that incorporates language enhancement features to obtain enhanced samples.

[0012] As a further solution of the present invention, the specific steps of step 1.2 are:

[0013] Step 1.2.1, analyze the Thai-Lao bilingual phonetic dictionary, select Thai and Lao translation words with high pronunciation similarity in the phonetic dictionary, and perform random replacement at the word level;

[0014] Step 1.2.2: Split the words with subword consistency after pronunciation transcription at a fine-grained level, and append the phonetic information of the subwords with subword pronunciation consistency as external information to the end of the original word sequence;

[0015] Step 1.2.3: Lao-Chinese or Thai-Chinese sentence pairs for parallel sentence pair extraction model training , replace the Southeast Asian language in the Southeast Asian language-Chinese bilingual sentence pair with step 1.2.1-step 1.2.2, and construct additional enhanced samples of subword pronunciation sequence , and word replacement + subword pronunciation sequence additional enhanced examples , X represents the set of bilingual corpus pairs between Thai and Chinese, Y represents the set of bilingual corpus pairs between Lao and Chinese, (X, Y) represents the set of bilingual corpus pairs between Thai / Lao and Chinese, x represents a specific Thai and Chinese sentence pair extracted from set X, y represents a specific Lao and Chinese sentence pair extracted from set Y, (x, y) represents a specific Thai / Lao and Chinese sentence pair extracted from sets X and Y, that is, a parallel sentence pair, x i A primitive sequence representing Thai or Lao, y i represents a raw sequence of Chinese sentences, Represents random word replacement of the original sequence. Represents the addition of a subword pronunciation sequence to the original or random word replacement sequence.

[0016] As a further solution of the present invention, the specific steps of step 2 are:

[0017] Step 2.1: Based on the twin network model, the E5 pre-trained language model is introduced to encode the text;

[0018] Step 2.2: Use enhanced examples to conduct multilingual joint training, improve the basic twin network model, and propose a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement;

[0019] Step 2.3: Based on the Chinese-Lao-Thai multi-language parallel sentence pair extraction model based on language similarity enhancement, optimization based on attention representation alignment is added.

[0020] As a further solution of the present invention, the specific steps of step 2.1 are:

[0021] Step 2.1.1: Define the input of the parallel sentence pair extraction model as Lao-Chinese or Thai-Chinese sentence pairs , and feed it into the E5 pre-trained language model embedding layer , and get the embedding vector output with sentence semantic information:

[0022] ;

[0023] Step 2.1.2: embed the output of the layer and Send to the encoding layer and Learning semantic representation, the encoded representation output obtained by the encoding layer is as follows:

[0024] ;

[0025] ;

[0026] Step 2.1.3: and Send it to the decoding prediction layer for matching to determine whether it is similar; use the multi-layer perceptron MLP to learn the matching relationship and represent the encoding obtained in the above steps , , first , And the difference between the two and bitwise multiplication Perform splicing and make the splicing result [ vec x , vec y , vec x - vec y , vec x ∙ vec y ] Input to the linear transformation layer of the MLP , and finally use The activation function improves the model's representation ability and captures the text matching relationship in the text latent vector representation. The calculation process of the decoding prediction layer is as follows:

[0027] h =tanh ( W 1 ∙ [ vec x , vec y , vec x - vec y , vec x ∙ vec y ]) ;

[0028] Step 2.1.4: Match the text to the relational latent vector Input a linear transformation layer , compress its features, and then pass a The layer scores the text matching relationship , the calculation process is as follows:

[0029] ;

[0030] Step 2.1.5: Use bilingual cross entropy loss to optimize the training of the basic twin network model. The loss function for extracting bilingual parallel sentence pairs is defined as follows:

[0031] L 0 = - [a ∙ log s+ 1 - a ∙ log (1 - s)] ;

[0032] Where a is a label indicating whether it is parallel.

[0033] As a further solution of the present invention, the specific steps of step 2.2 are:

[0034] Step 2.2.1, embed the text part of the data augmentation sample through the pre-trained model, embed the pronunciation sequence part through the randomly initialized pronunciation embedding layer, and introduce the cross attention fusion module to fuse the two embedding vectors;

[0035] Step 2.2.2: For the input sentence pairs constructed in step 1.2 after data preprocessing or , improve 2.1.1; add enhanced examples for subword pronunciation sequences , and send it to the embedding layer to get the vector output , It is the E5 pre-trained language model embedding layer Modeled sentence vectors, yes The corresponding subword pronunciation embedding vector:

[0036] ,

[0037] For pronunciation sequence input Vectorize it to get Random word embedding layer of

[0038] Step 2.2.3. Define multi-head attention :

[0039] ;

[0040] in, , , is the latent vector representation of the input sentence, represents the output of the Mth attention head, is the vector concatenation operation, is the parameter matrix, is the number of heads of the multi-head attention mechanism, and each head is calculated as follows:

[0041] ;

[0042] in, The attention mechanism is a neural network method used to help the model selectively focus on different parts of the input data when processing sequence data. express Function, used to normalize the attention score into a probability distribution for weighting The value of , and the final attention output is obtained. , , is the parameter matrix, is the dimension of the hidden layer;

[0043] Step 2.2.4: Send the sentence vector representation and subword pronunciation embedding vector representation to the text pronunciation representation fusion module, fuse the pronunciation sequence attached to the end of the sentence into the text sequence, and obtain the fused vector representation:

[0044] ,

[0045] Hyperparameters, used to control the fusion ratio;

[0046] Step 2.2.5: Replace the output of the embedding layer of the E5 pre-trained language model , and sent to the twin network for training.

[0047] As a further solution of the present invention, the specific steps of step 2.3 are:

[0048] Step 2.3.1: Define the parallel sentences in Thai, Lao and Chinese as For the input Thai and Lao, the Thai / Lao shared word embedding and encoder module of the twin network are used to obtain the encoding vector of the fusion pronunciation. For the input Chinese, the Chinese word embedding and encoder module in the twin network are used to obtain the encoding vector. The obtained (Thai, Lao, Chinese) three-language vectors are ;

[0049] Step 2.3.2: Integrate Chinese information into Thai and Lao as alignment anchor information through interactive attention, and align and optimize the Thai embedding vector and Lao embedding vector that incorporate Chinese semantic anchor information; Multi-head attention For the input Thai sentence vector , by integrating Chinese information as anchor information through interactive attention , the output of interactive attention integrated into Chinese sentences is:

[0050] ;

[0051] Similarly, the interactive attention of Lao sentences Output:

[0052] ;

[0053] Step 2.3.3, calculate semantic similarity:

[0054] ;

[0055] Step 2.3.4: Use semantic similarity to measure the alignment of the encoding representations of parallel sentences and obtain the loss function for the Thai-Lao attention representation alignment :

[0056] ;

[0057] Step 2.3.5: Analogously to the Thai-Lao alignment task with Chinese as the anchor in steps 2.3.2-2.3.3 above, swap Thai, Lao, and Chinese sentences as new sentence pairs to be aligned and anchor sentences, respectively, perform attention representation alignment on Thai-Chinese and Lao-Chinese, and obtain the loss function for Thai-Chinese attention representation alignment ; Loss function for Lao-Chinese attention representation alignment ;

[0058] ;

[0059] ;

[0060] In the attention representation alignment stage, the encoding layer and pronunciation information fusion layer of the twin network are trained, and the total loss is:

[0061] .

[0062] As a further solution of the present invention, the specific steps of step 3 are:

[0063] Step 3.1, input data processing: the two input texts are processed by adding subword pronunciation segments through step 1, and then molecular word operations are performed according to the corresponding texts and subword pronunciation word lists to obtain sentence pair data to be entered into the parallel sentence pair extraction model for judgment;

[0064] Step 3.2, build parallel sentence pair extraction service: build parallel sentence pair extraction server, deploy the trained ".bin" format model to the server; build a Chinese, Laotian and Thai multi-language parallel sentence pair extraction system on the Web, and provide services to multiple users through the web;

[0065] Step 3.3, parallel sentence pair extraction business implementation: input the Chinese-Lao or Chinese-Thai bilingual text to be judged for parallel sentence pairs into the Web end and transmit it to the server end through the API, and call the parallel sentence pair extraction model; specifically, input the Chinese data into the Chinese encoder to obtain the Chinese sentence encoding representation, attach the sub-word pronunciation fragments to the Lao or Thai data, and input the Lao and Thai shared encoder and text pronunciation representation fusion module to obtain the Lao and Thai sentence encoding representation, and then send the two encoding representations to the parallel reasoning module to determine whether the two sentences are parallel.

[0066] In a second aspect, the present invention provides a parallel sentence pair extraction device based on language similarity enhancement, comprising a module for executing the method described in the first aspect.

[0067] The beneficial effects of the present invention are:

[0068] 1. The present invention effectively utilizes the language similarity characteristics of Thai and Lao, and jointly trains the Thai-Chinese parallel sentence pair extraction task and the Lao-Chinese parallel sentence pair extraction task. Thai and Lao are used as one input end of the twin network, and Chinese is used as the other input end of the twin network for joint training, and knowledge sharing of Thai and Lao is achieved through joint training;

[0069] 2. The present invention uses Thai-Lao similarity combined with phonetic dictionary to design data enhancement method. Use Thai-Lao bilingual phonetic dictionary to replace Thai and Lao words and replace subwords, construct mixed corpus pseudo data, send the mixed corpus pseudo data to training, and improve data richness through data enhancement method;

[0070] 3. Use a small amount of accurate trilingual aligned corpus to perform representation alignment optimization, obtain the vector representation of the trilingual aligned corpus through the encoding layer of the parallel sentence pair extraction model, and then select any two language vectors of the trilingual vectors as the language pair to be determined, and use the third language as anchor information. It is integrated into the language pair to be determined through interactive attention fusion to obtain bilingual parallel sentence pairs with more consistent vector representation. The obtained bilingual parallel sentence pairs are used for alignment task training, so that the encoding layer of the model is optimized in one step, the consistency of the model encoding layer in the representation distribution of Thai, Lao, and Chinese is enhanced, and the model's parallel sentence pair extraction capability is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is the overall model architecture diagram of the present invention;

[0072] Figure 2 Schematic diagram of a twin network model based on a pre-trained language model in the present invention;

[0073] Figure 3 Schematic diagram of the improved twin network model based on the pre-trained language model in the present invention; wherein a) represents a schematic diagram of a twin structure bilingual parallel sentence pair extraction network, and b) represents a schematic diagram of a parallel sentence pair extraction network of multi-language joint training + data enhancement;

[0074] Figure 4 This is a network diagram of the attention representation alignment part in the present invention;

[0075] Figure 5 The figure is an overall flow chart of parallel sentence pair extraction based on language similarity enhancement in the present invention. DETAILED DESCRIPTION

[0076] Example 1: Figures 1 - 5 As shown, the present invention provides a parallel sentence pair extraction method based on language similarity enhancement, and the specific steps of the method are as follows:

[0077] Step 1, data preprocessing of fusion language enhancement features: Thai and Lao are both low-resource languages, and currently open-source Chinese-Thai and Chinese-Lao bilingual parallel sentence pairs are scarce. The present invention first obtains the basic data of Chinese-Thai and Chinese-Lao bilingual parallel sentence pairs from OPUS, Asian language tree banks and bilingual websites through crawler technology, and uses a data enhancement method based on phonetic dictionary word replacement. Based on the basic data, the language similarities between Thai and Lao in syntax, vocabulary and subwords are used to perform random word replacement expansion and subword pronunciation segment addition to construct a training corpus with fusion language enhancement features;

[0078] As a further solution of the present invention, the specific steps of step 1 are:

[0079] Step 1.1, basic data construction: firstly, crawler technology was used to obtain Chinese-Thai and Chinese-Lao parallel corpora from OPUS and Asian language treebanks, and the similarity calculation method of the E5 pre-trained language model was used to perform preliminary corpus screening, and combined with English entity word matching and sentence length screening methods, Chinese-Thai and Chinese-Lao bilingual parallel sentence pair datasets were constructed; since the dataset was not divided into training set, validation set and test set, the present invention selected 2000 Chinese-Thai and Chinese-Lao data as validation sets, 2000 data each as test sets, and the remaining data as training sets;

[0080] Step 1.2: Perform data enhancement based on word replacement in the phonetic dictionary: Use the constructed bilingual phonetic dictionary and take advantage of the language similarities between Thai and Lao in syntax, vocabulary, and subwords to perform random word replacement expansion and subword pronunciation segment addition to construct a training corpus that incorporates language enhancement features to obtain enhanced samples.

[0081] As a further solution of the present invention, the specific steps of step 1.2 are:

[0082] Step 1.2.1. Early studies on Thai and Lao have already constructed Thai-Lao bilingual phonetic dictionary, as shown in Table 1.

[0083] Table 1 is an example of a bilingual phonetic dictionary

[0084]

[0085] First, analyze the Thai-Lao bilingual phonetic dictionary, select the Thai and Lao translation words with high pronunciation similarity in the phonetic dictionary, and make random replacement at the word level; as shown in Table 1, the words in the first column of the second row and the third column of the second row are mutually translated words, so random word replacement is performed at a certain ratio, so as to expand the Chinese-Thai and Chinese-Lao bilingual corpus. Since Thai and Lao data have high syntactic similarity, the usage of Thai and Lao translation words in sentences is also relatively the same. The sentence-level word replacement of the present invention can enhance data abundance by introducing less noise interference. The present invention sets the random replacement probability to 0.3.

[0086] Step 1.2.2, the similarity analysis of the Thai and Lao translations after phonetic transcription shows that compared with the word granularity, Thai and Lao are more consistent in pronunciation at the subword granularity. Therefore, the present invention splits the words with subword consistency after pronunciation transcription at a fine granularity, and attaches the phonetic information of the subwords with subword pronunciation consistency as external information to the back of the original word sequence; manually split into the form of translation words + similar subword fragment pronunciation, and attach the subword fragment pronunciation to the back of the text sequence as additional alignment information. In order to more comprehensively obtain the pronunciation consistency information of the subword granularity, the present invention compares and analyzes the translation words in the phonetic dictionary from the beginning and end of the word, and performs the above processing on the words. The example here is the processing of the aligned words with aligned pronunciation at the end of the word. In the above way, the present invention obtains 1836 subword pronunciation fragments with pronunciation consistency from the phonetic dictionary, and verifies them on the Thai-Lao bilingual parallel corpus of the Asian language tree bank. The example results are shown in Table 2.

[0087] Table 2 shows examples of subword pronunciation segments.

[0088]

[0089] Step 1.2.3: Lao-Chinese or Thai-Chinese sentence pairs for parallel sentence pair extraction model training , replace the Southeast Asian language in the Southeast Asian language-Chinese bilingual sentence pair with step 1.2.1-step 1.2.2, and construct additional enhanced samples of subword pronunciation sequence , and word replacement + subword pronunciation sequence additional enhanced examples , X represents the set of bilingual corpus pairs between Thai and Chinese, Y represents the set of bilingual corpus pairs between Lao and Chinese, (X, Y) represents the set of bilingual corpus pairs between Thai / Lao and Chinese, x represents a specific Thai and Chinese sentence pair extracted from set X, y represents a specific Lao and Chinese sentence pair extracted from set Y, (x, y) represents a specific Thai / Lao and Chinese sentence pair extracted from sets X and Y, that is, a parallel sentence pair, x i A primitive sequence representing Thai or Lao, y i represents a raw sequence of Chinese sentences, Represents random word replacement of the original sequence. It represents the addition of subword pronunciation sequences to the original or random word replacement sequence. Since Thai and Lao have a high pronunciation similarity, adding subword segments to them can also enhance the knowledge sharing ability of Thai and Lao. The constructed Thai data enhancement segment is shown in Table 3:

[0090] Table 3 shows an example of Thai data augmentation.

[0091]

[0092] For the original Thai text in the second row and second column of Table 3, the subword appending operation attaches the subword pronunciation fragments with the same phonetic symbols to the end of the sentence in the order of appearance as external information, and constructs a new Thai output as shown in the second row and third column of Table 3. The word replacement + subword fragment appending operation adds the operation of random synonym word replacement to the original sequence, and constructs a new Thai input as shown in the second row and fourth column of Table 3.

[0093] The Lao-Chinese bilingual parallel sentence pairs are processed in the same way to construct data augmentation samples. The Lao data augmentation fragments obtained are shown in Table 4:

[0094] Table 4 shows an example of Lao data augmentation.

[0095]

[0096] The data-enhanced bilingual parallel sentence pairs constructed by the above method are sent to the multi-language joint training parallel sentence pair extraction model for training to enhance the model's parallel sentence pair extraction capability.

[0097] Step 2: Train a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement: When the existing parallel sentence pair extraction method is applied to Chinese-Thai and Chinese-Lao, the scarcity of Thai and Lao training data leads to weak model representation ability and poor extraction quality. A Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement is trained; first, the text is encoded through the E5 pre-trained language model to obtain a robust semantic representation; then a multilingual joint training strategy is introduced to achieve knowledge sharing in Thai and Lao; finally, a model optimization method based on attention representation alignment is added to enhance the representation consistency of the model in Chinese, Thai and Lao;

[0098] As a further solution of the present invention, the specific steps of step 2 are:

[0099] Step 2.1: Based on the twin network model, the E5 pre-trained language model is introduced to encode the text;

[0100] Step 2.2: In order to further share Thai and Lao knowledge and enhance the model's ability to represent Thai and Lao, based on the enhanced samples constructed in Step 1.2, the present invention uses the enhanced samples to conduct multilingual joint training, improves the basic twin network model, and proposes a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement;

[0101] Step 2.3: In order to further enhance the representation consistency of the model in Chinese, Thai and Lao, an optimization based on attention representation alignment is added to the Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement.

[0102] As a further solution of the present invention, the specific steps of step 2.1 are:

[0103] Step 2.1.1: Define the input of the parallel sentence pair extraction model as Lao-Chinese or Thai-Chinese sentence pairs , and feed it into the E5 pre-trained language model embedding layer , and get the embedding vector output with sentence semantic information:

[0104] ;

[0105] Step 2.1.2: embed the output of the layer and Send to the encoding layer and Learning semantic representation, the encoded representation output obtained by the encoding layer is as follows:

[0106] ;

[0107] ;

[0108] Step 2.1.3: and The code is sent to the decoding prediction layer for matching to determine whether it is similar; a multilayer perceptron (MLP) is used to learn the matching relationship and the encoding representation obtained in the above steps is , , first , And the difference between the two and bitwise multiplication Perform splicing and make the splicing result [ vec x , vec y , vec x - vec y , vec x ∙ vec y ] Input to the linear transformation layer of the MLP , and finally use The activation function improves the model's representation ability and captures the text matching relationship in the text latent vector representation. The calculation process of the decoding prediction layer is as follows:

[0109] h =tanh ( W 1 ∙ [ vec x , vec y , vec x - vec y , vec x ∙ vec y ]) ;

[0110] Step 2.1.4: To identify the latent vector containing the text matching relationship Classify and match the text to the relational latent vector Input a linear transformation layer , compress its features, and then pass a The layer scores the text matching relationship , the calculation process is as follows:

[0111] ;

[0112] Step 2.1.5: Use bilingual cross entropy loss to optimize the training of the basic twin network model. The loss function for extracting bilingual parallel sentence pairs is defined as follows:

[0113] L 0 = - [a ∙ log s+ 1 - a ∙ log (1 - s)] ;

[0114] Where a is a label indicating whether it is parallel.

[0115] As a further solution of the present invention, the specific steps of step 2.2 are:

[0116] Step 2.2.1: In order to effectively represent the data-augmented Thai and Lao inputs through the twin network, the text part of the data augmentation sample is embedded through the pre-trained model, and the pronunciation sequence part is embedded through the randomly initialized pronunciation embedding layer. The cross-attention fusion module is introduced to fuse the two embedding vectors, so that the external pronunciation information attached to the end of the sentence is effectively integrated into the Thai / Lao original text vector;

[0117] Step 2.2.2: For the input sentence pairs constructed in step 1.2 after data preprocessing or , improve 2.1.1; add enhanced examples for subword pronunciation sequences , and send it to the embedding layer to get the vector output , It is the E5 pre-trained language model embedding layer Modeled sentence vectors, yes The corresponding subword pronunciation embedding vector:

[0118] ,

[0119] For pronunciation sequence input Vectorize it to get Random word embedding layer of

[0120] Step 2.2.3. Define multi-head attention :

[0121] ;

[0122] in, , , is the latent vector representation of the input sentence, represents the output of the Mth attention head, is the vector concatenation operation, is the parameter matrix, is the number of heads of the multi-head attention mechanism, and each head is calculated as follows:

[0123] ;

[0124] in, The attention mechanism is a neural network method used to help the model selectively focus on different parts of the input data when processing sequence data. express Function, used to normalize the attention score into a probability distribution for weighting The value of , and the final attention output is obtained. , , is the parameter matrix, is the dimension of the hidden layer;

[0125] Step 2.2.4: Send the sentence vector representation and subword pronunciation embedding vector representation to the text pronunciation representation fusion module, fuse the pronunciation sequence attached to the end of the sentence into the text sequence, and obtain the fused vector representation:

[0126] ,

[0127] Hyperparameters are used to control the fusion ratio. is 0.8.

[0128] Step 2.2.5: Replace the output of the embedding layer of the E5 pre-trained language model , and sent to the twin network for training.

[0129] As a further solution of the present invention, the specific steps of step 2.3 are:

[0130] Step 2.3.1: Define the parallel sentences in Thai, Lao and Chinese as For the input Thai and Lao, the Thai / Lao shared word embedding and encoder module of the twin network are used to obtain the encoding vector of the fusion pronunciation. For the input Chinese, the Chinese word embedding and encoder module in the twin network are used to obtain the encoding vector. The obtained (Thai, Lao, Chinese) three-language vectors are ;

[0131] Step 2.3.2: Integrate Chinese information into Thai and Lao as alignment anchor information through interactive attention, and align and optimize the Thai embedding vector and Lao embedding vector that incorporate Chinese semantic anchor information; Multi-head attention Use the same method as in step 2.2.3 Same structure, multi-head attention For the input Thai sentence vector , by integrating Chinese information as anchor information through interactive attention , the output of interactive attention integrated into Chinese sentences is:

[0132] ;

[0133] It is worth noting that due to the The purpose of is to integrate Chinese anchor information into Thai, while 2.2.3 The function of is to merge Thai / Lao with its corresponding pronunciation features. The two have different functions, so The network is not shared with step 2.2.3.

[0134] Similarly, the interactive attention of Lao sentences Output:

[0135] ;

[0136] Step 2.3.3, calculate semantic similarity:

[0137] ;

[0138] Step 2.3.4: Use semantic similarity to measure the alignment of the encoding representations of parallel sentences and obtain the loss function for the Thai-Lao attention representation alignment :

[0139] ;

[0140] Step 2.3.5: Analogously to the Thai-Lao alignment task with Chinese as the anchor in steps 2.3.2-2.3.3 above, swap Thai, Lao, and Chinese sentences as new sentence pairs to be aligned and anchor sentences, respectively, perform attention representation alignment on Thai-Chinese and Lao-Chinese, and obtain the loss function for Thai-Chinese attention representation alignment ; Loss function for Lao-Chinese attention representation alignment ;

[0141] ;

[0142] ;

[0143] In the attention representation alignment stage, the encoding layer and pronunciation information fusion layer of the twin network are trained to improve the encoding consistency of the model for Chinese, Lao, and Thai sentences. The total loss is:

[0144] .

[0145] Step 3: Extraction of Chinese-Lao-Thai multilingual parallel sentence pairs: Deploy the trained Chinese-Lao-Thai multilingual parallel sentence pair extraction model as a parallel sentence pair extraction system to extract parallel sentence pairs from Chinese-Lao and Chinese-Thai comparable corpora.

[0146] As a further solution of the present invention, the specific steps of step 3 are:

[0147] Step 3.1, input data processing: the two input texts are processed by adding subword pronunciation segments through step 1, and then molecular word operations are performed according to the corresponding texts and subword pronunciation word lists to obtain sentence pair data to be entered into the parallel sentence pair extraction model for judgment;

[0148] Step 3.2, build parallel sentence pair extraction service: build parallel sentence pair extraction server, deploy the trained ".bin" format model to the server; build a Chinese, Laotian and Thai multi-language parallel sentence pair extraction system on the Web, and provide services to multiple users through the web;

[0149] Step 3.3, parallel sentence pair extraction business implementation: input the Chinese-Lao or Chinese-Thai bilingual text to be judged for parallel sentence pairs into the Web end and transmit it to the server end through the API, and call the parallel sentence pair extraction model; specifically, input the Chinese data into the Chinese encoder to obtain the Chinese sentence encoding representation, attach the sub-word pronunciation fragments to the Lao or Thai data, and input the Lao and Thai shared encoder and text pronunciation representation fusion module to obtain the Lao and Thai sentence encoding representation, and then send the two encoding representations to the parallel reasoning module to determine whether the two sentences are parallel.

[0150] According to the concept of the present invention, the present invention provides a parallel sentence pair extraction device based on language similarity enhancement, comprising the following integrated modules:

[0151] Data preprocessing module integrating language enhancement features: used to obtain basic data of Chinese-Thai and Chinese-Lao bilingual parallel sentence pairs, using a data enhancement method based on phonetic dictionary word replacement, based on the basic data, taking advantage of the language similarities between Thai and Lao in syntax, vocabulary and subwords, to perform random word replacement expansion and subword pronunciation segment addition, to construct training corpus integrating language enhancement features;

[0152] The Chinese-Lao-Thai multilingual parallel sentence pair extraction model training module based on language similarity enhancement is used to encode text through the E5 pre-trained language model to obtain a robust semantic representation; then a multilingual joint training strategy is introduced to achieve knowledge sharing between Thai and Lao; finally, a model optimization method based on attention representation alignment is added to enhance the representation consistency of the model in Chinese, Thai and Lao;

[0153] Chinese-Lao-Thai multilingual parallel sentence pair extraction module: used to deploy the trained Chinese-Lao-Thai multilingual parallel sentence pair extraction model as a parallel sentence pair extraction system to extract parallel sentence pairs from Chinese-Lao and Chinese-Thai comparable corpora.

[0154] In order to verify the effect of the parallel sentence pair extraction method based on language similarity enhancement proposed in the present invention, comparative experiments and ablation experiments were designed.

[0155] Precision (P), recall (R), and F1-measure (F1) are used as evaluation indicators for the performance of the parallel sentence pair extraction model. Precision represents the correct proportion of extracted sentences, recall represents the proportion of extracted sentences to all parallel sentences, and F1 is the harmonic mean of precision and recall. The evaluation indicator calculation formula is as follows:

[0156]

[0157]

[0158]

[0159] Among them, TP is a true positive example, FP is a false positive example, FN is a false negative example, and TN is a true negative example;

[0160] The comparative experimental results are shown in Table 5, which compares the proposed method with the existing seven baseline models:

[0161] (1) SVM: Support vector machine, which is a common binary classification method in the classification field. This method is used to train Chinese-Thai and Chinese-Lao parallel sentence pairs respectively.

[0162] (2) LR: Logistic regression model, a classic machine learning algorithm, is a type of regression model. This method is used to train Chinese-Thai and Chinese-Lao parallel sentence pairs.

[0163] (3) Bi-RNN: Bidirectional long short-term memory network, which uses recurrent neural network to realize semantic classification of text.

[0164] (4) mBERT: It uses the multilingual BERT pre-trained model to obtain semantic representation, calculates the semantic similarity of bilingual sentences, and determines whether they are parallel based on the similarity.

[0165] (5) Bi-LSTM+mBERT: The multilingual BERT pre-trained model is used to obtain semantic representation, which is then fed into Bi-LSTM to model the classification task based on semantic similarity and achieve the judgment of parallel sentence pairs.

[0166] (6) LaBSE: LaBSE is a model proposed by Google for representing language-independent sentences. The LaBSE sentence embedding model is used to implement network parallel text mining. The LaBSE model is used to obtain language-independent sentence representations, calculate the semantic similarity of bilingual sentences, and determine whether a sentence pair is parallel.

[0167] (7) E5: The E5 pre-trained language model trains text embedding representations based on weakly supervised contrastive learning. The E5 pre-trained language model is used to obtain semantic representations and calculate semantic similarity to achieve the judgment of parallel sentence pairs.

[0168] Table 5 Performance comparison of each model on the Chinese-Lao and Thai parallel sentence pair extraction tasks

[0169]

[0170] Compared with the traditional machine learning methods SVM and LR, the method of the present invention has achieved a leap-forward improvement in the extraction effects of the Chinese-Thai and Chinese-Lao test sets. This shows that the method of the present invention can better learn the semantic features of the language from the existing training data and achieve good knowledge sharing. However, the traditional machine learning-based methods rely on the features in the training data and have poor generalization ability. Since Thai and Lao are both low-resource languages, there is a lack of sufficient training data during the model training process, making it difficult to learn sufficient semantic representations, resulting in poor performance.

[0171] Compared with the method based on deep learning, the method of the present invention achieves the optimal value in the extraction effect of the two test sets of Chinese-Thai and Chinese-Lao. Compared with the Bi-RNN model, the present invention effectively improves the sentence pair extraction performance by introducing a pre-trained language model, and improves 17.61 F1 value and 36.82 F1 value on Chinese-Thai and Chinese-Lao tasks respectively. Compared with the mBert pre-trained model, the present invention introduces the E5 pre-trained language model, joint training, data enhancement and attention representation alignment to optimize the model encoding, so that the model obtains more accurate cross-language representation during encoding, and improves 18.15 F1 value and 42.80 F1 value on Chinese-Thai and Chinese-Lao tasks respectively. Compared with the Bi-LSTM+mBERT pre-trained model, the method of the present invention learns more powerful semantic representations and has better modeling capabilities, and improves 11.70 F1 value and 31.61 F1 value on Chinese-Thai and Chinese-Lao tasks respectively. Compared with the latest LaBSE pre-trained model and E5 large model, the method of the present invention optimizes the representation of specific languages ​​for Thai and Lao, improves the representation ability of the model for Thai and Lao, and obtains 4.41 F1 value and 5.00 F1 value improvement on Chinese-Thai and Chinese-Lao tasks compared with the LaBSE model, and improves 3.95 F1 value and 6.40 F1 value on Chinese-Thai and Chinese-Lao tasks compared with the E5 pre-trained language model. The method of the present invention achieves the optimal value in both Chinese-Thai and Chinese-Lao parallel sentence pair extraction tasks, indicating the effectiveness of the method of the present invention.

[0172] The ablation experiment results are shown in Tables 6 and 7, which verify the influence of different modules on the experimental results on the Chinese-Thai and Chinese-Lao parallel sentence pair extraction tasks.

[0173] In order to verify the effectiveness of each module proposed in this section, the present invention designed the following ablation experiment on the Chinese-Thai and Chinese-Lao parallel sentence pair extraction tasks.

[0174] Among them, the method of the present invention represents the method using all the innovative modules of the present invention, "-data enhancement" represents the elimination of the data enhancement method based on phonetic dictionary word replacement, "-attention representation alignment" represents the elimination of the model optimization method based on attention representation alignment, and E5 represents the use of only the E5 pre-trained language model to obtain semantic representation and calculate semantic similarity to perform the bilingual parallel sentence pair extraction task.

[0175] Table 6 Experimental results of parallel sentence pair extraction and dissolution when the test set is Chinese-Thai

[0176]

[0177] Table 7 Experimental results of extraction and fusion of parallel sentences when the test set is Chinese-Laotian

[0178]

[0179] From the analysis of Table 6 and Table 7, we can see that:

[0180] (1) The present invention adopts the pre-trained language model E5 as the basis for training. Compared with directly using the E5 model for parallel sentence pair extraction, the method of the present invention uses bilingual parallel corpus for retraining, so that the model achieves F1 values ​​of 94.60 and 98.18 in the Chinese-Thai and Chinese-Lao directions, respectively, and obtains significant translation performance improvement, indicating that the pre-trained model + fine-tuning paradigm can effectively improve performance.

[0181] (2) Each component proposed in the present invention can effectively improve the model performance. Among them, the phonetic dictionary data enhancement method contributes F1 values ​​of 0.93 and 2.19 to the model performance in the Chinese-Thai and Chinese-Lao directions, and the attention representation alignment method contributes F1 values ​​of 0.45 and 1.20 to the model performance in the Chinese-Thai and Chinese-Lao directions.

[0182] (3) After removing the data augmentation and representation alignment modules, joint training brings an F1 value improvement of 2.28 and 4.12 for the model performance in the Chinese-Thai and Chinese-Lao directions, indicating that joint training can effectively achieve knowledge sharing in multilingual neural machine translation and improve model capabilities. Moreover, this knowledge sharing brings greater benefits to tasks with lower resources (such as the Chinese-Lao parallel sentence pair extraction task).

[0183] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A parallel sentence pair extraction method based on language similarity enhancement, characterized by: The method comprises: Step 1: Data preprocessing for integrating language enhancement features: Obtain basic data of Chinese-Thai and Chinese-Lao bilingual parallel sentence pairs, use a data enhancement method based on phonetic dictionary word replacement, based on the basic data, take advantage of the language similarities between Thai and Lao in syntax, vocabulary and subwords, perform random word replacement expansion and subword pronunciation segment addition, and construct training corpus for integrating language enhancement features; Step 2: Train a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement: First, encode the text through the E5 pre-trained language model to obtain a robust semantic representation; then introduce a multilingual joint training strategy to achieve knowledge sharing between Thai and Lao; finally, add a model optimization method based on attention representation alignment to enhance the representation consistency of the model in Chinese, Thai and Lao; Step 3: Extraction of Chinese-Lao-Thai multilingual parallel sentence pairs: Deploy the trained Chinese-Lao-Thai multilingual parallel sentence pair extraction model as a parallel sentence pair extraction system to extract parallel sentence pairs from Chinese-Lao and Chinese-Thai comparable corpora; Step 1.2.1, analyze the Thai-Lao bilingual phonetic dictionary, select Thai and Lao translation words with high pronunciation similarity in the phonetic dictionary, and perform random replacement at the word level; Step 1.2.2: Split the words with subword consistency after pronunciation transcription at a fine-grained level, and append the phonetic information of the subwords with subword pronunciation consistency as external information to the end of the original word sequence; Step 1.2.3: Lao-Chinese or Thai-Chinese sentence pairs for parallel sentence pair extraction model training , replace the Southeast Asian language in the Southeast Asian language-Chinese bilingual sentence pair with step 1.2.1-step 1.2.2, and construct additional enhanced samples of subword pronunciation sequence , and word replacement + subword pronunciation sequence additional enhanced examples , X represents the set of bilingual corpus pairs between Thai and Chinese, Y represents the set of bilingual corpus pairs between Lao and Chinese, (X, Y) represents the set of bilingual corpus pairs between Thai / Lao and Chinese, x represents a specific Thai and Chinese sentence pair extracted from set X, y represents a specific Lao and Chinese sentence pair extracted from set Y, (x, y) represents a specific Thai / Lao and Chinese sentence pair extracted from sets X and Y, that is, a parallel sentence pair, x i A primitive sequence representing Thai or Lao, y i represents a raw sequence of Chinese sentences, Represents random word replacement of the original sequence. represents the addition of a subword pronunciation sequence to the original or random word replacement sequence; The specific steps of step 2 are: Step 2.1: Based on the twin network model, the E5 pre-trained language model is introduced to encode the text; Step 2.2: Use enhanced examples to conduct multilingual joint training, improve the basic twin network model, and propose a Chinese-Lao-Thai multilingual parallel sentence pair extraction model based on language similarity enhancement; Step 2.3: Based on the Chinese-Lao-Thai multi-language parallel sentence pair extraction model based on language similarity enhancement, optimization based on attention representation alignment is added; The specific steps of step 2.2 are: Step 2.2.1, embed the text part of the data augmentation sample through the pre-trained model, embed the pronunciation sequence part through the randomly initialized pronunciation embedding layer, and introduce the cross attention fusion module to fuse the two embedding vectors; Step 2.2.2: For the input sentence pairs constructed in step 1.2 after data preprocessing or , improve 2.1.1; add enhanced examples for subword pronunciation sequences , and send it to the embedding layer to get the vector output , It is the E5 pre-trained language model embedding layer Modeled sentence vectors, yes The corresponding subword pronunciation embedding vector: ； For pronunciation sequence input Vectorize it to get Random word embedding layer of Step 2.2.

3. Define multi-head attention : ； in, , , is the latent vector representation of the input sentence, represents the output of the Mth attention head, is the vector concatenation operation, is the parameter matrix, is the number of heads of the multi-head attention mechanism, and each head is calculated as follows: ； in, The attention mechanism is a neural network method used to help the model selectively focus on different parts of the input data when processing sequence data. express Function, used to normalize the attention score into a probability distribution for weighting The value of , and the final attention output is obtained. , , is the parameter matrix, is the dimension of the hidden layer; Step 2.2.4: Send the sentence vector representation and subword pronunciation embedding vector representation to the text pronunciation representation fusion module, fuse the pronunciation sequence attached to the end of the sentence into the text sequence, and obtain the fused vector representation: ； Hyperparameters, used to control the fusion ratio; Step 2.2.5: Replace the output of the embedding layer of the E5 pre-trained language model , sent to the twin network for training; The specific steps of step 2.3 are: Step 2.3.1: Define the parallel sentences in Thai, Lao and Chinese as For the input Thai and Lao, the Thai / Lao shared word embedding and encoder module of the twin network are used to obtain the encoding vector of the fusion pronunciation. For the input Chinese, the Chinese word embedding and encoder module in the twin network are used to obtain the encoding vector. The obtained Thai, Lao, and Chinese vectors are ; Step 2.3.2: Integrate Chinese information into Thai and Lao as alignment anchor information through interactive attention, and align and optimize the Thai embedding vector and Lao embedding vector that incorporate Chinese semantic anchor information; Multi-head attention For the input Thai sentence vector , by integrating Chinese information as anchor information through interactive attention , the output of interactive attention integrated into Chinese sentences is: ； Similarly, the interactive attention of Lao sentences Output: ； Step 2.3.3, calculate semantic similarity: ； Step 2.3.4: Use semantic similarity to measure the alignment of the encoding representations of parallel sentences and obtain the loss function for the Thai-Lao attention representation alignment : ； Step 2.3.5: Analogously to the Thai-Lao alignment task with Chinese as the anchor in steps 2.3.2-2.3.3 above, swap Thai, Lao, and Chinese sentences as new sentence pairs to be aligned and anchor sentences, respectively, perform attention representation alignment on Thai-Chinese and Lao-Chinese, and obtain the loss function for Thai-Chinese attention representation alignment ; Loss function for Lao-Chinese attention representation alignment ; ； ； In the attention representation alignment stage, the encoding layer and pronunciation information fusion layer of the twin network are trained, and the total loss is: 。 2. The method for extracting parallel sentence pairs based on language similarity enhancement according to claim 1, characterized in that: The specific steps of step 1 are: Step 1.1, basic data construction: First, we used crawler technology to obtain Chinese-Thai and Chinese-Lao parallel corpora from OPUS and Asian language treebanks, and used the similarity calculation method of the E5 pre-trained language model to perform preliminary corpus screening. Combined with English entity word matching and sentence length screening methods, we constructed Chinese-Thai and Chinese-Lao bilingual parallel sentence pair datasets, and divided them into training sets, validation sets, and test sets. Step 1.2: Perform data enhancement based on word replacement in the phonetic dictionary: Use the constructed bilingual phonetic dictionary and take advantage of the language similarities between Thai and Lao in syntax, vocabulary, and subwords to perform random word replacement expansion and subword pronunciation segment addition to construct a training corpus that incorporates language enhancement features to obtain enhanced samples.

3. The method for extracting parallel sentence pairs based on language similarity enhancement according to claim 1, characterized in that: The specific steps of step 2.1 are: Step 2.1.1: Define the input of the parallel sentence pair extraction model as Lao-Chinese or Thai-Chinese sentence pairs , and feed it into the E5 pre-trained language model embedding layer , and get the embedding vector output with sentence semantic information: ； Step 2.1.2: embed the output of the layer and Send to the encoding layer and Learning semantic representation, the encoded representation output obtained by the encoding layer is as follows: ； ； Step 2.1.3: and Send it to the decoding prediction layer for matching to determine whether it is similar; use the multi-layer perceptron MLP to learn the matching relationship and represent the encoding obtained in the above steps , , first , And the difference between the two and bitwise multiplication Perform splicing and make the splicing result Input to the linear transformation layer of the MLP , and finally use The activation function improves the model's representation ability and captures the text matching relationship in the text latent vector representation. The calculation process of the decoding prediction layer is as follows: ； Step 2.1.4: Match the text to the relational latent vector Input a linear transformation layer , compress its features, and then pass a The layer scores the text matching relationship , the calculation process is as follows: ； Step 2.1.5: Use bilingual cross entropy loss to optimize the training of the basic twin network model. The loss function for extracting bilingual parallel sentence pairs is defined as follows: ； Where a is a label indicating whether it is parallel.

4. The method for extracting parallel sentence pairs based on language similarity enhancement according to claim 1, characterized in that: The specific steps of step 3 are: Step 3.1, input data processing: the two input texts are processed by adding subword pronunciation segments through step 1, and then molecular word operations are performed according to the corresponding texts and subword pronunciation word lists to obtain sentence pair data to be entered into the parallel sentence pair extraction model for judgment; Step 3.2, build parallel sentence pair extraction service: build parallel sentence pair extraction server, deploy the trained ".bin" format model to the server; build a Chinese, Lao, and Thai multi-language parallel sentence pair extraction system on the Web, and provide services to multiple users through the web; Step 3.3, parallel sentence pair extraction business implementation: input the Chinese-Lao or Chinese-Thai bilingual text to be judged for parallel sentence pairs into the Web end and transmit it to the server end through the API, and call the parallel sentence pair extraction model; specifically, input the Chinese data into the Chinese encoder to obtain the Chinese sentence encoding representation, attach the sub-word pronunciation fragments to the Lao or Thai data, and input the Lao and Thai shared encoder and text pronunciation representation fusion module to obtain the Lao and Thai sentence encoding representation, and then send the two encoding representations to the parallel reasoning module to determine whether the two sentences are parallel.

5. A parallel sentence pair extraction device based on language similarity enhancement, characterized in that: The method comprises a module for executing the method according to any one of claims 1 to 4.

Images