Operating method for cross-language machine reading comprehension task and natural language processing system

Abstract

The present disclosure relates to an operating method for a cross-language machine reading comprehension task, and a natural language processing system. The method comprises: acquiring target language training data corresponding to source language training data; constructing a parallel language pair for training; feeding the parallel language pair to a two-column model constructed on the basis of a pre-trained language model, and using a reverse propagation algorithm for training, the two-column model comprising a source column and a target column, and by means of finding an answer fragment semantically similar to a source language output answer outputted by the source column from the context information given by the target column, calibrating the answer fragment outputted by the target column. The present disclosure uses an attention mechanism to skillfully transfer an answer fragment of a source language to an answer output space of a target language; and a cross-attention block is refined by means of gradient decoupling knowledge sharing technology. In addition, transportability across languages is improved by means of enabling the model to learn semantic alignment from multiple granularities and calibrating the model output under the guidance of a teacher.

Description

Operational methods and natural language processing systems for cross-language machine reading comprehension tasks

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 29, 2023, with application number 202310620516.5 and invention name “Operation method and natural language processing system for cross-language machine reading comprehension tasks”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of deep learning, and in particular to an operating method for a cross-language machine reading comprehension task and a natural language processing system. Background Art

[0003] With the recent development of research related to language model training, significant progress has been made in downstream tasks of natural language processing (NLP), including machine reading comprehension. Machine reading comprehension (MRC), a type of NLP task, involves extracting the answer from a given text and a question.

[0004] Pre-trained language models often require large amounts of training data to generalize to downstream tasks. However, in cross-lingual settings, training data is typically only available for the high-resource language, while training data is often scarce for the low-resource target language. In cross-lingual language understanding, machine translation is often used to enhance the cross-lingual transferability of models. For example, training data can be translated from the source language to the target language, or vice versa, to aid inference. However, in cross-lingual machine reading comprehension, due to the variability in the position of answer fragments across different languages, it is difficult to provide in-depth assistance through machine translation to enhance cross-lingual transferability.

[0005] To this end, a solution is needed to improve the cross-language transferability in cross-language machine reading comprehension tasks.

[0006] Summary of the Invention

[0007] The technical problem addressed by this disclosure is to provide a method and natural language processing system for cross-language machine reading comprehension tasks. This disclosed embodiment utilizes an attention mechanism to cleverly transfer source language answer fragments to the target language answer output space. Furthermore, the cross-attention block is improved through gradient decoupling knowledge sharing technology. Furthermore, cross-language transferability is improved by enabling the model to learn semantic alignment at multiple granularities and calibrating the model output under teacher guidance.

[0008] According to a first aspect of the present disclosure, a method for operating a cross-language machine reading comprehension task is provided, comprising: obtaining target language training data corresponding to source language training data; constructing a training parallel language pair comprising the source language training data and the target language training data; feeding the training parallel language pair into a two-column model constructed based on a pre-trained language model, and performing training for the cross-language machine reading comprehension task using a backpropagation algorithm, wherein the two-column model comprises a source column using the source language training data as input and a target column using the target language training data as input, wherein the two-column model calibrates the answer segment output by the target column by finding an answer segment that is semantically similar to the source language output answer output by the source column from contextual information provided by the target column.

[0009] According to a second aspect of the present disclosure, a natural language processing system is provided, comprising: a storage node for storing a trained two-column model obtained by the method described in the first aspect; and an interaction node for loading the trained two-column model, obtaining input data in a target language and source language translation data based on the input data as an inference parallel language pair as model input, and using the obtained model output as an inference result of a cross-language machine reading comprehension task.

[0010] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the first aspect above.

[0011] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in the first aspect above.

[0012] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0015] Figure 1 shows an example of executing a cross-language natural language reasoning task and a machine reading comprehension task assisted by machine translation.

[0016] FIG2 shows a schematic flowchart of an operating method for a cross-language machine reading comprehension task according to an embodiment of the present disclosure.

[0017] FIG3 shows a structural example of a double-row model disclosed in the present invention.

[0018] Figure 4 shows the t-SNE distribution of sentence representations before and after regularization.

[0019] Figure 5 shows an example of a two-column model with a regularization layer added.

[0020] FIG6 shows an implementation example of the composition and operation of a double-column model according to an embodiment of the present disclosure.

[0021] Figure 7 shows the attention distribution heatmaps with and without token-level alignment.

[0022] FIG8 shows a schematic diagram of the composition of a natural language processing system according to an embodiment of the present disclosure.

[0023] FIG9 shows a schematic diagram of the structure of a computing device that can be used to implement the operating method of the above-mentioned cross-language machine reading comprehension task according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] In recent years, with the development of research related to training language models, research on downstream tasks of natural language processing (NLP), including machine reading comprehension, has also made significant progress. Pre-trained language models often require a large amount of training data to generalize in downstream tasks. However, in cross-lingual settings, training data is usually only available for high-resource languages, while training data is lacking for low-resource target languages. In cross-lingual language understanding, machine translation is often used to enhance the cross-lingual transferability of the model, such as translating training data from the source language to the target language, or from the target language to the source language to assist in reasoning. However, in cross-lingual machine reading comprehension, due to the variations in the positions of answer fragments in different languages, it is difficult to enhance cross-lingual transferability through in-depth assistance through machine translation.

[0026] Figure 1 shows an example of a cross-language natural language inference task and a machine reading comprehension task performed with the assistance of machine translation. The upper portion of Figure 1 shows an example of a natural language inference (NLI) task, which is used to determine whether there is a contradiction between two given sentences. Here, English can be considered a high-resource language, and Chinese can be considered a (relatively) low-resource language. In the English description, the two sentences "I do not know why." and "I am pretty sure why." give contradictory descriptions, so in the reasoning of this task, the model should give a "contradiction" judgment result (for example, more appropriately, a "contradiction" judgment result). In the Chinese description, the Chinese translations of the two sentences "I do not know why." and "I am pretty sure why." can be directly used to judge that the descriptions given are contradictory, so in the reasoning of this task, the model should give a "contradiction" judgment result. That is, in the task of Figure 1, machine translation is used to enhance the cross-language transferability of the model.

[0027] The lower part of Figure 1 shows an example of a cross-language machine reading comprehension task. A machine reading comprehension task is one in which, given a text and a question, the answer needs to be extracted from the text. In the example in Figure 1, the English text is "Combs rose from poverty in his native Clay County..." and the question is "Where is Combs from?" The answer is obviously "Clay County." However, since machine reading comprehension tasks require the answer to be labeled with the answer span, that is, the starting and ending positions of the corresponding content, in the above example, the correct output of the model should be 38 and 48, that is, the answer span starts at the 38th character after the beginning of the text (i.e., the position of the "C" in "Clay") and ends at the 48th character (i.e., the position of the "y" in "County"). When machine translation is used to obtain Chinese text content and questions, the answer to the question is obviously "Clay County", and the correct input to the model should start with 5 and end with 7. That is, the answer fragment is "Combs was born in Clay County and came from a poor family [...]". The underlined part represents the content protected by the fifth character after the beginning of the text content and ends at the seventh character.

[0028] As can be seen from the above, English and Chinese have different sentence structures, word orders, and morphological features, so the model output for English text cannot be directly used to calibrate the model output for Chinese text. One of the most important challenges in cross-lingual MRC is the lack of annotated datasets for low-resource languages. Most of the current MRC datasets are in English. Due to the linguistic and cultural differences between different languages, they exhibit different sentence structures, word orders, and morphological features. For example, languages ​​such as Japanese, Chinese, Hindi, and Arabic have different writing systems and more complex grammatical systems than English, which makes it impossible for MRC models to directly use machine translation and makes it challenging to understand text.

[0029] To address this issue, the present disclosure proposes a new method for cross-language MRC. In one embodiment, the present disclosure utilizes an attention mechanism to cleverly transfer source language answer fragments to the target language answer output space. Furthermore, a gradient decoupling knowledge sharing technique is proposed to improve cross-attention blocks. Furthermore, by allowing the model to learn semantic alignment at multiple granularities and calibrating the model output under teacher guidance, cross-language transferability is improved.

[0030] Figure 2 shows a schematic flow chart of an operation method for a cross-language machine reading comprehension task according to an embodiment of the present disclosure. The operation method shown in Figure 2 can first be regarded as a training method for a cross-language machine reading comprehension task.

[0031] In step S210, target language training data corresponding to the source language training data is obtained. The source language training data can be translated into target language training data via an external translation model. Here, an "external" translation model refers to a model used for translation other than the two-column model described below for performing cross-language machine reading comprehension tasks.

[0032] In step S220, a training parallel language pair is constructed, comprising source language training data and target language training data. Here, it is assumed that only source language training data is available, and the target language training data is obtained by translation using the translation model. The source language training data and the translated target language training data can be used as a training parallel language pair for subsequent model input and training.

[0033] Therefore, in step S230, the training parallel language pairs are fed into a two-column model constructed based on the pre-trained language model and trained for the cross-language machine reading comprehension task using the backpropagation algorithm. The model disclosed herein for performing cross-language machine reading comprehension tasks is constructed based on the pre-trained language model. Since parallel language pairs from the source and target languages ​​need to be processed simultaneously, the model has a two-column structure and can specifically include a source column that uses the source language training data as input and a target column that uses the target language training data as input. In one embodiment, the source and target columns can have similar structures, for example, each including L transformer layers.

[0034] To accurately output answer segments in the target language, the dual-column model of the present disclosure can calibrate the output answers by finding answer segments with semantic similarity to the source language output answers from the context of the target language. For example, the dual-column model can calibrate the answer segments output by the target column by finding answer segments with semantic similarity to the source language output answers from the source column from the context information provided by the target column.

[0035] For ease of understanding, Figure 3 shows a structural example of a two-column model of the present disclosure. As shown in Figure 3, the two-column model includes a source column on the left and a target column on the right. The source column and the target column each include L transformer layers. The transformer layer has a relatively fixed structure (for example, see the left part of Figure 6) and uses an attention mechanism for feature extraction. The attention mechanism is an algorithm used in deep learning that allows the model to focus on the parts related to the current task when processing input data, while ignoring the parts that are not related to the task. This mechanism can improve the performance and efficiency of the model, especially when processing long sequence data. In deep learning, the attention mechanism is usually implemented through a special layer called an "attention layer". This layer can calculate the importance of each input and use these importances as weights to weight the inputs, so that the model pays more attention to the important parts. It can be considered that each transformer layer includes an attention layer.

[0036] In addition, it can be noted that, in addition to the source column on the left and the target column on the right, the dual-column model of the present disclosure also includes an additional attention teacher module for guiding the hidden layer representation of the target column from the last hidden layer representation of the source column and the output of the source column. To this end, in one embodiment, the dual-column model calibrates the answer fragment output by the target column by finding an answer fragment that is semantically similar to the source language output answer output by the source column from the context information given by the target column, including: using the hidden layer state of the target language training data obtained by the target column as a query (i.e., Q in the attention mechanism); using the hidden layer state of the source language training data obtained by the source column as a key (i.e., K in the attention mechanism); using the source language output answer output by the source column as a value (i.e., V in the attention mechanism), based on the multi-head attention mechanism, using the above query, key, and value to obtain the output distribution of the migration (for example, the output distribution p of the migration in the following application example) T|S ), as the answer fragment with similar semantics. In one embodiment, the weighted average of the answer fragment with similar semantics and the answer fragment output by the target column can be used as the output of the two-column model, for example, the output distribution p of the migration output by the attention teacher module is used in the application example. T|S with the target column output distribution p T The average value of is used as the output of the two-column model.

[0037] In one embodiment, the attention teacher module can be trained by constructing a first loss function. At this time, the training parallel language pairs are fed into a two-column model constructed based on the pre-trained language model and the cross-language machine reading comprehension task is trained using a back propagation algorithm, which may include: constructing a first loss (e.g., corresponding to the teacher guidance loss in the application example) based on the answer fragments with similar semantics to the answer found. ) to shorten the distance between tokens with the same semantics but different languages; and training the cross-language machine reading comprehension task using a backpropagation algorithm based on the first loss.

[0038] Since a certain language may have a large number of phenomena such as function words, the average representation of the language is prominent. In other words, sentence representations of different languages ​​tend to be distributed together. In order to further improve the task reasoning accuracy of the model, the hidden state of the last layer can be regularized before being used as K and Q in the attention guidance. Therefore, in one embodiment, the hidden state used to obtain the output distribution of the migration can be regularized to remove the influence of the average representation of the language. Figure 4 shows the t-SNE distribution of the sentence representation before and after regularization. The figure shows that the sentence representations after regularization are no longer clustered together by language. Therefore, using the hidden representation of the target language training data obtained by the target column as a query can include: using the hidden state of the last layer of the target column to subtract the average representation of the target language and projecting the result back to the standard normal distribution to obtain the regularized target column hidden state as the query. Correspondingly, using the hidden representation of the source language training data obtained by the source column as a key can include: using the hidden state of the last layer of the source column to subtract the average representation of the source language and projecting the result back to the standard normal distribution to obtain the regularized source column hidden state as the key. The above regularization operation can correspond to the following application example, for example, from the last hidden layer state h L After regularization Figure 5 shows an example of a two-column model with a regularization layer added. The hidden state after regularization is It is actually used for the attention operation performed by the attention teacher module.

[0039] In one embodiment, a cross-attention mechanism can be used to fuse features from two columns. The cross-attention mechanism is a technique used in deep learning to effectively transfer information between multiple feature maps. In the cross-attention mechanism, each feature map calculates its similarity with other feature maps, and these similarities are used for weighted feature aggregation. In the present disclosure, an intermediate layer in the two-column model can be selected, such as the lth layer among L transformer layers, where l is a value other than 1 and L. In this intermediate layer, the hidden layer states of the intermediate layer of the target column and the hidden layer states of the intermediate layer of the source column are fused based on the cross-attention mechanism, and the fused features are merged into the source column. For example, to avoid performance loss in the source language due to feature fusion, gradients related to feature fusion can be prevented from backpropagating to the source column (for example, using the following sg(·) function). For example, due to the differences between different languages, in order to avoid the adverse effects of over-fusion, the feature fusion is only performed in one intermediate layer of the two-column model. In other words, feature fusion based on the cross-attention mechanism is only performed once in one of the L transformer layers. Therefore, the present disclosure extracts knowledge from the source language by using parallel language pairs as model input, thereby enhancing the understanding of the target language while avoiding the degradation of the source language representation.

[0040] In one embodiment, the knowledge transfer of the model can also be enhanced by semantic alignment. In order to improve attention, in one embodiment, token-level alignment can be performed. At this time, the training parallel language pairs are fed into a two-column model constructed based on the pre-trained language model and the cross-language machine reading comprehension task is trained using the backpropagation algorithm, including: constructing a second loss function that aligns the source tokens in the source language training data with the target tokens in the target language training data at the token level (for example, corresponding to the token-level alignment loss in the application example). ); and performing training for the cross-language machine reading comprehension task using a back propagation algorithm based on the second loss.

[0041] Alternatively, a common method of learning alignment, i.e., sentence-level alignment, can be used. In this case, a third loss function (e.g., corresponding to the sentence-level alignment loss in the application example) is constructed based on contrastive learning to align the source sentence representation in the source language training data with the target sentence representation in the target language training data at the sentence level. ); and using a backpropagation algorithm based on the third loss to train the cross-language machine reading comprehension task. In one embodiment, the second and third loss functions can also be used simultaneously, for example, by weighted summing the second and third loss functions as a multi-granularity semantic alignment loss function.

[0042] In addition, the loss function of the MRC task itself can also be used. At this time, the training parallel language pair is fed into a two-column model constructed based on the pre-trained language model and the cross-language machine reading comprehension task is trained using the back propagation algorithm, including: constructing a fourth loss function for characterizing the loss of the cross-language machine reading comprehension task (for example, corresponding to the MRC loss in the application example). ); and performing training for the cross-language machine reading comprehension task using a backpropagation algorithm based on the fourth loss.

[0043] It should be understood that the multiple embodiments described above are all preferred solutions that enable the dual-column model of the present disclosure to better perform cross-language machine reading comprehension tasks. The preferred features described above can be used together to form a preferred application example. An application example of the present disclosure will be described below in conjunction with Figure 6. However, combined with the attached ablation experiments, it can be seen that although these preferred features are effective in improving the level of model reasoning results, they are not essential.

[0044] Application Examples

[0045] FIG6 shows an implementation example of the composition and operation of a double-column model according to an embodiment of the present disclosure.

[0046] First, define the task using the criterion.

[0047] Given a context C and a question Q, the MRC task is to extract a subsequence from the context C as the correct answer to the question Q. The input sequence is represented as Where N is the sequence length. You can use and To express the probability distribution of the answer start and end positions. For simplicity, the two can be spliced ​​together to become Similarly, Represents a one-hot gold label sequence. For cross-lingual scenarios, there is only annotated training data from the source language. and original test data from the target language Available. S and T represent the source language and target language. Machine translation can be used to obtain training data in the target language. and source language test data In addition, using h l represents the hidden state of the sequence in layer l∈L, where L is the total number of layers of the transformer. Therefore, in order to predict the start and end positions of the correct answer segment in X, we can use h L Feed the linear classification layer and softmax function to introduce the probability distribution p over the entire sequence: p = softmax(Wh L+b), where W and b are the weights and biases of the linear classifier.

[0048] 1. Gradient decoupling knowledge sharing

[0049] Although machine translation from high-resource languages ​​to low-resource languages ​​can be used to train multilingual models, its drawbacks are significant. i) Machine translation quality varies across languages. ii) Original semantics can be easily lost during translation. iii) Acquiring task labels is relatively expensive, especially for token-level cross-lingual tasks. Therefore, as shown in Figure 6, we utilize parallel language pairs as input and fuse cross-lingual representations.

[0050] Cross-attention can be used for feature fusion. However, a performance loss can be observed in the source language. A reasonable speculation for this loss is that helping the target language extract target-related information from the hidden state of the source language will lead to the degradation of the source language representation. To alleviate this problem, the present disclosure proposes Gradient Decoupled Knowledge Sharing (GDKS), which is an improved version of the cross-attention block. In one embodiment, this method prevents the back propagation from the target language output to the source language hidden state. As compensation, a trainable correction term is added:

[0051] Here, sg(·) is used to stop the backpropagation gradient to prevent interference with the source language representation. f(·) refers to a trainable linear transformation with dropout (Dropout is a common operation in the field, which is used to stop the activation value of a certain neuron with a certain probability p during forward propagation to make the model more generalizable). Then use the target hidden state as the query (Q) and use the transformed source hidden state As the key (K) and value (V) to perform cross attention, defined as follows:

[0052] Where MHA is multi-head attention. Then, the target hidden state is fused with the source-aware target hidden state through the weight λ, which is calculated as follows:

[0053] where λ = ω * λ0 + b, where ω and b are trainable parameters. It's worth noting that GDKS is only implemented in a specific transformer layer. As shown in the middle of Figure 6, in a transformer intermediate layer that includes cross-attention, the source column does not incorporate features from the target column, while the target column incorporates features from the source column via an additional multi-head attention module.

[0054] 2. Attention Teacher Guided Calibration (ATGC)

[0055] GDKS focuses on transferring knowledge from the hidden states of a teacher model (trained on the source language) and can also calibrate the model output distribution through teacher guidance.

[0056] Regularization. A prerequisite for obtaining good guidance is to first regularize the representations of different languages. Here, it is assumed that the representation of the multilingual model consists of language-specific representations and language-independent representations. We estimate the language-specific features as the mean of the language representations and remove the language-specific features by subtracting the mean to retain only the universal semantic features. The basis for this operation is that a certain language may have a large number of phenomena such as function words. Therefore, the average representation of the language is prominent. Inspired by batch normalization, we transform the universal semantic representation into the standard normal distribution space:

[0057] where μ β and σ β are the mean and variance of the token-level representations in batch β. ε is a constant for numerical stability. To facilitate use in inference, this constant is set to be language independent.

[0058] Calibration. After regularization, the hidden state of the target language is used as the query (Q), and the hidden state and output distribution of the source language are used as the key (K) and value (V). MHA is still used and the conversion results of multiple heads are averaged. Therefore, the output distribution of the transfer for:

[0059] in and are the regularized hidden states of the source and target languages, respectively.

[0060] During the model training phase, teacher-guided loss can be incorporated To calculate p T|S Therefore, tokens with the same semantics but different languages ​​can still be brought closer by annotated data, even if their representations are very different. In one embodiment, we use the sample-level loss The definition is as follows:

[0061] For model inference, we calibrate the output of the target language by averaging the two output distributions, i.e.

[0062] 3. Multi-Grained Semantic Alignment (MGSA)

[0063] Multi-granularity semantic alignment technology can further enhance the knowledge transfer of the model.

[0064] 3.1 Sentence-level alignment

[0065] The common approach to learning alignment is to start at the sentence level. Here, contrastive learning can be used to strengthen cross-lingual alignment:

[0066] Where r is the average pooled sentence representation. + and r - Represent the positive samples from the parallel translation data and the negative samples in the mini-batch. sim(r1, r2) is the cosine similarity, i.e. τ is a temperature hyperparameter, which we set to 0.05 by default.

[0067] 3.2 Token-level alignment

[0068] The entropy of the cross-attention distribution (ECA) can be used to measure the quality of machine translation. The smaller the entropy of the attention distribution, that is, the more focused the attention, the higher the translation quality. Similarly, ECA can also be used to represent the quality of cross-lingual alignment. We use it as a penalty term for training cross-lingual models to avoid distracting attention in GDKS. Token-level alignment loss Can be defined as:

[0069] in represents the attention weight, n is the hidden size, I is the number of target tokens, and J is the number of source tokens. Next, the total alignment loss is summed up by two parts, where and η are coefficients:

[0070] 4. Final training goal

[0071] In one embodiment, the final training goal of X-STA is:

[0072] where γ is the teacher guidance loss factor. Refers to the cross entropy loss of the MRC task. The MRC loss can be Decomposed into source language and target language losses with a balance factor α:

[0073] 5. Ablation studies

[0074] We conduct an ablation study by removing each key component individually to evaluate the effectiveness of each module. The following table shows the effect of removing each key component on the performance of the commonly used cross-semantic MRC dataset. and Indicates a performance regression between 1.1 / 1.0 and 1.3 / 1.0 for English.

[0075] As shown in the table above, there is a performance degradation when any component is removed. Although removing GDKS has little impact on the overall performance, it significantly affects the representation of the source language, resulting in a significant performance degradation for the source language (i.e., English).

[0076] Removing the Attention Teacher Guided Calibration (ATGC) component degrades model performance the most, and the results show that mapping the output of the source language to the target language space is effective and feasible. In addition, there is still some performance loss compared to removing ATGC only at inference time, which shows that the improvement of ATGC does not only come from the weighted output of the source and target languages. Using answer fragments as additional knowledge can be achieved through the teacher guided loss To enhance cross-lingual alignment. As mentioned above, Figure 4 shows the visual distribution of sentence representations before and after regularization. In the original space, the distributions of the same language (same color) tend to cluster together, while after regularization, these representations are sparsely dispersed, which also shows that regularization can indeed decompose some language-specific representations.

[0077] Finally, we analyze the effectiveness of MGSA. As we can see, the token-level alignment t ) than sentence-level alignment s ) contributes more. A reasonable speculation is that the MRC task focuses more on token-level representation. Figure 7 shows the attention distribution heatmap with and without token-level alignment. The figure shows the average results of multi-head attention. As shown in the figure, without token-level alignment, the attention will be more scattered and the cross-lingual alignment will be poor. Token-level alignment will penalize this behavior and focus the attention on tokens related to the question-answer (e.g., "many").

[0078] The above-mentioned operating method for cross-language machine reading comprehension tasks disclosed herein can also be incorporated into existing natural language processing systems to enhance the reasoning capabilities of the system (e.g., cloud platform) for cross-language machine reading comprehension tasks. Figure 8 shows a schematic diagram of the composition of a natural language processing system according to an embodiment of the present disclosure. As shown in the figure, the system 800 includes a storage node 810 for storing various models, and in particular, can store trained double-column models obtained using the above-mentioned method of the present disclosure. The interactive node 820 can be used for interaction with system users. In one embodiment, the interactive node 820 is used to load the trained double-column model, obtain input data in the target language and source language translation data based on the input data as a parallel language pair for reasoning as a model input, and output the obtained model as the reasoning result of the cross-language machine reading comprehension task.

[0079] Thus, the operational method proposed in this paper for the cross-lingual machine reading comprehension task addresses the challenges of cross-lingual MRC in effectively leveraging translation data and linguistic and cultural differences. Through sharing, teaching, and coordination, this paper utilizes gradient-decoupled knowledge sharing techniques to transfer language representations, and further improves the model's cross-lingual transfer capabilities through output calibration and semantic alignment. Experimental results on three datasets demonstrate that our method achieves state-of-the-art performance compared to strong baselines.

[0080] FIG9 shows a schematic diagram of the structure of a computing device that can be used to implement the operating method of the above-mentioned cross-language machine reading comprehension task according to an embodiment of the present disclosure.

[0081] 9 , a computing device 900 includes a memory 910 and a processor 920 .

[0082] The processor 920 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, the processor 920 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0083] The memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0084] The memory 910 stores executable codes. When the executable codes are processed by the processor 920 , the processor 920 can execute the operating method of the cross-language machine reading comprehension task mentioned above.

[0085] The above description details the method and natural language processing system for performing a cross-language machine reading comprehension task, as well as the accompanying drawings. This disclosure utilizes training data not only for feature fusion but also for calibrating the target language output, thereby improving the accuracy of finding the corresponding answer.

[0086] In this paper, we propose a novel operational approach for the cross-lingual machine reading comprehension task that addresses the challenges of cross-lingual MRC in effectively leveraging translation data and linguistic and cultural differences. Through sharing, teaching, and coordination, we leverage gradient-decoupled knowledge sharing techniques to transfer language representations, and further improve the model's cross-lingual transfer capabilities through output calibration and semantic alignment. Experimental results on three datasets demonstrate that our approach achieves state-of-the-art performance compared to strong baselines.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0088] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0089] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor is caused to perform the various steps of the above-mentioned method according to the present disclosure.

[0090] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0091] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for operating a cross-language machine reading comprehension task, comprising: Acquire target language training data corresponding to the source language training data; Constructing a training parallel language pair including source language training data and target language training data; The training parallel language pairs are fed into a two-column model constructed based on the pre-trained language model, and a back-propagation algorithm is used to train the cross-language machine reading comprehension task, wherein the two-column model includes a source column using the source language training data as input and a target column using the target language training data as input, The dual-column model calibrates the answer segment output by the target column by finding the answer segment that is semantically similar to the source language output answer output by the source column from the context information given by the target column.

2. The method of claim 1, wherein: The dual-column model finds an answer segment that is semantically similar to the source language output answer output by the source column from the context information provided by the target column to calibrate the answer segment output by the target column, including: Using the hidden state of the target language training data obtained by the target column as a query; Using the hidden state of the source language training data obtained from the source column as a key; Taking the source language output answer output by the source column as a value, Based on the multi-head attention mechanism, the output distribution of the migration is obtained using the above query, key and value as the answer fragments with similar semantics to the answer.

3. The method of claim 2, wherein: Using the hidden representation of the target language training data obtained by the target column as a query includes: Using the hidden state of the last layer of the target column minus the average representation of the target language and projecting the result back to the standard normal distribution to obtain the regularized target column hidden state as the query, and Using the hidden representation of the source language training data obtained from the source column as a key comprises: The average representation of the source language is subtracted from the hidden state of the last layer of the source column and the result is projected back to the standard normal distribution to obtain the regularized source column hidden state as the key.

4. The method of claim 1, wherein: Feeding the training parallel language pairs into a two-column model constructed based on the pre-trained language model and using the backpropagation algorithm to train cross-language machine reading comprehension tasks includes: Constructing a first loss function based on the found answer segments with similar semantics to the answer, so as to shorten the distance between tokens with the same semantics but different languages; and The cross-language machine reading comprehension task is trained using a back-propagation algorithm based on the first loss function.

5. The method of claim 1, wherein: The dual-column model finds an answer segment that is semantically similar to the source language output answer output by the source column from the context information provided by the target column to calibrate the answer segment output by the target column, including: The weighted average of the answer fragments with similar answer semantics and the answer fragments output by the target column is used as the output of the two-column model.

6. The method of claim 1, wherein: In the middle layer of the dual-column model, feature fusion of the hidden layer state of the middle layer of the target column and the hidden layer state of the middle layer of the source column is performed based on a cross-attention mechanism, and the fused features are merged into the source column.

7. The method of claim 6, wherein The feature fusion is performed only in one intermediate layer of the two-column model; and Prevents gradients associated with feature fusion from backpropagating to the source columns.

8. The method of claim 1, wherein: Feeding the training parallel language pairs into a two-column model constructed based on the pre-trained language model and using the backpropagation algorithm to train cross-language machine reading comprehension tasks includes: Constructing a second loss function that enables token-level alignment of source tokens in the source language training data and target tokens in the target language training data; and A back propagation algorithm is used based on the second loss to perform training for the cross-language machine reading comprehension task.

9. The method of claim 1, wherein: Feeding the training parallel language pairs into a two-column model constructed based on the pre-trained language model and using the back-propagation algorithm to train cross-language machine reading comprehension tasks includes: Constructing a third loss function based on contrastive learning to align the source sentence representation in the source language training data with the target sentence representation in the target language training data at the sentence level; and The cross-language machine reading comprehension task is trained using a back-propagation algorithm based on the third loss.

10. The method of claim 1, wherein: Feeding the training parallel language pairs into a two-column model constructed based on the pre-trained language model and using the back-propagation algorithm to train cross-language machine reading comprehension tasks includes: Constructing a fourth loss function for characterizing the loss of the cross-language machine reading comprehension task; and A back propagation algorithm is used based on the fourth loss to perform training for the cross-language machine reading comprehension task.

11. A natural language processing system, comprising: A storage node, used to store the trained double-column model obtained by the method according to any one of claims 1 to 10; as well as The interactive node is used to load the trained two-column model, obtain input data in the target language and source language translation data based on the input data as a reasoning parallel language pair as a model input, and use the obtained model output as a reasoning result of the cross-language machine reading comprehension task.

12. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 10.

13. A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, cause the processor to perform the method according to any one of claims 1 to 10.

14. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 10.

Images