Data processing method and device for low-resource language, equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device for low-resource languages, equipment and storage medium. The method comprises: obtaining prompt data to be processed corresponding to a first language; inputting the prompt data to be processed into a preset target language model to obtain a corresponding processing result; wherein the target language model is obtained by training a preset language model based on a first loss, a second loss and a third loss; the first loss is used to indicate the adjustment of the parameters of a first semantic alignment key layer in the preset language model; the second loss is used to indicate the adjustment of the parameters of a second semantic alignment key layer in the preset language model; the third loss is used to indicate the adjustment of the parameters of a plurality of network layers contained in the preset language model, and the third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label. In this way, the data processing performance of the model for low-resource languages can be significantly improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, device and storage medium for low-resource languages. Background Technology

[0002] With the rapid development of large-scale pre-training technology, Multilingual Large Language Models (MLLMs) trained on massive multilingual corpora have demonstrated strong cross-lingual understanding, reasoning, and text generation capabilities, and have been widely applied in natural language processing tasks such as machine translation, question answering, and text summarization. However, due to the significant differences in the available corpora for different languages ​​on the Internet, MLLMs tend to learn knowledge representations of high-resource languages ​​during the pre-training stage. This results in their semantic understanding and text generation capabilities for low-resource languages ​​being significantly weaker than those for high-resource languages, thus limiting the practical application effectiveness of MLLMs in low-resource language scenarios.

[0003] In related technologies, knowledge transfer from high-resource languages ​​to low-resource languages ​​is typically achieved by adding additional network modules. Specifically, a semantic alignment adapter or cross-lingual attention module is inserted between the encoding and decoding layers of a multilingual large-scale model. This new module is then jointly trained using parallel corpora, enabling it to learn the mapping relationship from the semantic space of the high-resource language to the semantic space of the low-resource language, thereby constructing a shared cross-lingual semantic representation space within the model. However, adding additional network modules increases computational overhead, and the parameters of these modules are trained independently of the main model, making it difficult to effectively synergize with the model's existing cross-lingual representation capabilities. Consequently, the model's performance in processing low-resource language data cannot be effectively improved. Summary of the Invention

[0004] This application proposes a data processing method, apparatus, device, and storage medium for low-resource languages, which can achieve bidirectional semantic alignment between low-resource and high-resource languages ​​with low training cost, and significantly improve the data processing performance of the model for low-resource languages.

[0005] To achieve the above objectives, a first aspect of this application proposes a data processing method for low-resource languages, the method comprising: Retrieve the pending prompt data corresponding to the first language; The pending prompt data is input into a preset target language model to obtain the corresponding processing result; The target language model is obtained by training a preset language model based on a first loss, a second loss, and a third loss. The first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model. The first semantic alignment key layer is selected based on the activation values ​​of the network layers during the process of translating the input first language text sample into the predicted second language text by the preset language model. The second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model. The second semantic alignment key layer is selected based on the activation values ​​of the network layer during the process of translating the input second language text sample into the predicted first language text by the preset language model. The first semantic alignment key layer and the second semantic alignment key layer belong to different network layers. The corpus size corresponding to the second language is larger than the corpus size corresponding to the first language. The third loss is used to indicate the adjustment of parameters of multiple network layers included in the preset language model. The third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label.

[0006] Accordingly, a second aspect of this application provides a data processing apparatus for low-resource languages, the apparatus comprising: The acquisition module is used to acquire the pending prompt data corresponding to the first language; The input module is used to input the prompt data to be processed into a preset target language model to obtain the corresponding processing result; The target language model is obtained by training a preset language model based on a first loss, a second loss, and a third loss. The first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model. The first semantic alignment key layer is selected based on the activation values ​​of the network layers during the process of translating the input first language text sample into the predicted second language text by the preset language model. The second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model. The second semantic alignment key layer is selected based on the activation values ​​of the network layer during the process of translating the input second language text sample into the predicted first language text by the preset language model. The first semantic alignment key layer and the second semantic alignment key layer belong to different network layers. The corpus size corresponding to the second language is larger than the corpus size corresponding to the first language. The third loss is used to indicate the adjustment of parameters of multiple network layers included in the preset language model. The third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label.

[0007] In some embodiments, the data processing apparatus for low-resource languages ​​further includes a training module for: Obtain a first instruction sample corresponding to the first language, input the first instruction sample into a preset language model to obtain a predicted first translation text corresponding to the second language, and determine a first loss based on the difference between the predicted first translation text and the first translation text label corresponding to the second language; From the multiple network layers included in the preset language model, at least one first semantic alignment key layer is determined. Based on the first loss, the parameters of the at least one first semantic alignment key layer are adjusted to obtain a first intermediate language model. Obtain a second instruction sample corresponding to the second language, input the second instruction sample into the first intermediate language model for translation, and obtain a predicted second translated text corresponding to the first language. Based on the difference between the predicted second translated text and the label of the second translated text corresponding to the first language, determine the second loss. At least one second semantic alignment key layer is determined from the plurality of network layers, and the parameters of the at least one second semantic alignment key layer are adjusted based on the second loss to obtain a second intermediate language model; Obtain a third instruction sample corresponding to the first language, input the third instruction sample into the second intermediate language model to obtain the predicted language response text corresponding to the first language, and determine the third loss based on the difference between the predicted language response text and the response text label; The parameters of the second intermediate language model are adjusted based on the third loss to obtain the target language model.

[0008] In some implementations, the training module is further configured to: Acquire a first language text sample and first instruction data that instructs the first language text sample to be translated into a second language text; In the process of obtaining the first activation value of each first neuron in each network layer during the translation of the preset language model into predicted second language text based on the first language text sample and the first instruction data; For each network layer, based on the difference between the first activation value corresponding to each first neuron and the second activation value corresponding to each second neuron in the adjacent preceding network layer, the first activation difference value corresponding to each first neuron is calculated. For each network layer, the change in the corresponding first average activation value is calculated based on the multiple first activation difference values ​​corresponding to multiple first neurons; Based on the magnitude relationship between the changes in multiple first average activation values ​​corresponding to multiple network layers, at least one first semantic alignment key layer with a larger change in the first average activation value is determined from the multiple network layers.

[0009] In some implementations, the training module is further configured to: Acquire second language text samples and second instruction data that instructs the second language text samples to be translated into first language text; In the process of obtaining the preset language model based on the second language text sample and the second instruction data to predict the first language text, the third activation value of each third neuron contained in each network layer; For each network layer, based on the difference between the third activation value corresponding to each third neuron and the fourth activation value corresponding to each fourth neuron in the adjacent preceding network layer, a second activation difference value is calculated for each third neuron. For each network layer, the change in the corresponding second average activation value is calculated based on the multiple second activation difference values ​​corresponding to multiple third neurons; Based on the magnitude relationship between the changes in multiple second average activation values ​​corresponding to multiple network layers, at least one second semantic alignment key layer with a larger change in the second average activation value is determined from the multiple network layers.

[0010] In some implementations, the training module is further configured to: For each first instruction sample, determine multiple first word tags contained in the first translated text tag, and based on the predicted first translated text, determine the first prediction sub-probability of the preset language model for each corresponding first word tag; For each first instruction sample, the first prediction probability corresponding to the predicted first translated text is calculated based on the multiple first prediction sub-probabilities corresponding to the multiple first word tags; Obtain multiple first prediction probabilities corresponding to multiple first instruction samples, and determine the first loss based on the mean of the multiple first prediction probabilities.

[0011] In some implementations, the training module is further configured to: For each second instruction sample, determine multiple second word tags contained in the second translated text tag, and based on the predicted second translated text, determine the second prediction sub-probability of the first intermediate language model for each corresponding second word tag; For each second instruction sample, the second prediction probability corresponding to the predicted second translated text is calculated based on the multiple second prediction sub-probabilities corresponding to the multiple second word tags; Obtain multiple second prediction probabilities corresponding to multiple second instruction samples, and determine a second loss based on the mean of the multiple second prediction probabilities.

[0012] In some implementations, the training module is further configured to: For each third instruction sample, determine multiple third word tags contained in the response text tag, and based on the predicted language response text, determine the third predictor probability for each corresponding third word tag predicted by the second intermediate language model; For each third instruction sample, the third prediction probability corresponding to the predicted language response text is calculated based on the multiple third prediction sub-probabilities corresponding to the multiple third word tags; Obtain multiple third prediction probabilities corresponding to multiple third instruction samples, and determine the third loss based on the mean of the multiple third prediction probabilities.

[0013] Accordingly, a third aspect of the embodiments of this application proposes a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the data processing method for low-resource languages ​​according to any one of the embodiments of the first aspect of this application.

[0014] Accordingly, a fourth aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data processing method for low-resource languages ​​according to any one of the embodiments of the first aspect of this application.

[0015] This application embodiment obtains the pending prompt data corresponding to a first language; inputs the pending prompt data into a preset target language model to obtain the corresponding processing result; wherein, the target language model is trained on the preset language model based on a first loss, a second loss, and a third loss; the first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model, and the first semantic alignment key layer is selected based on the activation values ​​of the network layers in the process of translating the input first language text sample into the predicted second language text by the preset language model; the second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model, and the second semantic alignment key layer is selected based on the activation values ​​of the network layers in the process of translating the input second language text sample into the predicted first language text by the preset language model, the first semantic alignment key layer and the second semantic alignment key layer belong to different network layers, and the corpus size corresponding to the second language is larger than the corpus size corresponding to the first language; the third loss is used to indicate the adjustment of the parameters of multiple network layers included in the preset language model, and the third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label. This approach enables the establishment of a bidirectional semantic mapping path between low-resource and high-resource languages ​​within the model by updating parameters only on key layers precisely located based on activation value changes. Specifically, this application avoids adding extra network modules or introducing parameters independent of the main model. Instead, it quantifies the sensitivity of each layer to cross-lingual semantic mapping based on the activation value changes of the feedforward neural network layers during actual translation using a pre-defined language model. Based on this, key semantic alignment layers that play crucial roles in both the low-resource to high-resource and high-resource to low-resource directions are selected. Subsequently, translation task-driven parameter adjustments are performed only on these two types of key layers, allowing the model to achieve bidirectional alignment of the representation space with minimal parameter updates. Finally, fine-tuning of the low-resource language's general instructions using a third loss adapts the aligned semantic representation to downstream tasks. In this process, because the selection of key layers directly anchors the network positions most sensitive to semantic mapping, redundant updates of irrelevant layers are avoided. Furthermore, the entire optimization process is performed only on the original model parameters without increasing the number of parameters or network structure complexity; therefore, the computational overhead and memory usage during training do not significantly increase. In summary, this application can achieve bidirectional semantic alignment between low-resource languages ​​and high-resource languages ​​with lower training costs, significantly improving the model's data processing performance for low-resource languages. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of a data processing system for low-resource languages ​​provided in an embodiment of this application; Figure 2 This is a flowchart of a data processing method for low-resource languages ​​provided in an embodiment of this application; Figure 3 This is a flowchart of the model training process provided in the embodiments of this application; Figure 4 This is a general flowchart of the data processing method for low-resource languages ​​provided in the embodiments of this application; Figure 5 This is a functional module diagram of the data processing device for low-resource languages ​​provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] With the rapid development of large-scale pre-training technology, Multilingual Large Language Models (MLLMs) trained on massive multilingual corpora have demonstrated strong cross-lingual understanding, reasoning, and text generation capabilities, and have been widely applied in natural language processing tasks such as machine translation, question answering, and text summarization. However, due to the significant differences in the available corpora for different languages ​​on the Internet, MLLMs tend to learn knowledge representations of high-resource languages ​​during the pre-training stage. This results in their semantic understanding and text generation capabilities for low-resource languages ​​being significantly weaker than those for high-resource languages, thus limiting the practical application effectiveness of MLLMs in low-resource language scenarios.

[0021] In related technologies, knowledge transfer from high-resource languages ​​to low-resource languages ​​is typically achieved by adding additional network modules. Specifically, a semantic alignment adapter or cross-lingual attention module is inserted between the encoding and decoding layers of a multilingual large-scale model. This new module is then jointly trained using parallel corpora, enabling it to learn the mapping relationship from the semantic space of the high-resource language to the semantic space of the low-resource language, thereby constructing a shared cross-lingual semantic representation space within the model. However, adding additional network modules increases computational overhead, and the parameters of these modules are trained independently of the main model, making it difficult to effectively synergize with the model's existing cross-lingual representation capabilities. Consequently, the model's performance in processing low-resource language data cannot be effectively improved.

[0022] Based on this, embodiments of this application provide a data processing method, apparatus, device, and storage medium for low-resource languages, which can achieve bidirectional semantic alignment between low-resource and high-resource languages ​​with lower training costs, and significantly improve the data processing performance of models for low-resource languages.

[0023] The data processing method, apparatus, device, and storage medium for low-resource languages ​​provided in this application are specifically described through the following embodiments. First, the data processing system for low-resource languages ​​in this application is described.

[0024] Please refer to Figure 1 In some implementations, embodiments of this application provide a data processing system for low-resource languages, including a terminal 11 and a server 12.

[0025] In some implementations, terminal 11 can be used to acquire pending prompt data corresponding to a first language (i.e., a low-resource language) and provide a human-computer interface for users to interact with the target language model. For example, it can be a hardware device such as a smartphone, tablet computer, personal computer, smart voice assistant device or dedicated artificial intelligence terminal.

[0026] Furthermore, the terminal 11 can receive questions, instructions or text to be processed by the user in the first language through the input interface (such as a touch screen, microphone or keyboard) configured on it, and send the pending prompt data to the server 12 through the communication module, as well as receive and display the processing results returned by the server 12, thereby realizing a low-resource language interaction service for users.

[0027] In some implementations, the server 12 can be used to deploy a preset language model and train the model to obtain a target language model, and use the trained target language model to perform inference processing on the pending prompt data sent by the terminal 11. For example, it can be a hardware device such as a standalone server, a server cluster, or a cloud server.

[0028] Specifically, the server 12 can run the model training module through its configured processor and memory. The training module adjusts the parameters of at least one first semantic alignment key layer in the preset language model based on the first loss, adjusts the parameters of at least one second semantic alignment key layer in the preset language model based on the second loss, and adjusts the parameters of multiple network layers contained in the preset language model based on the third loss, thereby obtaining the target language model. The target language model is then used to infer the pending prompt data sent by the terminal 11 and generate the corresponding processing result.

[0029] In some implementations, terminal 11 can encapsulate the user-input first language prompt data into a request message and send it to server 12. After receiving the request message, server 12 can call the trained target language model to perform semantic understanding and text generation processing on the prompt data, obtain the processing result, and return the result to terminal 11 in the form of a response message. Terminal 11 then presents the processing result to the user in the form of text, voice, or graphical user interface, thereby completing a complete service loop from low-resource language input to intelligent processing result output.

[0030] The data processing method for low-resource languages ​​in this application can be illustrated through the following embodiments.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0032] In this application embodiment, the description will focus on a data processing device for low-resource languages, which can specifically be integrated into a computer device. See [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating the steps of a data processing method for low-resource languages ​​provided in this application embodiment. Taking the integration of a low-resource language data processing device into a terminal or server as an example, the specific process when the processor on the terminal or server executes the program instructions corresponding to the low-resource language data processing method is as follows: Step 101: Obtain the pending prompt data corresponding to the first language.

[0033] In some implementations, in order to ensure that the model can accurately identify the type of the language to be processed and call the corresponding semantic mapping mechanism, the model can obtain the pending prompt data required for the processing task, thereby providing data support for the model's subsequent processing.

[0034] The first language can be a low-resource language with a small corpus size and relatively weak model representation ability during model training and inference, such as Swahili, Tamil, Lao, etc.

[0035] The pending prompt data can be prompt statements consisting of translation requests, knowledge question and answer requests, or text generation instructions made in the first language.

[0036] For example, taking Swahili as the low-resource language (first language), the prompt data to be processed can be the Swahili question "Je,mji mkuu wa Tanzania ni upi?" entered by the user. This data can be used to input into the target language model to trigger the model to perform natural language understanding and processing for the first language, and output the corresponding processing result such as "Tanzania's capital is Dodoma." or the corresponding Swahili response.

[0037] In some implementations, the pending prompt data may also include a corresponding task type identifier, which can be used to indicate the downstream task category to which the current pending prompt data belongs, including translation tasks, question-answering tasks, text summarization tasks, or dialogue generation tasks, so that the target language model can call the corresponding processing logic according to the task type.

[0038] By using the above methods, the task input of low-resource language can be incorporated into the model's processing flow, thereby providing a reliable data entry guarantee for the subsequent model to output accurate and robust processing results during low-resource language inference.

[0039] Step 102: Input the prompt data to be processed into the preset target language model to obtain the corresponding processing result; The target language model is obtained by training a pre-defined language model based on the first loss, the second loss, and the third loss. The first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model. The first semantic alignment key layer is selected by the activation values ​​of the network layer in the process of translating the input first language text sample into the predicted second language text according to the preset language model. The second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model. The second semantic alignment key layer is selected based on the activation values ​​of the network layer during the process of translating the input second language text sample into the predicted first language text by the preset language model. The first semantic alignment key layer and the second semantic alignment key layer belong to different network layers, and the corpus size corresponding to the second language is larger than the corpus size corresponding to the first language. The third loss is used to indicate the adjustment of parameters of multiple network layers contained in the preset language model. The third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label.

[0040] In some implementations, in order to enable a multilingual large model to generate accurate processing results for input prompts in low-resource languages ​​without adding additional model parameters, three stages of training can be performed sequentially on a preset language model. This establishes a bidirectional semantic alignment relationship between low-resource and high-resource languages ​​within the model and enhances the representation quality of low-resource languages. This allows the model to accurately understand and generate prompts in the first language based on the aligned cross-lingual semantic space during actual reasoning.

[0041] The target language model can be a multilingual large model obtained by sequentially training a one-way semantic mapping from a low-resource language to a high-resource language, a reverse semantic mapping from a high-resource language to a low-resource language, and a fine-tuning training of general instructions in the low-resource language, based on a preset language model. For example, it can be a model obtained by optimizing the above three training stages based on the LLaMA3-8B or Qwen2.5-7B architecture.

[0042] The processing result can be the text content output by the target language model based on the input prompt data, such as the translation result of the first language, the answer to the knowledge question in the first language, or the text generation result of the first language. It can be used as the final response to the user for the first language task request.

[0043] The first loss can be a quantitative metric used to measure the accuracy of the preset language model in predicting the translation of a first language text sample into a second language text. It can be used to indicate the gradient update and optimization of the parameters of at least one first semantic alignment key layer selected in the bottom layer of the preset language model.

[0044] The second loss can be a quantitative metric used to measure the accuracy of the preset language model in predicting the translation of second language text samples into first language text. It can be used to indicate the gradient update and optimization of the parameters of at least one second semantic alignment key layer selected in the top layer of the preset language model.

[0045] The third loss can be a quantitative indicator used to measure the accuracy of the preset language model in predicting language response text based on the general instruction data of the first language. It can be used to indicate the gradient update and optimization of the parameters of multiple network layers contained in the preset language model.

[0046] The preset language model can be a large multilingual model pre-trained on a large-scale multilingual corpus, such as the LLaMA series or Qwen series of multilingual models based on the Transformer architecture.

[0047] The first semantic alignment key layer can be at least one network layer selected from the bottom network layer of the preset language model based on the magnitude of the change in activation values ​​of each feedforward neural network layer during the translation process from low-resource language to high-resource language.

[0048] The first language text samples can be low-resource language sentence texts used to construct training data for translation instructions, such as the Swahili sentence "Hivi karibuni alipoteza dhidi ya Raonic kwenyeBrisbane Open.".

[0049] The predicted second language text can be the second language translation result text output by a preset language model based on the input first language text sample and translation instructions, such as the English sentence obtained by translating a Swahili sentence.

[0050] The second semantic alignment key layer can be at least one network layer selected from the top network layer of the preset language model based on the magnitude of the change in activation values ​​of each feedforward neural network layer during the translation process from high-resource language to low-resource language. It can be used to adjust parameters through the second loss to achieve reverse alignment mapping from the high-resource language representation space to the low-resource language representation space.

[0051] The second language text samples can be high-resource language sentence texts used to construct training data for translation instructions, such as the English sentence "He recently lost against Raonic in the Brisbane Open."

[0052] The predicted first language text can be the first language translation result text output by a preset language model based on the input second language text sample and translation instructions, such as a Swahili sentence obtained by translating an English sentence.

[0053] Among them, the predicted language response text can be the predicted response content output by the preset language model during the general instruction fine-tuning stage based on the input first language general instruction data, such as the response text generated by the model for first language knowledge question and answer instructions.

[0054] The response text label can be the standard answer text given in the first language general instruction data. Specifically, it can be the correct answer content corresponding to the first language question and answer instruction, and can be used as a reference benchmark to measure the accuracy of the predicted language response text.

[0055] In some implementations, the corpus size for the second language is larger than that for the first language. Specifically, the first language is a low-resource language, and the second language is a high-resource language. During the pre-training phase, the distribution of different languages ​​in the training corpus of the multilingual large-scale model varies significantly. The second language (such as English) dominates in the large-scale internet corpus, possessing massive amounts of training data. Therefore, the model has strong representation and generation capabilities for the second language. In contrast, the first language (such as Swahili, Tamil, etc.) accounts for a very small proportion of the pre-training corpus, and the model's ability to understand and generate its semantics is significantly insufficient. It is precisely because of this difference in corpus size that the model performs far worse than the second language when processing the first language. Therefore, it is necessary to transfer the rich semantic knowledge of the second language to the first language to compensate for the performance deficiency caused by insufficient data.

[0056] Specifically, the target language model can receive the aforementioned prompt data corresponding to the first language as input, perform word segmentation and vectorization on the input data, convert the text-based prompt data into an embedded representation sequence that the model can process, and then perform layer-by-layer forward propagation calculations through the encoding layer, decoding layer, and output layer of the target language model, finally outputting the processing result corresponding to the prompt data. For example, taking the prompt data to be processed as the Swahili translation instruction "Translate the sentence from Swahili to English: Hivi karibuni alipoteza dhidi yaRaonic kwenye Brisbane Open." as an example, the prompt data to be processed is input into the target language model, and after forward inference, the model outputs the corresponding processing result such as "He recently lost against Raonic in the Brisbane Open."

[0057] In some implementations, the processing results output by the target language model can adopt different output formats depending on the type of downstream task. For example, for translation tasks, the output is the translated target language text; for question-answering tasks, the output is the answer text to the question; and for text generation tasks, the output is the generated text that meets the instructions.

[0058] In some implementations, the pre-trained language model is a large multilingual model pre-trained on a large-scale multilingual corpus, such as the LLaMA series models or Qwen series models based on the Transformer architecture. The training process for the target language model can include three sequentially executed training phases: Phase 1: Using a first loss, parameters of a pre-selected first semantic alignment key layer in the pre-trained language model are adjusted to achieve positive semantic alignment from low-resource language representations to high-resource language representations; Phase 2: Using a second loss, parameters of a pre-selected second semantic alignment key layer in the model are adjusted to achieve reverse semantic alignment from high-resource language representations to low-resource language representations; Phase 3: Using a third loss, the parameters of multiple network layers in the model are fine-tuned overall to further adapt the model to the downstream task distribution of low-resource languages. For example, using LLaMA3-8B as the pre-trained language model, training in these three phases on a translation instruction dataset containing parallel corpora of Swahili and English can yield a target language model for Swahili enhancement.

[0059] Specifically, first-language text samples and their corresponding translation instructions can be input into a pre-defined language model for forward encoding. The activation response values ​​of each neuron in each feedforward neural network layer of the model are obtained, and the average change in activation value between any two adjacent layers is calculated. A larger change in average activation value indicates a higher level of participation in the cross-language semantic conversion process and greater sensitivity to semantic mapping. Subsequently, in the bottom layers of the model, several layers with the largest changes in average activation value are selected as the first semantic alignment key layers. The bottom layers refer to the layers closest to the input in the model, such as layers 1 to k in a model with a total of L layers. These layers are responsible for the initial semantic encoding and feature extraction of the input text. By selecting the first semantic alignment key layers in the bottom layers, the model can establish a mapping relationship from low-resource language to high-resource language during the semantic encoding stage, thereby preserving and enhancing the mapped semantic information in subsequent layers.

[0060] In some implementations, the first loss can be used to instruct the adjustment of parameters of at least one first semantic alignment key layer in a preset language model. Specifically, the first loss is determined based on the prediction accuracy of the preset language model in translating a first language text sample into a predicted second language text, for example, calculated based on the difference between the first language text sample and the corresponding second language translated text label. During the first stage of training the model, only the parameters of the aforementioned selected first semantic alignment key layer are updated and optimized using gradients, while the parameters of other network layers in the preset language model, excluding the first semantic alignment key layer, remain unchanged. For example, taking Swahili as the first language, English as the second language, and k1 as 3, the three layers with the largest changes in average activation value among the bottom 16 layers of the model are selected as the first semantic alignment key layers. By adjusting the parameters of these three layers using the first loss, a positive alignment mapping from low-resource language representation to high-resource language representation can be completed.

[0061] In some implementations, second language text samples and their corresponding reverse translation instructions can be input into a pre-defined language model for forward encoding. This allows for the acquisition of activation response values ​​at each neuron in each feedforward neural network layer of the model, and the average change in activation values ​​between adjacent layers is determined using the same calculation method. Subsequently, in the top layers of the model, several layers with the largest changes in average activation values ​​are selected as the second semantic alignment key layers. The top layers refer to the layers closest to the output in the model, such as layers k to L-1 in a model with a total of L layers. These layers are responsible for decoding the semantic representation after multi-level encoding and for text generation. By selecting the second semantic alignment key layers in the top layers, the model can utilize the semantic information of the high-resource language to guide the generation process of the low-resource language during the text generation stage, thereby improving the accuracy of the generation.

[0062] It should be noted that the first semantic alignment key layer and the second semantic alignment key layer belong to different network layers in the model. That is, the first semantic alignment key layer is located in the bottom region of the model, while the second semantic alignment key layer is located in the top region of the model. They are spatially separated from each other. This differentiated selection method can give full play to the semantic encoding advantages of the bottom layer and the text generation advantages of the top layer, thereby maximizing the semantic alignment effect.

[0063] Specifically, the second loss can be determined based on the prediction accuracy of translating a second-language text sample into a predicted first-language text using a pre-defined language model, for example, by calculating the difference between the second-language text sample and the corresponding first-language translated text label. During the second stage of model training, only the parameters of the aforementioned selected second semantic alignment key layers are updated and optimized using gradients, while the parameters of other network layers in the frozen model remain unchanged. For example, taking Swahili as the first language, English as the second language, and k2 as 3, the three layers with the largest change in average activation value among the top 16 layers of the model are selected as the second semantic alignment key layers. The parameters of these three layers are adjusted using the second loss to complete the reverse alignment mapping from high-resource language representation to low-resource language representation.

[0064] In some implementations, the third loss can be determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label. After completing the forward semantic alignment guided by the first loss and the reverse semantic alignment guided by the second loss, the model has initially acquired the ability to perform bidirectional semantic mapping between the first language and the second language.

[0065] Building upon this foundation, the model can be further fine-tuned using general first-language instruction data. This fine-tuning adjusts the parameters of multiple network layers by minimizing the third loss. For example, the general instruction data can include instruction-response sample pairs for various task types, such as question-and-answer instructions, summary instructions, and dialogue instructions in the first language. Inputting the first-language task instructions into the bidirectionally semantically aligned model yields predicted language response text. The third loss is then calculated based on the difference between this predicted language response text and the labeled response text. During the third stage of training, the parameters of some network layers (such as attention networks and feedforward neural networks) can be fine-tuned. The selected network layers can be chosen based on the specific circumstances. For example, only the parameters of the attention network and feedforward neural network in the pre-defined language model can be optimized, while keeping the parameters of other modules unchanged. This achieves better low-resource language processing performance with the same training cost. Through this stage of training, the model can further master the specific task execution capabilities of the first language based on the established bidirectional semantic alignment, thereby outputting accurate processing results for the first-language prompts during actual inference.

[0066] This application embodiment obtains the pending prompt data corresponding to a first language; inputs the pending prompt data into a preset target language model to obtain the corresponding processing result; wherein, the target language model is trained on the preset language model based on a first loss, a second loss, and a third loss; the first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model, and the first semantic alignment key layer is selected based on the activation values ​​of the network layers in the process of translating the input first language text sample into the predicted second language text by the preset language model; the second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model, and the second semantic alignment key layer is selected based on the activation values ​​of the network layers in the process of translating the input second language text sample into the predicted first language text by the preset language model, the first semantic alignment key layer and the second semantic alignment key layer belong to different network layers, and the corpus size corresponding to the second language is larger than the corpus size corresponding to the first language; the third loss is used to indicate the adjustment of the parameters of multiple network layers included in the preset language model, and the third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label. This approach enables the establishment of a bidirectional semantic mapping path between low-resource and high-resource languages ​​within the model by updating parameters only on key layers precisely located based on activation value changes. Specifically, this application avoids adding extra network modules or introducing parameters independent of the main model. Instead, it quantifies the sensitivity of each layer to cross-lingual semantic mapping based on the activation value changes of the feedforward neural network layers during actual translation using a pre-defined language model. Based on this, key semantic alignment layers that play crucial roles in both the low-resource to high-resource and high-resource to low-resource directions are selected. Subsequently, translation task-driven parameter adjustments are performed only on these two types of key layers, allowing the model to achieve bidirectional alignment of the representation space with minimal parameter updates. Finally, fine-tuning of the low-resource language's general instructions using a third loss adapts the aligned semantic representation to downstream tasks. In this process, because the selection of key layers directly anchors the network positions most sensitive to semantic mapping, redundant updates of irrelevant layers are avoided. Furthermore, the entire optimization process is performed only on the original model parameters without increasing the number of parameters or network structure complexity; therefore, the computational overhead and memory usage during training do not significantly increase. In summary, this application can achieve bidirectional semantic alignment between low-resource languages ​​and high-resource languages ​​with lower training costs, significantly improving the model's data processing performance for low-resource languages.

[0067] In some implementation methods, please refer to Figure 3To enhance the processing capability of a multilingual large-scale model for low-resource languages ​​without introducing additional parameters or significantly increasing training costs, selective parameter tuning of the bottom and top key layers (pre-selected based on activation value changes) can be achieved using the first and second losses, respectively. This establishes bidirectional semantic alignment between low-resource and high-resource languages. Then, a third loss is used to fine-tune the model for general instructions, enabling the model to possess accurate cross-lingual semantic mapping capabilities and low-resource language instruction compliance capabilities while maintaining its original knowledge. This results in a target language model with enhanced low-resource language processing performance. For example, the target language model can be trained in the following way: Step 201: Obtain the first instruction sample corresponding to the first language, input the first instruction sample into the preset language model to obtain the predicted first translation text corresponding to the second language, and determine the first loss based on the difference between the predicted first translation text and the first translation text label corresponding to the second language; Step 202: Determine at least one first semantic alignment key layer from the multiple network layers contained in the preset language model, and adjust the parameters of at least one first semantic alignment key layer based on the first loss to obtain a first intermediate language model. Step 203: Obtain the second instruction sample corresponding to the second language, input the second instruction sample into the first intermediate language model for translation, and obtain the predicted second translated text corresponding to the first language. Based on the difference between the predicted second translated text and the label of the second translated text corresponding to the first language, determine the second loss. Step 204: Determine at least one corresponding second semantic alignment key layer from multiple network layers, and adjust the parameters of at least one second semantic alignment key layer based on the second loss to obtain a second intermediate language model; Step 205: Obtain the third instruction sample corresponding to the first language, input the third instruction sample into the second intermediate language model to obtain the predicted language response text corresponding to the first language, and determine the third loss based on the difference between the predicted language response text and the response text label; Step 206: Adjust the parameters of the second intermediate language model based on the third loss to obtain the target language model.

[0068] The first instruction sample can be training data consisting of first language text, second language text, and instruction information indicating the direction of translation. For example, it can be a complete prompt sample that combines the Swahili sentence "Hivi karibuni alipoteza..." with the English sentence "He recently lost..." and the translation instruction "Translate the sentence from Swahili to English:".

[0069] The predicted first translation text can be the second language translation result generated and output by a preset language model based on the first language text and translation instructions in the first instruction sample.

[0070] The first intermediate language model can be a model obtained by adjusting the parameters of at least one first semantic alignment key layer selected from the bottom network layer based on a preset language model and using a first loss.

[0071] The second instruction sample can be training data consisting of second language text, first language text, and instruction information indicating the direction of translation. For example, it can be a complete prompt sample that combines the English sentence "He recently lost..." with the Swahili sentence "Hivi karibuni alipoteza..." and the instruction "Translate the sentence from English to Swahili:".

[0072] The predicted second translation text can be the first language translation result generated and output by the first intermediate language model based on the second language text and translation instructions in the second instruction sample. For example, the model translates the input English sentence and outputs a Swahili sentence.

[0073] The second intermediate language model can be a model obtained by adjusting the parameters of at least one second semantic alignment key layer selected from the top network layer based on the first intermediate language model and using a second loss.

[0074] The third instruction sample can be training data consisting of task instruction text in the first language and corresponding response text labels, such as the first language question-and-answer instruction "Please answer the following question:..." and its corresponding standard answer.

[0075] Specifically, the first semantic alignment key layer can be determined as follows: The first instruction sample is encoded using a pre-defined language model to obtain the activation response value of each neuron in each feedforward neural network layer of the model. The average change in activation value between any two adjacent layers is calculated; this change reflects the degree of participation of each layer in the cross-language semantic conversion process. Subsequently, in the bottom network layers of the model (e.g., layers 1 to k in a model with a total of L layers), the layers are sorted from largest to smallest according to the average change in activation value, and the k1 layers with the largest changes are selected as the first semantic alignment key layers. For example, taking the LLaMA3-8B model as an example, which contains a 32-layer Transformer structure, with k=16 and k1=3, the three layers with the largest changes in average activation value (e.g., layers 5, 8, and 12) from the bottom 16 layers of the model are selected as the first semantic alignment key layers.

[0076] In some implementations, the first instruction sample comprises a first language text, a second language text, and a translation instruction to translate the first language into the second language. For example, taking Swahili as the first language and English as the second language, parallel sentence pairs are sampled from a Swahili-English parallel corpus. The first instruction sample is constructed according to a preset template format: "Translate the sentence from Swahili to English: {Swahili text}". The Swahili text "Hivi karibuni alipoteza dhidi ya Raonic kwenye Brisbane Open." serves as input, and the corresponding English translation text "He recently lost against Raonic in the Brisbane Open." serves as the translation text label. The Swahili text and translation instruction from the first instruction sample are input into a preset language model (such as LLaMA3-8B). The model can perform forward reasoning based on the input Swahili text and translation instruction, generating a predicted first translation text word by word, i.e., the model's predicted English translation result.

[0077] Subsequently, the word-by-word probability difference between the predicted first translation text and the actual English translation text labeled "He recently lost against Raonic in the Brisbane Open." is calculated to obtain the first loss. This first loss can be used to quantify the degree of prediction bias of the model on the current translation sample; the larger the loss value, the lower the translation accuracy of the model.

[0078] Furthermore, the aforementioned first loss can be used to update and optimize the parameters of only the three key layers, while freezing the parameters of other network layers in the preset language model, excluding the three key layers, and keeping them unchanged. Through this selective parameter adjustment method, the model can gradually establish a positive alignment mapping from low-resource language representation to high-resource language semantic space in the bottom semantic encoding stage, resulting in a first intermediate language model with the initial ability to map from the first language to the second language.

[0079] Furthermore, the determination method for the second semantic alignment key layer is similar to that for the first semantic alignment key layer, except that the second semantic alignment key layer is selected from the top layers of the model (e.g., layers k to L-1 in a model with a total of L layers). For example, still using the LLaMA3-8B model (32 layers), with k=16 and k2=3, the three layers with the largest changes in average activation values ​​(e.g., layers 20, 24, and 28) from the top 16 layers (layers 16 to 31) are selected as the second semantic alignment key layers. It should be noted that the first semantic alignment key layer is located in the bottom region of the model, while the second semantic alignment key layer is located in the top region. They are spatially separated, and this differentiated selection method can leverage the advantages of the bottom layers in semantic encoding and the top layers in text generation, respectively.

[0080] In some implementations, the second instruction sample may consist of a second language text, a first language text, and a translation instruction to translate the second language into the first language. For example, following the Swahili-English example described above, corresponding reverse parallel sentence pairs can be sampled from the same parallel corpus. The second instruction sample is constructed according to a preset template format: "Translate the sentence from English to Swahili: {English text}", where the English text "He recentlylost against Raonic in the Brisbane Open." serves as input, and the corresponding Swahili translation "Hivi karibuni alipoteza dhidi ya Raonic kwenye Brisbane Open." serves as the translation text label. The translation instruction and English text contained in the second instruction sample are input into a first intermediate language model. The model performs forward inference based on the input English text and the reverse translation instruction, generating a predicted second translation text word by word—that is, the Swahili translation result predicted by the model. Subsequently, the predicted second translation text and the actual Swahili translation text used as labels are compared word by word to calculate the probability difference, and the second loss is determined in the same manner. The second loss is used to quantify the degree of prediction bias of the model on the back-translation task.

[0081] Furthermore, after determining the second semantic alignment key layers, the second loss can be used to update and optimize the parameters of only these three key layers, while freezing the parameters of other network layers in the first intermediate language model, excluding these three key layers, and keeping them unchanged. Through this selective parameter adjustment, the model can establish a reverse alignment mapping from high-resource language representation to low-resource language semantic space at the top text generation stage, resulting in a second intermediate language model with bidirectional semantic alignment capabilities.

[0082] Specifically, the third instruction sample can consist of the task instruction text in the first language and its corresponding response text label. For example, a question-and-answer instruction sample is sampled from a general Swahili instruction dataset. The instruction text is “Je, mji mkuu wa Tanzania ni upi?”, and the corresponding response text label is “Dodoma ndiyo mji mkuuwa Tanzania.”. The task instruction text contained in the above third instruction sample is input into a second intermediate language model. The model performs forward inference based on the input first language instruction text, generating a predicted language response text word by word. Subsequently, the word-by-word probability difference between the predicted language response text and the actual response text (which serves as the label) is calculated, and the third loss is determined in the same manner. This third loss can be used to quantify the model's instruction compliance ability and response accuracy when performing a general first language task.

[0083] Furthermore, after completing the forward semantic alignment guided by the first loss and the reverse semantic alignment guided by the second loss, the model has initially acquired the ability to perform bidirectional semantic mapping between the first and second languages. Based on this, a third loss can be used to fine-tune the parameters of multiple network layers in the second intermediate language model. For example, the parameters in the attention network and feedforward neural network of the second intermediate language model can be optimized and updated, while keeping the parameters of other modules such as layer normalization unchanged. Through this stage of overall fine-tuning, the model can be further adapted to the downstream task distribution of the first language based on the established bidirectional semantic alignment, enhancing the model's ability to follow various task instructions in the first language, ultimately resulting in a target language model with significantly enhanced low-resource language processing performance.

[0084] For example, when determining the first and second semantic alignment key layers, the number and position of the selected first semantic alignment key layers can be dynamically adjusted during multiple training iterations. That is, after each round of training, the average activation value change of each layer is recalculated and the set of semantic alignment key layers is updated, so that the selected semantic alignment key layers can be adaptively adjusted with the training process, rather than being fixed layers, thereby more accurately capturing the network layer with the most active semantic mapping and further optimizing the semantic alignment effect.

[0085] By using the above methods, a bidirectional semantic alignment relationship between low-resource and high-resource languages ​​can be established within the model. This avoids the risk of knowledge forgetting and increased computational overhead caused by overall fine-tuning of all parameters. Based on this, the model can be fine-tuned using the third loss to apply general instructions for low-resource languages. This allows the model to adapt to the task distribution of downstream tasks in low-resource languages ​​while maintaining the established bidirectional alignment capability. This provides a complete training path guarantee for generating accurate processing results for the first language prompts during subsequent actual inference.

[0086] In some implementations, to quantify the participation of each network layer in cross-linguistic semantic mapping, the average change in activation values ​​between adjacent feedforward neural network layers during translation can be calculated. Layers with larger changes are identified as the first semantic alignment key layers most sensitive to semantic mapping, thus providing a precise layer localization basis for subsequent selective parameter adjustment only on key layers. For example, step 202, "determining at least one corresponding first semantic alignment key layer from multiple network layers included in a preset language model," may include: (202.1) Obtain a first language text sample and first instruction data that instructs the first language text sample to be translated into a second language text; (202.2) Obtain the first activation value of each first neuron in each network layer during the process of translating the first language text sample and the first instruction data into the predicted second language text based on the preset language model; (202.3) For each network layer, based on the difference between the first activation value corresponding to each first neuron and the second activation value corresponding to each second neuron in the adjacent preceding network layer, calculate the first activation difference value corresponding to each first neuron; (202.4) For each network layer, based on the multiple first activation difference values ​​corresponding to multiple first neurons, calculate the corresponding change in the first average activation value; (202.5) Based on the magnitude relationship between the changes in the first average activation values ​​corresponding to multiple network layers, at least one first semantic alignment key layer with a larger change in the first average activation value is determined from the multiple network layers.

[0087] The first neuron can be a basic computational unit contained in a feedforward neural network of a certain network layer in a preset language model. It can be used to perform nonlinear transformation processing on the representation input to the layer and output the corresponding activation response value.

[0088] The first activation value can be the numerical response output by the first neuron of a certain network layer in a preset language model during the process of translating a first language text sample into a predicted second language text. It can be used to quantify the activity level of the neuron in the semantic conversion process.

[0089] The second neuron can be a basic computational unit contained in the feedforward neural network of the preorder network layer adjacent to the layer where the first neuron is located in the preset language model, and can be used to provide the activation response value of the preorder network layer.

[0090] The second activation value can be the numerical response output by the second neuron in the preorder network layer adjacent to the first neuron in the preset language model during the process of translating the first language text sample into the predicted second language text.

[0091] The first activation difference value can be the absolute value obtained by calculating the difference between the first activation value of each first neuron in the current network layer and the second activation value of the corresponding second neuron in the adjacent preceding network layer. Specifically, it can be the result of subtracting the activation value of the ith neuron in the (l-1)th layer from the activation value of the ith neuron in the ith neuron in the ith layer in a certain sample and taking the absolute value.

[0092] The first average activation value change can be the sum of the first activation differences of all first neurons in a certain network layer on all translated samples. The larger the value, the higher the degree of participation of that layer in the cross-language semantic conversion process.

[0093] Specifically, several parallel sentence pairs can be sampled from parallel corpora of the first and second languages. Each first-language text and translation instruction are then concatenated according to a preset template to construct translation instruction data for activation value analysis. For example, taking Swahili as the first language and English as the second language, the Swahili text “Hivikaribuni alipoteza dhidi ya Raonic kwenye Brisbane Open.” and its corresponding English translation “He recently lost against Raonic in the Brisbane Open.” are selected as parallel sentence pairs. The first instruction data is constructed according to the template “Translate the sentence from Swahili to English: {Swahili text}”.

[0094] In some implementations, the first instruction data can be input into a preset language model for forward propagation computation. After the model converts the input text into a vector representation through an embedding layer, it sequentially passes through multiple Transformer layers, each containing a feedforward neural network module. During the forward computation layer by layer, the activation output values ​​of all neurons in each feedforward neural network layer on the current batch of samples are recorded as the first activation value of the corresponding network layer. For example, in each Transformer layer of the LLaMA3-8B model, the feedforward neural network contains multiple neurons (e.g., 4096). After inputting N translation samples, the activation value of the i-th neuron in the l-th feedforward neural network on the n-th sample can be obtained, denoted as... .

[0095] In some implementations, after obtaining the activation values ​​of each layer of the feedforward neural network, for each first neuron in layer l, its first activation value can be calculated by subtracting its second activation value from the second activation value of the corresponding second neuron in the preceding layer (layer l-1), and the absolute value is taken to obtain the activation difference value of the first neuron. This activation difference value reflects the magnitude of the change in activation response of the same neuron between adjacent layers.

[0096] For example, in the case where the l-th layer contains M neurons and the current batch contains N translation samples, the change in the first average activation value is... It can be determined in the following ways: ; in, This represents the average change in activation value between layer l-1 and layer l; denoted by , where represents the activation value of the i-th neuron in the l-th layer of the feedforward neural network on the n-th sample; M is the number of neurons in the l-th layer of the feedforward neural network; and N is the number of translation samples in the current batch. This average activation value change quantifies the degree of change in the overall semantic representation of the l-th layer relative to its preceding layers in the cross-language translation task. A larger value indicates that the layer is more actively involved in the semantic conversion process.

[0097] Furthermore, in the bottom network layers of the preset language model, the change in the first average activation value between adjacent layers can be calculated sequentially as described above. The changes in the average activation values ​​of all bottom layers are then sorted in descending order, and the k1 layers with the largest changes are selected as the first semantic alignment key layers from low-resource language to high-resource language. For example, taking the LLaMA3-8B model, which contains 32 Transformer layers, the bottom 16 layers are used as candidate layers, and the corresponding values ​​for each layer are calculated... After selecting the value, choose the three largest layers (e.g.) , , The corresponding layers 5, 8, and 12 are used as the first key semantic alignment layers. In this way, the network layers most sensitive to semantic mapping from low-resource language to high-resource language can be accurately located, providing reliable layer location guidance for subsequent parameter adjustments only to key layers.

[0098] For example, when determining the first semantic alignment key layer, in addition to using the method of obtaining activation values ​​based on a single forward propagation, a strategy of averaging multiple samples can also be adopted. Specifically, multiple different subsets of first instruction data are randomly sampled from a parallel corpus, and multiple independent forward propagations and activation value changes are performed to obtain the average activation value change of each layer under each data set. Then, the results of multiple calculations are averaged or fused by voting, thereby eliminating the random bias of data caused by a single sampling and making the selection of the key layer more stable and reliable.

[0099] In some implementations, a cumulative moving average can be used to update the change in average activation value of each layer. Specifically, as the model parameters are gradually updated during training, the sensitivity of each layer to semantic mapping may change. To address this, after each training round, the change in average activation value of each layer can be recalculated based on the current model parameters, and a weighted moving average can be applied to it and the historical calculation results. This dynamically tracks the changing trend of the sensitivity of each layer, allowing the selected first semantic alignment key layer to adaptively adjust with the training process, further improving the semantic alignment effect.

[0100] By using the above methods, we can quantify and compare the sensitivity of different network layers in the semantic mapping process from low-resource language to high-resource language. Then, we can select the layer with the most dramatic activation changes from multiple network layers as the key layer for semantic alignment. This provides a precise basis for subsequent selective parameter adjustment of only the key layer, while avoiding computational redundancy and knowledge forgetting risks caused by adjusting parameters of irrelevant layers.

[0101] In some implementations, to quantify the participation of each network layer in the back-mapping process from a high-resource language to a low-resource language, the average activation value change between adjacent feedforward neural network layers during back-translation can be calculated. The top layer with the largest change can be identified as the second semantic alignment key layer sensitive to semantic back-mapping, thereby providing a layer localization basis for subsequent back-semantic alignment training. For example, step 204, "determining at least one corresponding second semantic alignment key layer from multiple network layers," may include: (204.1) Obtain a second language text sample and second instruction data that instructs the second language text sample to be translated into first language text; (204.2) Obtain the third activation value of each third neuron in each network layer during the process of translating the second language text sample and the second instruction data into the predicted first language text based on the preset language model; (204.3) For each network layer, based on the difference between the third activation value corresponding to each third neuron and the fourth activation value corresponding to each fourth neuron in the adjacent preceding network layer, calculate the second activation difference value corresponding to each third neuron; (204.4) For each network layer, based on the multiple second activation difference values ​​corresponding to multiple third neurons, calculate the corresponding change in the second average activation value; (204.5) Based on the magnitude relationship between the changes in the second average activation values ​​corresponding to multiple network layers, determine at least one second semantic alignment key layer with a larger change in the second average activation value from the multiple network layers.

[0102] The third neuron can be a basic computational unit contained in a feedforward neural network of a certain network layer in a pre-defined language model. It can be used to perform nonlinear transformation processing on the representation input to the layer and output the corresponding activation response value.

[0103] The third activation value can be the numerical response output by the third neuron of a certain network layer in the preset language model during the process of translating a second language text sample into a predicted first language text. For example, during the forward propagation of the model, the activation output value calculated by the i-th neuron of the l-th layer on the first sample.

[0104] The fourth neuron can be a basic computational unit contained in the feedforward neural network of the preorder network layer adjacent to the layer containing the third neuron in the preset language model, and can be used to provide the activation response value of the previous layer.

[0105] The fourth activation value can be the numerical response output by the fourth neuron in the preorder network layer adjacent to the third neuron in the preset language model during the process of translating the second language text sample into the predicted first language text. For example, it can be the activation output value calculated by the i-th neuron in the (l-1)-th layer on the first sample during the forward propagation of the model.

[0106] The second activation difference value can be the absolute value obtained by calculating the difference between the third activation value of each third neuron in the current network layer and the fourth activation value of the corresponding fourth neuron in the adjacent preceding network layer. For example, the result of subtracting the activation value of the i-th neuron in the l-1th layer from the activation value of the i-th neuron in the l-1th layer in a certain sample and taking the absolute value can be used to measure the change in the activation response of the same neuron between two adjacent layers.

[0107] The second average activation value change can be the sum of the second activation differences of all third neurons in a certain network layer across all translated samples. The larger the value, the higher the degree of participation of that layer in the reverse semantic back-mapping process from high-resource language to low-resource language.

[0108] In some implementations, several parallel sentence pairs can be sampled from parallel corpora of the first and second languages, and the second language text and the reverse translation instructions can be concatenated according to a preset template to construct translation instruction data for reverse activation value analysis. For example, taking Swahili as the first language and English as the second language, the English text “He recently lost against Raonic in the Brisbane Open.” and its corresponding Swahili translation “Hivi karibuni alipoteza dhidi ya Raonic kwenye Brisbane Open.” are selected as parallel sentence pairs, and the second instruction data is constructed according to the template “Translate the sentence from English to Swahili: {English text}”.

[0109] Furthermore, the second instruction data can be input into a preset language model for forward propagation computation. After the model converts the input text into a vector representation through the embedding layer, it sequentially passes through multiple Transformer layers, each containing a feedforward neural network module. During the forward computation layer by layer, the activation output values ​​of all neurons in each feedforward neural network layer on the current batch of samples are recorded as the third activation value of the corresponding network layer. For example, in each Transformer layer of the LLaMA3-8B model, the feedforward neural network contains multiple neurons (e.g., 4096). After inputting N back-translated samples, the activation value of the i-th neuron in the l-th feedforward neural network on the n-th sample can be obtained, denoted as... .

[0110] Specifically, after obtaining the activation values ​​of each layer of the feedforward neural network, for each third neuron in the l-th layer, its third activation value can be... The fourth activation value of the fourth neuron at the corresponding position in the preceding layer (layer l-1). The difference is calculated and its absolute value is taken to obtain the second activation difference value of the third neuron. This second activation difference value reflects the magnitude of the change in activation response of the same neuron between two adjacent layers.

[0111] For example, in the case where layer l contains M neurons and the current batch contains N back-translated samples, the change in the second average activation value between layer l-1 and layer l is... It can be determined in the following ways: ; in, This represents the activation value of the i-th neuron in the l-th layer of the feedforward neural network on the n-th back-translated sample. This represents the activation value of the i-th neuron in the (l-1)-th layer of the feedforward neural network on the n-th back-translated sample. This second average activation value change quantifies the degree of change in the overall semantic representation of the l-th layer relative to its preceding layers in the high-resource language to low-resource language back-translation task. A larger value indicates that the layer is more active in the back-semantic back-mapping process.

[0112] In some implementations, the change in the second average activation value between adjacent layers can be calculated sequentially in the top network layer of a preset language model, as described above. The changes in the second average activation value of all top layers are then sorted in descending order, and the k2 layers with the largest changes are selected as the second semantic alignment key layers from high-resource language to low-resource language. For example, taking the LLaMA3-8B model, which contains 32 Transformer layers, the top 16 layers (layers 16 to 31) are used as the candidate range, and the corresponding values ​​for each layer are calculated... After selecting the value, choose the three largest layers (e.g.) , , The corresponding layers 20, 24, and 28 are used as the second key semantic alignment layers. In this way, the network layers most sensitive to the reverse semantic back-mapping from high-resource language to low-resource language can be accurately located, providing reliable layer position guidance for subsequent selective parameter adjustment of only the top key layers to establish the reverse semantic mapping.

[0113] In some implementations, a hierarchical threshold screening strategy can be used to determine the number of key layers for second semantic alignment. Specifically, a threshold for the change in activation value is preset, and all layers in the top network layer whose second average activation value change exceeds this threshold are considered as key layers for second semantic alignment, instead of selecting a fixed number of k2 layers. This makes the number of semantic alignment parameters more flexible and controllable, avoiding insufficient alignment due to too few layers or increased risk of knowledge forgetting due to too many layers.

[0114] By using the above method, the layer with the most dramatic activation changes can be selected from multiple network layers as the second semantic alignment key layer. This provides a precise layer location basis for subsequently implementing selective parameter adjustment only on the top key layer to establish a reverse semantic mapping from high-resource language to low-resource language. At the same time, it complements the forward semantic alignment key layer, laying the foundation for finally achieving complete bidirectional semantic alignment between low-resource language and high-resource language within the model.

[0115] In some implementations, to quantify the prediction accuracy of a pre-defined language model on a low-resource language to high-resource language translation task and thus provide training supervision signals for subsequent parameter tuning, a first loss value can be calculated to reflect the overall prediction bias of the model on the current translation task, thereby providing a standardized gradient update basis for subsequent parameter tuning of the semantic alignment key layer. For example, step 201, "determining the first loss based on the difference between the predicted first translated text and the first translated text label corresponding to the second language," may include: (201.1) For each first instruction sample, determine the multiple first word tags contained in the first translated text tag, and based on the predicted first translated text, determine the first prediction sub-probability of the preset language model for each corresponding first word tag; (201.2) For each first instruction sample, calculate the first prediction probability corresponding to the first translated text based on the multiple first prediction sub-probabilities corresponding to multiple first word tags; (201.3) Obtain multiple first prediction probabilities corresponding to multiple first instruction samples, and determine the first loss based on the mean of multiple first prediction probabilities.

[0116] The first word tag can be any of the basic semantic units contained in the word sequence of the first translated text tag.

[0117] The first predictor probability can be the probability value that the word predicted by the preset language model in the process of generating the first translation text word by word is the correct output for a certain first word label.

[0118] The first prediction probability can be the overall prediction probability value of a sample obtained by comprehensively calculating the first prediction sub-probabilities corresponding to all first word tags in a given first instruction sample.

[0119] Specifically, during the word-by-word generation and prediction of the first translated text by the pre-defined language model, the output layer generates a probability distribution covering all candidate words in the entire vocabulary at each generation position. Using the real word at that position in the first translated text label as an index, the corresponding probability value is extracted from this probability distribution, which is the first predicted sub-probability of that word label. For example, taking Swahili as the first language and English as the second language, the first translated text label is "He recentlylost against Raonic in the Brisbane Open." During the word-by-word prediction process, for the first word label "He", the model output layer generates a probability distribution of each candidate word in the vocabulary and extracts the probability value corresponding to "He" (e.g., 0.85) as the first predicted sub-probability of that word label; for the second word label "recently", the probability value corresponding to "recently" (e.g., 0.72) is extracted, and so on, until the predicted sub-probabilities of all word labels are extracted.

[0120] For example, for a first translated text label containing T word tags, the first predictor probabilities at each word position are multiplied together to obtain the predicted probability of the sample. Since multiplication may cause numerical underflow, in actual calculations, the log-likelihood method is usually used, that is, the logarithm of the first predictor probabilities of each word tag is taken and then summed to obtain the predicted log-likelihood value of the sample. For the i-th sample among N first instruction samples, its first predicted probability is the joint probability of the model's predicted probabilities of all word tags on that sample.

[0121] In some implementations, the first loss can be determined by first obtaining the first predicted probability (or log-likelihood value) of each of the N first instruction samples, and then calculating the negative logarithmic average of the predicted probabilities of all samples as the first loss value for the current batch. The following formula can be used for calculation: ; Where N is the number of samples in the first instruction sample; For the i-th first instruction sample, the input first language text and the corresponding second language translation text label are given. These are the parameters that need to be fine-tuned in the first semantic alignment key layer; For the model to perform on a given input and parameters Generate target translation text under the conditions The conditional probability is calculated. This loss function comprehensively and accurately quantifies the prediction bias of the pre-defined language model on low-resource language to high-resource language translation tasks, providing a reliable gradient descent direction for subsequent parameter adjustment of the first semantic alignment key layer.

[0122] In some implementations, when calculating the first predicted probability for each first instruction sample, a length-related normalization factor can be added to the loss function based on the word sequence length of the first translated text label of the sample. This eliminates the scale difference in loss calculation for samples of different lengths, ensuring that the loss value does not favor short or long texts, and thus more accurately reflects the model's translation quality for samples of different lengths.

[0123] By employing the above methods, the first loss can comprehensively and accurately reflect the prediction bias of the preset language model in the low-resource language to high-resource language translation task. This provides a reliable basis for adjusting the parameters of the first semantic alignment key layer using this loss, ensuring that selective semantic alignment training can effectively improve translation accuracy.

[0124] In some implementations, to quantify the prediction accuracy of a pre-defined language model on a high-resource language to low-resource language reverse translation task and thus provide training supervision signals for subsequent parameter tuning, a second loss value can be calculated to reflect the overall prediction bias of the model on the reverse translation task, thereby providing a standardized gradient update basis for subsequent parameter tuning of the second semantic alignment key layer. For example, step 203, "determining the second loss based on the difference between the predicted second translated text and the second translated text label corresponding to the first language," may include: (203.1) For each second instruction sample, determine the multiple second word labels contained in the second translated text label, and based on the predicted second translated text, determine the second predictor probability for each corresponding second word label by the first intermediate language model; (203.2) For each second instruction sample, calculate the second prediction probability corresponding to the predicted second translation text based on the multiple second prediction sub-probabilities corresponding to multiple second word tags; (203.3) Obtain multiple second prediction probabilities corresponding to multiple second instruction samples, and determine the second loss based on the mean of multiple second prediction probabilities.

[0125] The second word tag can be any of the basic semantic units contained in the word sequence of the second translated text tag. For example, when the second translated text tag is the Swahili phrase "Hivi karibuni alipoteza dhidi ya Raonickwenye Brisbane Open.", each word or sub-word obtained after decomposing the tag, such as "Hivi", "karibuni", and "alipoteza", can serve as a second word tag.

[0126] The second predictor probability can be the probability value that the word predicted by the preset language model in the process of generating the second translation text word by word is the correct output for a certain second word label.

[0127] The second prediction probability can be the overall prediction probability value of a sample obtained by combining the second prediction sub-probabilities corresponding to all second word tags in a certain second instruction sample. It can be used to measure the accuracy of the model's prediction of the complete translation sample in the back translation task.

[0128] Specifically, during the word-by-word generation and prediction of the second translated text by the pre-defined language model, the output layer generates a probability distribution covering all candidate words in the entire vocabulary at each generation position. Using the real word at that position in the second translated text label as an index, the corresponding probability value is extracted from this probability distribution, which is the second predicted sub-probability for that word label. For example, taking English as the second language and Swahili as the first language, the second translated text label is “Hivikaribuni alipoteza dhidi ya Raonic kwenye Brisbane Open.”. During the word-by-word prediction process, for the first word label “Hivi”, the model output layer generates a probability distribution of each candidate word in the vocabulary and extracts the probability value corresponding to “Hivi” (e.g., 0.78) as the second predicted sub-probability for that word label; for the second word label “karibuni”, the probability value corresponding to “karibuni” (e.g., 0.65) is extracted, and so on, until the predicted sub-probabilities for all word labels are extracted.

[0129] For example, for a second translated text label containing T word tags, the second predictor probabilities at each word position can be multiplied together to obtain the predicted probability of the sample. Since multiplication may cause numerical underflow, a log-likelihood approach is typically used in practice, which involves taking the logarithm of the second predictor probabilities of each word tag and summing the sums to obtain the predicted log-likelihood value of the sample. For the i-th sample among N second instruction samples, its second predicted probability is the joint probability of the model's predicted probabilities for all word tags on that sample.

[0130] Furthermore, the second loss can be determined as follows: First, obtain the second predicted probability (or log-likelihood value) for each of the N second instruction samples. Then, calculate the negative logarithmic average of the predicted probabilities of all samples, and use this as the second loss value for the current batch. The specific calculation can be performed using the following formula: ; Where N is the number of samples in the second instruction sample; For the i-th second instruction sample, the input second language text and the corresponding first language translation text label are given. The parameters that need to be fine-tuned in the second semantic alignment key layer; For the model to perform on a given input and parameters Generate target translation text under the conditions The conditional probability is calculated. This loss function comprehensively and accurately quantifies the prediction bias of the pre-defined language model in the high-resource language to low-resource language reverse translation task, providing a reliable gradient descent direction for subsequent parameter tuning of the second semantic alignment key layer.

[0131] For example, a Focal Loss mechanism can be introduced when calculating the second loss. Specifically, considering the highly uneven word frequency distribution in the first language (low-resource language), a higher loss contribution is given to low-frequency words that are more difficult to predict. This can be achieved by introducing a dynamic modulation factor based on the predictor probability on top of the standard negative log-likelihood loss. This modulation factor takes a larger value when the predictor probability is low, making the model pay more attention to the gradient update of difficult-to-predict samples, thereby effectively improving the model's ability to generate rare words in low-resource languages. For example, the second loss can be determined in the following way: ; Where N is the number of samples in the second instruction sample; Let be the word sequence length of the second translated text label in the i-th second instruction sample; The second predictor probability is the model's prediction of the correct second word label at the t-th word position of the i-th sample. This is a focusing parameter used to control the intensity of the modulation factor; its value is a real number greater than 0 (for example, taking the value of...). =2); As the modulation factor, when When the modulation factor is large, it approaches 0, weakening the loss contribution of easily classified samples. When the modulation factor is small, it approaches 1, thus preserving the loss contribution of hard-to-classify samples.

[0132] In this way, the second loss can comprehensively and accurately reflect the prediction bias of the preset language model in the high-resource language to low-resource language reverse translation task. This provides a reliable basis for adjusting the parameters of the second semantic alignment key layer using this loss to establish the reverse semantic mapping from high-resource language to low-resource language.

[0133] In some implementations, to quantify the response accuracy of the bidirectional semantically aligned model when performing a low-resource language general instruction task, thereby providing training supervision signals for subsequent comprehensive parameter optimization, a third loss value can be calculated to reflect the overall prediction bias of the model on the current general instruction task. This provides a standardized gradient update basis for subsequent overall fine-tuning of the parameters of multiple network layers of the model. For example, step 205, "determining the third loss based on the difference between the predicted language response text and the response text label," may include: (205.1) For each third instruction sample, determine the multiple third word tags contained in the response text tag, and based on the predicted language response text, determine the third predictor probability of the second intermediate language model predicting each corresponding third word tag; (205.2) For each third instruction sample, calculate the third prediction probability corresponding to the predicted language response text based on the multiple third prediction sub-probabilities corresponding to multiple third word tags; (205.3) Obtain multiple third prediction probabilities corresponding to multiple third instruction samples, and determine the third loss based on the mean of multiple third prediction probabilities.

[0134] The third word tag can be any of the basic semantic units contained in the word sequence of the response text tag. For example, when the response text tag is “The official languages ​​of Tanzania are Swahili and English”, each word or sub-word obtained after decomposing the tag, such as “Tanzania”, “of”, “official languages”, “is”, “Swahili”, “and”, and “English”, can be used as a third word tag. It can be used as the basic comparison unit for the prediction accuracy of the computational model in the general instruction task.

[0135] The third predictor probability can be the probability value that a word predicted by a pre-defined language model as the correct output for a certain third word tag during the process of generating predictive language response text word by word. For example, when the model generates the Swahili position, it predicts the normalized probability value of Swahili from all candidate words in the vocabulary. This can be used to measure the confidence of the model in predicting the position of a single word in a general instruction task.

[0136] The third prediction probability can be the overall prediction probability value of a sample obtained by comprehensively calculating the third prediction sub-probabilities corresponding to all third word tags in a given third instruction sample. It can be used to measure the accuracy of the model's prediction of the complete response sample in a general instruction task.

[0137] In some implementations, during the process of generating predictive language response text word by word, the output layer of the preset language model generates a probability distribution covering the entire vocabulary at each generation position. The real word at that position in the response text label is used as an index, and the corresponding probability value is extracted from the probability distribution, which is the third predictive sub-probability of that word label. For example, taking Swahili as the first language, the instruction text in the third instruction sample is “Je,mji mkuu waTanzania ni upi?”, and the corresponding response text label is “Dodoma ndiyo mji mkuu wa Tanzania.”. This instruction sample is input into the second intermediate language model, and the model generates the predicted language response text word by word. For the first word label “Dodoma”, the model output layer generates the probability distribution of each candidate word on the vocabulary, and extracts the probability value corresponding to “Dodoma” (e.g., 0.82) as the third predictor sub-probability of the word label; for the second word label “ndiyo”, the probability value corresponding to “ndiyo” (e.g., 0.74) is extracted, and so on, until the predictor sub-probabilities of all word labels in the response text label are extracted.

[0138] Furthermore, the conditional probability of the model generating the entire response sequence is equal to the product of the predicted probabilities at each word position. For a response text tag containing T third-word tags, the predicted probability of the sample can be obtained by multiplying the third-predictor probabilities at each word position. It should be noted that since multiplication may lead to numerical underflow, a log-likelihood approach can be used in actual calculations. This involves taking the logarithm of the third-predictor probabilities of each word tag and summing the results to obtain the predicted log-likelihood value of the sample. For the i-th sample among N third-instruction samples, its third-predictor probability is the joint probability of the model's predicted probabilities for all word tags on that sample.

[0139] In some implementations, the third loss The specific calculation can be performed using the following formula: ; Where N is the number of samples in the third instruction sample; Let i be the instruction text and the corresponding response text label in the i-th third instruction sample; These are the parameters that need to be fine-tuned in the bidirectional semantic alignment model; For the model to perform given input instructions and parameters Generate target response under the conditions The conditional probability is equal to the product of the probabilities of the word positions in the response sequence.

[0140] For example, assuming the current batch contains 4 third instruction samples (N=4), the negative average of the predicted log-likelihood value for each sample is taken to obtain the third loss value for this batch. This loss function can comprehensively and accurately quantify the prediction bias of the bidirectional semantically aligned model on low-resource language general instruction tasks, providing a reliable gradient descent direction for subsequent overall fine-tuning of the parameters of multiple network layers in the model.

[0141] For example, a multi-task joint training strategy can be introduced when fine-tuning general instructions for low-resource languages. Specifically, the third instruction samples can cover various task types such as translation, question answering, text summarization, and code generation. When calculating the third loss, different loss weights can be assigned to instruction samples of different task types, enabling the model to adaptively allocate learning capacity according to task difficulty and data volume during fine-tuning. This avoids overfitting the model on task types with a large number of training samples while maintaining sufficient generalization ability on task types with fewer training samples, thereby comprehensively improving the model's processing performance for various downstream tasks in low-resource languages.

[0142] By using the above methods, the third loss can comprehensively and accurately reflect the prediction bias of the model after bidirectional semantic alignment on the general instruction task of low-resource language. This provides a reliable basis for gradient descent direction and update magnitude for subsequent overall fine-tuning of the parameters of multiple network layers of the model using this loss to enhance the model's ability to follow low-resource language instructions and the accuracy of response. This ensures that the fine-tuning training of general instructions can be effectively carried out in the direction of improving the overall performance of the model on downstream tasks of low-resource language.

[0143] In some implementations, combined Figure 4 The overall process of the embodiments of this application will be introduced.

[0144] In some implementations, training data is first obtained from a parallel corpus, consisting of first instruction samples from the first language to the second language, second instruction samples from the second language to the first language, and general instruction samples from the first language. For example, the first language can be Swahili, a low-resource language to be enhanced, and the second language can be English, a high-resource language. Parallel sentence pairs are sampled from an open-source parallel corpus, and forward translation instructions and backward translation instructions are constructed according to preset instruction templates.

[0145] In some implementations, taking the LLaMA3-8B language model as an example, after acquiring the training data, the pre-defined language model can be used to perform forward encoding on both the forward and backward sets of translation instruction samples, obtaining the activation response value of each neuron in each feedforward neural network layer of the model, and calculating the average change in activation value between adjacent layers. Taking a model with a total of L layers as an example, for the l-th layer, the activation value of the i-th neuron on the n-th sample is denoted as... The change in the average activation value of this layer relative to the preceding layer. This can be determined by calculating the absolute average of the differences in activation values ​​across all samples and all neurons. The larger the value, the more active the layer is in the cross-language semantic conversion process and the more sensitive it is to semantic mapping. Subsequently, the average activation value changes of the bottom network layers are sorted by magnitude, and the k1 layers with the largest changes are selected as the first semantic alignment key layers to establish the forward semantic mapping from the first language to the second language; the average activation value changes of the top network layers are sorted by magnitude, and the k2 layers with the largest changes are selected as the second semantic alignment key layers to establish the reverse semantic back-mapping from the second language to the first language.

[0146] In some implementations, after determining the key semantic alignment layers in each direction, the parameters of the first key semantic alignment layer can be trained and optimized using first instruction samples, and the parameters of the second key semantic alignment layer can be trained and optimized using second instruction samples, achieving bidirectional alignment between the first and second language representation spaces in the model. It should be noted that during the entire training process, only the parameters of the selected key layers are updated, while the parameters of other network layers in the model are frozen, thus achieving cross-language semantic alignment without introducing additional parameters or significantly increasing training costs. Next, based on the bidirectional semantic alignment, the model is further fine-tuned using general instruction samples from the first language, enabling the model to better adapt to the instruction distribution of downstream tasks in the first language. For example, first language instruction data containing various task types such as translation, question answering, and text summarization is used as the third instruction sample to fine-tune the bidirectional semantically aligned model with full parameter fine-tuning or partial module fine-tuning, ultimately resulting in a target language model with accurate understanding and generation capabilities in the first language.

[0147] Through the above processing, this application can fully utilize the existing cross-lingual representation capabilities within a large multilingual model without increasing the model parameter size or significantly increasing training costs. It achieves a significant improvement in low-resource language processing performance through bidirectional semantic alignment and fine-tuning of general instructions. This application is widely applicable to cross-lingual knowledge transfer tasks between different language pairs, providing an efficient and practical solution for the application of large multilingual models in low-resource language scenarios.

[0148] Please see Figure 5 This application also provides a data processing apparatus for low-resource languages, which can implement the above-described data processing method for low-resource languages. The data processing apparatus for low-resource languages ​​includes: Module 51 is used to obtain the pending prompt data corresponding to the first language; Input module 52 is used to input the prompt data to be processed into the preset target language model to obtain the corresponding processing result; The target language model is obtained by training a pre-defined language model based on the first loss, the second loss, and the third loss. The first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model. The first semantic alignment key layer is selected by the activation values ​​of the network layer in the process of translating the input first language text sample into the predicted second language text according to the preset language model. The second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model. The second semantic alignment key layer is selected based on the activation values ​​of the network layer during the process of translating the input second language text sample into the predicted first language text by the preset language model. The first semantic alignment key layer and the second semantic alignment key layer belong to different network layers, and the corpus size corresponding to the second language is larger than the corpus size corresponding to the first language. The third loss is used to indicate the adjustment of parameters of multiple network layers contained in the preset language model. The third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label.

[0149] The specific implementation of the data processing device for low-resource languages ​​is basically the same as the specific embodiment of the data processing method for low-resource languages ​​described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the data processing device for low-resource languages ​​may also be provided with other functional modules to implement the data processing method for low-resource languages ​​in the above embodiments.

[0150] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned data processing method for low-resource languages. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0151] Please see Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 61 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 62 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 62 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 62 and is called and executed by the processor 61 using the data processing method for low-resource languages ​​in the embodiments of this application. Input / output interface 63 is used to implement information input and output; The communication interface 64 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 65 transmits information between various components of the device (e.g., processor 61, memory 62, input / output interface 63, and communication interface 64); The processor 61, memory 62, input / output interface 63, and communication interface 64 are interconnected within the device via bus 65.

[0152] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data processing method for low-resource languages.

[0153] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0154] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0155] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0158] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0159] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0160] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0161] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A data processing method for low-resource languages, characterized in that, The method includes: Retrieve the pending prompt data corresponding to the first language; The pending prompt data is input into a preset target language model to obtain the corresponding processing result; The target language model is obtained by training a preset language model based on a first loss, a second loss, and a third loss. The first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model. The first semantic alignment key layer is selected based on the activation values ​​of the network layers during the process of translating the input first language text sample into the predicted second language text by the preset language model. The second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model. The second semantic alignment key layer is selected based on the activation values ​​of the network layer during the process of translating the input second language text sample into the predicted first language text by the preset language model. The first semantic alignment key layer and the second semantic alignment key layer belong to different network layers. The corpus size corresponding to the second language is larger than the corpus size corresponding to the first language. The third loss is used to indicate the adjustment of parameters of multiple network layers included in the preset language model. The third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label.

2. The data processing method for low-resource languages ​​according to claim 1, characterized in that, The target language model is trained in the following way: Obtain a first instruction sample corresponding to the first language, input the first instruction sample into a preset language model to obtain a predicted first translation text corresponding to the second language, and determine a first loss based on the difference between the predicted first translation text and the first translation text label corresponding to the second language; From the multiple network layers included in the preset language model, at least one first semantic alignment key layer is determined. Based on the first loss, the parameters of the at least one first semantic alignment key layer are adjusted to obtain a first intermediate language model. Obtain a second instruction sample corresponding to the second language, input the second instruction sample into the first intermediate language model for translation, and obtain a predicted second translated text corresponding to the first language. Based on the difference between the predicted second translated text and the label of the second translated text corresponding to the first language, determine the second loss. At least one second semantic alignment key layer is determined from the plurality of network layers, and the parameters of the at least one second semantic alignment key layer are adjusted based on the second loss to obtain a second intermediate language model; Obtain a third instruction sample corresponding to the first language, input the third instruction sample into the second intermediate language model to obtain the predicted language response text corresponding to the first language, and determine the third loss based on the difference between the predicted language response text and the response text label; The parameters of the second intermediate language model are adjusted based on the third loss to obtain the target language model.

3. The data processing method for low-resource languages ​​according to claim 2, characterized in that, The step of determining at least one first semantic alignment key layer from the multiple network layers included in the preset language model includes: Acquire a first language text sample and first instruction data that instructs the first language text sample to be translated into a second language text; In the process of obtaining the first activation value of each first neuron in each network layer during the translation of the preset language model into predicted second language text based on the first language text sample and the first instruction data; For each network layer, based on the difference between the first activation value corresponding to each first neuron and the second activation value corresponding to each second neuron in the adjacent preceding network layer, the first activation difference value corresponding to each first neuron is calculated. For each network layer, the change in the corresponding first average activation value is calculated based on the multiple first activation difference values ​​corresponding to multiple first neurons; Based on the magnitude relationship between the changes in multiple first average activation values ​​corresponding to multiple network layers, at least one first semantic alignment key layer with a larger change in the first average activation value is determined from the multiple network layers.

4. The data processing method for low-resource languages ​​according to claim 2, characterized in that, The step of determining at least one corresponding second semantic alignment key layer from the plurality of network layers includes: Acquire second language text samples and second instruction data that instructs the second language text samples to be translated into first language text; In the process of obtaining the preset language model based on the second language text sample and the second instruction data to predict the first language text, the third activation value of each third neuron contained in each network layer; For each network layer, based on the difference between the third activation value corresponding to each third neuron and the fourth activation value corresponding to each fourth neuron in the adjacent preceding network layer, a second activation difference value is calculated for each third neuron. For each network layer, the change in the corresponding second average activation value is calculated based on the multiple second activation difference values ​​corresponding to multiple third neurons; Based on the magnitude relationship between the changes in multiple second average activation values ​​corresponding to multiple network layers, at least one second semantic alignment key layer with a larger change in the second average activation value is determined from the multiple network layers.

5. The data processing method for low-resource languages ​​according to claim 2, characterized in that, The determination of the first loss based on the difference between the predicted first translated text and the first translated text label corresponding to the second language includes: For each first instruction sample, determine multiple first word tags contained in the first translated text tag, and based on the predicted first translated text, determine the first prediction sub-probability of the preset language model for each corresponding first word tag; For each first instruction sample, the first prediction probability corresponding to the predicted first translated text is calculated based on the multiple first prediction sub-probabilities corresponding to the multiple first word tags; Obtain multiple first prediction probabilities corresponding to multiple first instruction samples, and determine the first loss based on the mean of the multiple first prediction probabilities.

6. The data processing method for low-resource languages ​​according to claim 2, characterized in that, The determination of the second loss based on the difference between the predicted second translated text and the second translated text label corresponding to the first language includes: For each second instruction sample, determine multiple second word tags contained in the second translated text tag, and based on the predicted second translated text, determine the second prediction sub-probability of the first intermediate language model for each corresponding second word tag; For each second instruction sample, the second prediction probability corresponding to the predicted second translated text is calculated based on the multiple second prediction sub-probabilities corresponding to the multiple second word tags; Obtain multiple second prediction probabilities corresponding to multiple second instruction samples, and determine a second loss based on the mean of the multiple second prediction probabilities.

7. The data processing method for low-resource languages ​​according to claim 2, characterized in that, The determination of the third loss based on the difference between the predicted language response text and the response text label includes: For each third instruction sample, determine multiple third word tags contained in the response text tag, and based on the predicted language response text, determine the third predictor probability for each corresponding third word tag predicted by the second intermediate language model; For each third instruction sample, the third prediction probability corresponding to the predicted language response text is calculated based on the multiple third prediction sub-probabilities corresponding to the multiple third word tags; Obtain multiple third prediction probabilities corresponding to multiple third instruction samples, and determine the third loss based on the mean of the multiple third prediction probabilities.

8. A data processing device for low-resource languages, characterized in that, The device includes: The acquisition module is used to acquire the pending prompt data corresponding to the first language; The input module is used to input the prompt data to be processed into a preset target language model to obtain the corresponding processing result; The target language model is obtained by training a preset language model based on a first loss, a second loss, and a third loss. The first loss is used to indicate the adjustment of the parameters of at least one first semantic alignment key layer in the preset language model. The first semantic alignment key layer is selected based on the activation values ​​of the network layers during the process of translating the input first language text sample into the predicted second language text by the preset language model. The second loss is used to indicate the adjustment of the parameters of at least one second semantic alignment key layer in the preset language model. The second semantic alignment key layer is selected based on the activation values ​​of the network layer during the process of translating the input second language text sample into the predicted first language text by the preset language model. The first semantic alignment key layer and the second semantic alignment key layer belong to different network layers. The corpus size corresponding to the second language is larger than the corpus size corresponding to the first language. The third loss is used to indicate the adjustment of parameters of multiple network layers included in the preset language model. The third loss is determined based on the difference between the predicted language response text output by the preset language model and the corresponding response text label.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data processing method for low-resource languages ​​as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method for low-resource languages ​​as described in any one of claims 1 to 7.

Images