Text processing method, fine tuning method of pre-training language model and electronic equipment

Abstract

The invention discloses a text processing method, a fine tuning method of a pre-training language model and electronic equipment, and relates to the field of large model technology and cross-language migration. The method comprises the steps of obtaining target text data of a target language; processing the target text data by using a pre-training language model inserted with the target model component to obtain a target processing result of the target text data under the target task, the pre-training language model being obtained by pre-training by using a source training text of a source language, the target model component is obtained by combining the target task component and the target language component, and the pre-training language model inserted with the target task component is used for processing the source text data of the source language to obtain a processing result of the source text data under the target task. The technical problem that cross-language migration accuracy of a large language model is low in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the fields of large model technology and cross-language migration, and more specifically, to a text processing method, a method for fine-tuning a pre-trained language model, and an electronic device. Background Art

[0002] Existing language models mainly use aligned representation or translation into an intermediate language when performing cross-language migration, but the above solutions are not suitable for large language models. As for the optimization hint method for large language models, due to the optimization space priority, there will be certain limitations when applied to low-resource languages, which leads to the limited processing capacity of large language models when performing cross-language migration, and further, it will also lead to low accuracy of cross-language migration of large language models.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a text processing method, a fine-tuning method for a pre-trained language model, and an electronic device to at least solve the technical problem of low cross-language migration accuracy of large language models in the related art.

[0005] According to one aspect of an embodiment of the present application, a text processing method is provided, including: acquiring target text data in a target language; processing the target text data using a pretrained language model into which a target model component is inserted, to obtain a target processing result of the target text data under a target task, wherein the pretrained language model is pretrained using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pretrained language model into which the target task component is inserted is used to process source text data in a source language to obtain a processing result of the source text data under the target task, and the pretrained language model into which the target language component is inserted is used to process the target text data to obtain a processing result of the target text data under a preset task.

[0006] According to another aspect of an embodiment of the present application, a method for fine-tuning a pre-trained language model is also provided, comprising: obtaining a target training text in a target language and a first processing result of a source training text in a source language under a target task; using the source training sample and the first processing result, training a pre-trained language model into which an initial task component is inserted to obtain a target task component, wherein the pre-trained language model is obtained by pre-training using the source training text; using the difference between the target training text and the source training text, training the pre-trained language model into which the initial language component is inserted to obtain a target language component; merging the target task component and the target language component to obtain a target model component; inserting the target model component into the pre-trained language model to fine-tune the pre-trained language model, wherein the pre-trained language model into which the target model component is inserted is used to process target text data in the target language.

[0007] According to another aspect of an embodiment of the present application, a text processing method is also provided, including: responding to an input instruction acting on an operation interface, displaying target text data in a target language on the operation interface; responding to a processing instruction acting on the operation interface, displaying a target processing result of the target text data under a target task on the operation interface, wherein the target processing result is obtained by processing the target text data using a pre-trained language model with a target model component inserted therein, the pre-trained language model is pre-trained using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process source text data in a source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task.

[0008] According to another aspect of an embodiment of the present application, a text processing method is also provided, including: obtaining target text data of a target language by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the target text data; processing the target text data using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is pre-trained using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task; outputting the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target processing result.

[0009] According to another aspect of an embodiment of the present application, a text processing device is also provided, including: an acquisition module, used to acquire target text data in a target language; a processing module, used to process the target text data using a pre-trained language model with a target model component inserted therein, to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language, to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data, to obtain a processing result of the target text data under a preset task.

[0010] According to another aspect of an embodiment of the present application, a fine-tuning device for a pre-trained language model is also provided, including: an acquisition module, used to acquire a target training text in a target language, and a first processing result of a source training text in a source language under a target task; a first training module, used to train a pre-trained language model with an initial task component inserted therein using the source training sample and the first processing result, to obtain a target task component, wherein the pre-trained language model is obtained by pre-training using the source training text; a second training module, used to train the pre-trained language model with the initial language component inserted therein using the difference between the target training text and the source training text, to obtain a target language component; a merging module, used to merge the target task component and the target language component to obtain a target model component; and a fine-tuning module, used to insert the target model component into the pre-trained language model to fine-tune the pre-trained language model, wherein the pre-trained language model with the target model component inserted therein is used to process target text data in the target language.

[0011] According to another aspect of an embodiment of the present application, a text processing device is also provided, including: a first display module, used to respond to an input instruction acting on an operation interface, and display target text data in a target language on the operation interface; a second display module, used to respond to a processing instruction acting on the operation interface, and display a target processing result of the target text data under a target task on the operation interface, wherein the target processing result is obtained by processing the target text data using a pre-trained language model with a target model component inserted therein, the pre-trained language model is pre-trained using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process source text data in a source language, and obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data, and obtain a processing result of the target text data under a preset task.

[0012] According to another aspect of an embodiment of the present application, a text processing device is also provided, including: an acquisition module, used to acquire target text data of a target language by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the target text data; a processing module, used to process the target text data using a pre-trained language model with a target model component inserted therein, to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language, to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data, to obtain a processing result of the target text data under a preset task; an output module, used to output the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target processing result.

[0013] According to another aspect of an embodiment of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein any one of the above methods is executed when the program is running.

[0014] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above methods.

[0015] In an embodiment of the present application, a method is adopted in which target text data in a target language is obtained; a pre-trained language model with a target model component inserted therein is used to process the target text data to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task. It is easy to notice that by adjusting the capabilities of the pre-trained language model through the target task component and the target language component, the processing capability of the pre-trained language model in the cross-language transfer scenario can be improved. Secondly, by processing the target text data through the pre-trained language model with the target model component inserted after the capability enhancement, the purpose of improving the cross-language transfer accuracy of the pre-trained language model is achieved, thereby achieving the technical effect of accurate cross-language transfer of large language models, and then solving the technical problem of low cross-language transfer accuracy of large language models in related technologies.

[0016] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an application scenario of a text processing method according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of a text processing method according to Embodiment 1 of the present application;

[0020] Figure 3 is a schematic diagram of an optional pre-trained language model according to Example 1 of the present application;

[0021] Figure 4 is a flowchart of a method for fine-tuning a pre-trained language model according to Embodiment 2 of the present application;

[0022] Figure 5 is a flowchart of a text processing method according to Embodiment 3 of the present application;

[0023] Figure 6is a schematic diagram of an optional operation interface according to Example 3 of the present application;

[0024] Figure 7 is a flowchart of a text processing method according to Embodiment 4 of the present application;

[0025] Figure 8 is a schematic diagram of a text processing device according to Embodiment 5 of the present application;

[0026] Fig. 9 is a schematic diagram of a fine-tuning device for a pre-trained language model according to Example 6 of the present application;

[0027] Fig.10 is a schematic diagram of a text processing device according to Embodiment 7 of the present application;

[0028] Fig.11 is a schematic diagram of a text processing device according to Embodiment 8 of the present application;

[0029] Fig.12 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0033] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, the data processing is performed by pre-trained language models and target model components inserted in pre-trained language models in the large language model scenario of cross-language text migration as an example for explanation,

[0034] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0035] Adapter: Represents a module that is inserted into different layers of a pre-trained model for efficient model fine-tuning.

[0036] Adapter merging: merging the parameters of the same adapters obtained from different training data.

[0037] Cross-lingual transfer: transferring the abilities acquired from training in a large language to a small language.

[0038] LoRA: Low-Rank Adaptation of Large Language Models, is used to solve the problem of fine-tuning large language models. By focusing on the transformer attention block of a large language model, the fine-tuning quality of LoRA is comparable to that of the full model, while being faster and requiring less computation.

[0039] (IA)3: By suppressing and amplifying internal activation injection adapters, we scale the activation layer weights via a learned vector, leading to stronger performance while introducing only a relatively small number of new parameters.

[0040] Reference tasks: easily accessible tasks with training corpora available for low-resource languages, such as causal language modeling.

[0041] Example 1

[0042] According to an embodiment of the present application, a text processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above text processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a text processing method according to an embodiment of the present application. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0044] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device generates text data in a target language and sends it to the server; the server processes the text data using a pre-trained language model and a target model component inserted in the pre-trained language model, and returns the target processing result to the client device. It should be noted that, if the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be performed in the client device.

[0045] Under the above operating environment, this application provides Figure 2 The text processing method shown. Figure 2 1 is a flowchart of a text processing method according to Example 1 of the present application. Figure 2 As shown, the method may include the following steps:

[0046] Step S202: acquiring target text data in a target language.

[0047] The target language mentioned above may be a specific language, for example, a minority language, but is not limited thereto. The target text data mentioned above may be text data composed of characters of the target language, but is not limited thereto.

[0048] In an optional embodiment, when the target text data needs to be processed, the target text data of the input target language can be obtained first. For example, when the target language (such as French) needs to be translated, the data composed of French characters can be obtained first. For another example, when the key content of the target language (Italian) needs to be extracted, the data composed of Italian characters can be obtained first, but it is not limited to this.

[0049] Step S204, using the pre-trained language model with the target model component inserted therein to process the target text data, and obtain the target processing result of the target text data under the target task, wherein the pre-trained language model is obtained by pre-training using the source training text of the source language, the target model component is obtained by merging the target task component and the target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language, and obtain the processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data, and obtain the processing result of the target text data under the preset task.

[0050] The above-mentioned pre-trained language model can be a large language model trained in advance, wherein the large language model is trained by a large language, and the large language can be a language with a large number of users, or a high frequency of use, or a language with a large amount of text data, for example, it can be Chinese, English, etc., but not limited to this. It should be noted that the large language model usually performs tasks such as translation and content extraction of text data of a large language. If the large language model is directly used to process the target text data of a small language, it will lead to the problem of low processing accuracy of the target text data of the small language. Therefore, in this application, a target model component can be inserted into the pre-trained language model to improve the processing capability of the pre-trained language model for the target text data of the target language. The above-mentioned target model component can be an adapter inserted into the pre-trained language model, which is used to enhance the data processing capability of the pre-trained language model, so that the pre-trained language model with the adapter inserted can adapt to specific tasks, wherein the target model component can be obtained by merging parameters of multiple adapters. For multiple adapters of the same type, the parameters can be merged to improve the processing capability of the target model component. It should be noted that the insertion position of the target model component can default to all positions in the pre-trained language model where model components can be inserted. The above-mentioned target task can be a task that the user wants to perform on the text data of the target language, which may be different from the task of the pre-trained language model. For example, it can be translation, extraction, content generation, etc. of the text data of the target language, but it is not limited to this. The above-mentioned source language can be the default input language of the pre-trained language model, for example, it can be English or Chinese, but it is not limited to this. The above-mentioned source training text can be the text data of the source language obtained for training. The above-mentioned target task component can be a target task capability adapter, which is used to adjust the ability of the pre-trained language model to perform the target task. The above-mentioned target language component can be a target language capability adapter, which is used to adjust the ability of the pre-trained language model to process text data in the target language.

[0051] In an optional embodiment, since the target task of the target language may be different from the default task of the pre-trained language model, after acquiring the text data of the target language, the parameters of the target model component and the pre-trained language model can be first merged to obtain the merged parameters, and then the pre-trained language model with the target model component can be inserted to process the target text data based on the merged parameters under the target task, so that the target processing result of the target language under the target task can be obtained.

[0052] It should be noted that the pre-trained language model is obtained by pre-training using the source training text of the source language. The pre-trained language model with the target task component inserted is used to process the source text data of the source language to obtain the processing result of the source text data under the target task. The pre-trained language model with the target language component inserted is used to process the target text data to obtain the processing result of the target text data under the preset task.

[0053] It should be noted that before processing text data by inserting a target model component having a target model component, the method further includes: firstly merging parameters of a plurality of target task components and target language components of the same type to obtain a target model component.

[0054] For example, taking the translation of French into Italian as an example, when the French text data needs to be translated, first, the data composed of French characters can be obtained, and then the parameters of the target model component and the pre-trained language model can be merged to obtain the merged parameters, and then the French text data can be input into the large language model with the merged adapter, and the target text data can be translated under the target task by inserting the large language model with the merged adapter, that is, the Italian text can be obtained. In this step, the pre-trained language model is adjusted through the target task component and the target language component, and the processing capability of the pre-trained language model for the target language can be increased, and further, the processing accuracy and efficiency of the pre-trained language model for the target language can be improved.

[0055] In an embodiment of the present application, a method is adopted in which target text data in a target language is obtained; a pre-trained language model with a target model component inserted therein is used to process the target text data to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task. It is easy to notice that by adjusting the capabilities of the pre-trained language model through the target task component and the target language component, the processing capability of the pre-trained language model in the cross-language transfer scenario can be improved. Secondly, by processing the target text data through the pre-trained language model with the target model component inserted after the capability enhancement, the purpose of improving the cross-language transfer accuracy of the pre-trained language model is achieved, thereby achieving the technical effect of accurate cross-language transfer of large language models, and then solving the technical problem of low cross-language transfer accuracy of large language models in related technologies.

[0056] In the above embodiments of the present application, the target task component is obtained by training a pre-trained language model with an initial task component inserted therein using source training samples and the first processing result of the source training samples under the target task, and the target language component is obtained by training a pre-trained language model with an initial language component inserted therein using the difference between the target training text and the source training text of the target language.

[0057] The source training sample can be text data obtained in the source language for training, and the target training text can be text data obtained in the target language for training. The difference can be the difference in results obtained after the target language component performs the same task on the target training text and the source training text.

[0058] In an optional embodiment, first, the source training sample can be processed by a pre-trained language model under the target task to obtain a first processing result, and then the pre-trained language model with the initial task component inserted can be trained by the source training sample and the first processing result to obtain the target task component. At the same time, the pre-trained language model with the initial language component inserted can be trained by the difference between the target training text and the source training text in the target language to obtain the target language component.

[0059] In the above embodiments of the present application, the target language component is obtained by merging the first component and the second component based on the difference between the target training text and the source training text, wherein the first component is trained using the source training text, and the second component is trained using the target training text.

[0060] In an optional embodiment, the pre-trained language model with the initial first component inserted can be firstly trained with the source training text to obtain the first component, and the pre-trained language model with the initial second component inserted can be trained with the target training text to obtain the second component, and finally the parameters of the first component and the second component can be merged based on the difference to obtain the target language component.

[0061] In the above embodiments of the present application, the first component is obtained by training a pre-trained language model with the first initial component inserted using a source training text and a second processing result of the source training text under a preset task, and the second component is obtained by training a pre-trained language model with the second initial component inserted using a target training text and a third processing result of the target training text under a preset task.

[0062] The above-mentioned preset task can be a task set in advance by the user, and the complexity of the task can be lower than the complexity of the target task, thereby improving the training efficiency of the first component and the second component and simplifying the training difficulty while ensuring the training results.

[0063] In an optional embodiment, first, the source training text can be processed by a pre-trained language model under a preset task to obtain a second processing result, and secondly, the pre-trained language model with the first initial component inserted can be trained with the source training text and the second processing result to obtain the first component, and at the same time, the target training text can be processed by the pre-trained language model under the preset task to obtain a third processing result, and then the pre-trained language model with the second initial component inserted can be trained with the target training text and the third processing result to obtain the second component.

[0064] In the above embodiment of the present application, the method further includes: determining a merging method based on a target type of a target model component; and merging parameters of a target task component and a target language component based on the merging method.

[0065] In an optional embodiment, when merging the target task component and the target language component, the target type of the target model component can be determined first, wherein the target type is related to the structure of the target model component, and the target type may include but is not limited to: prefix-tuning, AdapterH, LoRA, and (IA)3. In this embodiment, LoRA and (IA)3 are used as examples for illustration. Next, a specific merging method can be determined based on the target type, and finally, the target task component and the target language component can be parameter-merged based on the merging method. For example, when it is determined that the target type of the target model component is LoRA or (IA)3, because the structures corresponding to LoRA and (IA)3 are different, different parameter merging methods need to be adopted.

[0066] It should be noted that after determining the target type based on the structure of the target model component, a model merging method based on structure adaptation can be implemented, and the structures of different models can be fully considered.

[0067] In the above embodiments of the present application, the merging method is determined based on the target type of the target model component, including: when the target type is a low-rank adaptive type, the merging method is determined to be element-by-element addition or subtraction; when the target type is an injection type by suppressing and amplifying internal activations, the merging method is determined to be element-by-element multiplication or division.

[0068] In an optional embodiment, when the target type is determined to be a low-rank adaptation type, the target task component and the target language component can be determined to be merged in an element-by-element addition or subtraction manner; when the target type is determined to be an injection type by suppressing and amplifying internal activations, the target task component and the target language component can be determined to be merged in an element-by-element multiplication or division manner.

[0069] For example, when the target type of the target model component is LoRA, it can be determined that the merging method corresponding to LoRA is element-by-element addition or subtraction merging, and then the target task component and the target language component can be merged by element addition or subtraction. For another example, when the target type of the target model component is (IA)3, it can be determined that the merging method corresponding to (IA)3 is element-by-element multiplication or division merging, and then the target task component and the target language component can be merged by element multiplication or division.

[0070] In the above embodiment of the present application, the target text data is processed using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, including: merging parameters of the target model component and the pre-trained language model to obtain merged parameters; and processing the target text data using the merged parameters to obtain a target processing result.

[0071] In an optional embodiment, first, the parameters of the target model component and the pre-trained language model can be merged to obtain merged parameters. Secondly, the acquired target text data can be input into the pre-trained language model with the target model component inserted. Then, the target text data can be processed by the pre-trained language model with the target model component inserted and the merged parameters to obtain the target processing result. Finally, the target processing result can be output through the model and presented to the user through a display interface.

[0072] Figure 3 is a schematic diagram of an optional pre-trained language model according to Example 1 of the present application, such as Figure 3As shown, first, the input query (Query, Q), key (Key, K) and value (Value, V) vectors can be received through the pre-trained language model, and the received text data can be directly input into the connection residual and layer normalization (Add&Norm). Secondly, Q and K can be input into the activation function, and then the output of the activation function and V can be input into the connection residual and layer normalization. After the connection residual and layer normalization are processed based on the received data, the processed data are sent to the feedforward neural network (Feed forward) and the connection residual and layer normalization at the same time. The connection residual and layer normalization process the received data to complete the training of the pre-trained language model. Secondly, the adapter at the center of the adapter can be fine-tuned (Fine Tuning) based on the trained pre-trained language model to obtain the adjusted adapter. Finally, the adjusted task capability adapter and language capability adapter are adapter merged (Adaptive Adapter Merging) to obtain the merged adapter (as shown in Figure 3 ), insert the merged adapter into the pre-trained language model. At the same time, multiple adapters of the same type in different languages ​​in the language capability adapter can also be merged. Finally, the target text data of the target language can be processed by inserting the large language model (i.e., the pre-trained language model) with the merged adapter to obtain the target processing result of the target language. Among them, the adapters in the adapter center may include but are not limited to: LoRA and (IA)3. The types of task capability adapters may include but are not limited to: LoRA and (IA)3. The types of language capability adapters may include but are not limited to: LoRA and (IA)3. The types of merged adapters may include but are not limited to: LoRA and (IA)3.

[0073] In this application, the ability of a large language model on a specific task and language can be regarded as a combination of "task ability" and "language ability". The former refers to the ability of the model to perform a specific task, while the latter refers to the model's proficiency in a given language.

[0074] In the case of efficient fine-tuning of large language model parameters, given that language capabilities are independent of any specific task, one can assume that the differences between adapters fine-tuned in different languages ​​on a specific task follow the same distribution across tasks.

[0075] In the context of cross-language transfer, language competence is quantified by the difference between adapters for tasks in the source and target languages. Task competence is represented by the adapter being fine-tuned to the target task in the source language. Based on the assumption that the difference between adapters fine-tuned in different languages ​​for a specific task follows a consistent distribution across tasks, another reference task (i.e., a pre-set task) can be introduced to obtain the adapter representing language competence.

[0076] Therefore, the present application introduces three types of adapters: an adapter adjusted for a target task in a source language (i.e., a target task component), an adapter adjusted for a reference task in a target language (i.e., a second component), and an adapter adjusted for a reference task in a source language (i.e., a first component). The first type of adapter represents task capability, while the difference between the latter two types represents language capability. By merging these three types of adapters, an adapter suitable for a target task in a target language can be obtained.

[0077] In addition, the model merging method should be consistent with the way the adapter is integrated with the language model. Specifically, for LoRA, which merges adapters via element-wise addition, the merging method should also involve element-wise addition or subtraction. In contrast, (IA)3 merges adapters with language models via element-wise multiplication, and the adapter merging method should also use element-wise multiplication or division.

[0078] Compared with traditional cross-language transfer, this application adopts a model merging method to fully utilize the potential of large language models. Compared with previous model fusion methods, a structure-adaptive model merging method is adopted, which fully considers the structures of different models.

[0079] The method of this application (AdaMergeX) is evaluated on a total of 5 multilingual tasks covering 12 languages, showing that AdaMergeX consistently outperforms other state-of-the-art methods in cross-lingual transfer via hints and adapter merging methods. Notably, compared to XLT, a method for implementing cross-lingual transfer methods on large language models, AdaMerges achieves a relative improvement of 67.3% and 11.3% in all languages ​​and all tasks, respectively, both without fine-tuning on English labeled data and with fine-tuning on English labeled data. Compared to the current adapter merging method Arimerge, AdaMergeX achieves an average relative improvement of 31.1% in all languages ​​and all tasks. In addition, ablation analysis shows that AdaMergeX consistently performs well across different models, different source languages, and different reference tasks.

[0080] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0081] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0083] Example 2

[0084] According to an embodiment of the present application, a method for fine-tuning a pre-trained language model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0085] Figure 4 is a flow chart of a method for fine-tuning a pre-trained language model according to Embodiment 2 of the present application. Figure 4 As shown, the method may include the following steps:

[0086] Step S402, obtaining a target training text in a target language and a first processing result of a source training text in a source language under a target task;

[0087] Step S404, using the source training sample and the first processing result, training the pre-trained language model into which the initial task component is inserted to obtain the target task component, wherein the pre-trained language model is obtained by pre-training using the source training text;

[0088] Step S406, using the difference between the target training text and the source training text, training the pre-trained language model into which the initial language component is inserted to obtain the target language component;

[0089] Step S408, merging the target task component and the target language component to obtain a target model component;

[0090] Step S4010, inserting the target model component into the pre-trained language model to fine-tune the pre-trained language model, wherein the pre-trained language model into which the target model component is inserted is used to process target text data in the target language.

[0091] In an optional embodiment, when the pre-trained language model needs to be fine-tuned, first, the target training text in the target language and the first processing result of the source training text in the source language under the target task can be obtained. Secondly, the source training sample and the first processing result can be used to train the pre-trained language model with the initial task component inserted to obtain the target task component, wherein the pre-trained language model is obtained by pre-training with the source training text. Then, the difference between the target training text and the source training text can be used to train the pre-trained language model with the initial language component inserted to obtain the target language component. Then, the target task component and the target language component can be merged to obtain the target model component. Finally, the target model component can be inserted into the pre-trained language model to fine-tune the pre-trained language model, wherein the pre-trained language model with the target model component inserted is used to process the target text data in the target language.

[0092] In the above embodiment of the present application, the difference between the target training text and the source training text is used to train the pre-trained language model into which the initial language component is inserted to obtain the target language component, including: using the source training text to train the pre-trained language model into which the first initial component is inserted to obtain the first component; using the target training text to train the pre-trained language model into which the second initial component is inserted to obtain the second component; using the difference between the target training text and the source training text, merging the first component and the second component to obtain the target language component.

[0093] In an optional embodiment, the pre-trained language model with the first initial component inserted can be first trained using the source training text to obtain the first component, and the pre-trained language model with the second initial component inserted can be trained using the target training text to obtain the second component. Finally, the parameters of the first component and the second component can be merged based on the difference to obtain the target language type component.

[0094] In the above embodiment of the present application, the pre-trained language model with the first initial component inserted is trained using the source training text to obtain the first component, including: obtaining the second processing result of the source training text under the preset task, and using the source training text and the second processing result to train the pre-trained language model with the first initial component inserted to obtain the first component; the pre-trained language model with the second initial component inserted is trained using the target training text to obtain the second component, including: obtaining the third processing result of the target training text under the preset task, and using the target training text and the third processing result to train the pre-trained language model with the second initial component inserted to obtain the second component.

[0095] In an optional embodiment, first, the second processing result of the source training text under the preset task can be obtained, and the source training text and the second processing result are used to train the pre-trained language model with the first initial component inserted to obtain the first component. At the same time, the third processing result of the target training text under the preset task can be obtained, and the target training text and the third processing result are used to train the pre-trained language model with the second initial component inserted to obtain the second component.

[0096] In the above embodiment of the present application, the target task component and the target language component are merged to obtain a target model component, including: determining a merging method based on a target type of the target task component and a target type of the target language component; and merging parameters of the target task component and the target language component based on the merging method to obtain a target model component.

[0097] In an optional embodiment, firstly, a merging mode may be determined based on a target type of a target task component and a target type of a target language component, and secondly, the target task component and the target language component may be merged based on the merging mode to obtain a target model component.

[0098] Example 3

[0099] According to an embodiment of the present application, a text processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0100] Figure 5 is a flowchart of a text processing method according to Example 3 of the present application. Figure 5 As shown, the method may include the following steps:

[0101] Step S502, in response to an input instruction on the operation interface, displaying target text data in the target language on the operation interface;

[0102] Step S504, in response to the processing instruction acting on the operation interface, the target processing result of the target text data under the target task is displayed on the operation interface, wherein the target processing result is obtained by processing the target text data using a pre-trained language model with a target model component inserted therein, the pre-trained language model is pre-trained using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language to obtain the processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain the processing result of the target text data under the preset task.

[0103] In an optional embodiment, Figure 6 is a schematic diagram of an optional operation interface according to Example 3 of the present application, such as Figure 6 As shown, the operation interface includes: an input button, a data processing button and a display area. When the target text data needs to be processed, the user can first perform an operation on the input button. Secondly, in response to the input instruction generated based on the operation, the operation interface can display the target text data in the target language in the display area. Then the user can perform an operation on the data processing button on the operation interface. Then, in response to the processing instruction generated based on the user operation, the operation interface can display the target processing result of the target text data under the target task in the display area, wherein the target processing result is obtained by processing the target text data using a pre-trained language model with a target model component inserted therein, the pre-trained language model is pre-trained using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language to obtain the processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain the processing result of the target text data under the preset task.

[0104] Example 4

[0105] According to an embodiment of the present application, a text processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0106] Figure 7 4 is a flowchart of a text processing method according to Example 4 of the present application. Figure 7 As shown, the method may include the following steps:

[0107] Step S702, acquiring target text data in a target language by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the target text data;

[0108] Step S704, using the pre-trained language model with the target model component inserted therein to process the target text data, and obtain the target processing result of the target text data under the target task, wherein the pre-trained language model is obtained by pre-training using the source training text of the source language, the target model component is obtained by merging the target task component and the target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language, and obtain the processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data, and obtain the processing result of the target text data under the preset task;

[0109] Step S706: output the target processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target processing result.

[0110] The first interface mentioned above may be an interface for the client device to obtain text data in the target language from the server, and the second interface mentioned above may be an interface for the client device to output the target processing result to the server.

[0111] In an optional embodiment, when the target text data needs to be processed, the target text data in the target language can first be obtained from the server through the first interface. Secondly, the server can use a pre-trained language model with a target model component inserted to process the target text data to obtain the target processing result of the target text data under the target task. Finally, the server can return the target processing result to the client device through the second interface.

[0112] It should be noted that the first interface includes a first parameter, the parameter value of the first parameter includes the target text data, the pre-trained language model is obtained by pre-training using the source training text of the source language, the target model component is obtained by merging the target task component and the target language component, the pre-trained language model with the target task component inserted is used to process the source text data of the source language to obtain the processing result of the source text data under the target task, the pre-trained language model with the target language component inserted is used to process the target text data to obtain the processing result of the target text data under the preset task, and the second interface includes a second parameter, the parameter value of the second parameter includes the target processing result.

[0113] Example 5

[0114] According to an embodiment of the present application, a text processing device for implementing the above-mentioned text processing method is also provided. Figure 8 is a schematic diagram of a text processing device according to Embodiment 5 of the present application, such as Figure 8 As shown, the device includes: an acquisition module 82 and a processing module 84.

[0115] Among them, the acquisition module is used to acquire target text data in a target language; the processing module is used to process the target text data using a pre-trained language model with a target model component inserted therein, and obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language, and obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data, and obtain a processing result of the target text data under a preset task.

[0116] It should be noted that the acquisition module 82 and the processing module 84 correspond to steps S202 to S204 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may run in the server 10 provided in Example 1.

[0117] In the above embodiments of the present application, the target task component is obtained by training a pre-trained language model with an initial task component inserted therein using source training samples and the first processing result of the source training samples under the target task, and the target language component is obtained by training a pre-trained language model with an initial language component inserted therein using the difference between the target training text and the source training text of the target language.

[0118] In the above embodiments of the present application, the target language component is obtained by merging the first component and the second component based on the difference between the target training text and the source training text, wherein the first component is trained using the source training text, and the second component is trained using the target training text.

[0119] In the above embodiments of the present application, the first component is obtained by training a pre-trained language model with the first initial component inserted using a source training text and a second processing result of the source training text under a preset task, and the second component is obtained by training a pre-trained language model with the second initial component inserted using a target training text and a third processing result of the target training text under a preset task.

[0120] In the above embodiments of the present application, the device further includes: a determination module and a merging module.

[0121] Among them, the determination module is used to determine the merging method based on the target type of the target model component; the merging module is used to merge the parameters of the target task component and the target language component based on the merging method.

[0122] In the above embodiments of the present application, the determination module includes: a first determination unit and a second determination unit.

[0123] Among them, the first determination unit is used to determine that the merging method is element-by-element addition or subtraction when the target type is a low-rank adaptation type; the second determination unit is used to determine that the merging method is element-by-element multiplication or division when the target type is an injection type by suppressing and amplifying internal activation.

[0124] In the above embodiments of the present application, the processing module includes: a merging unit and a processing unit.

[0125] Among them, the merging unit is used to merge the parameters of the target model component and the pre-trained language model to obtain merged parameters; the processing unit is used to process the target text data using the merged parameters to obtain the target processing results.

[0126] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0127] Example 6

[0128] According to an embodiment of the present application, a device for fine-tuning a pre-trained language model for implementing the above-mentioned method for fine-tuning a pre-trained language model is also provided. Fig. 9 is a schematic diagram of a fine-tuning device for a pre-trained language model according to Example 6 of the present application, such as Fig. 9 As shown, the apparatus includes: an acquisition module 92 , a first training module 94 , a second training module 96 , a merging module 98 and a fine-tuning module 910 .

[0129] Among them, the acquisition module is used to obtain the target training text of the target language and the first processing result of the source training text of the source language under the target task; the first training module is used to use the source training sample and the first processing result to train the pre-trained language model inserted with the initial task component to obtain the target task component, wherein the pre-trained language model is obtained by pre-training with the source training text; the second training module is used to use the difference between the target training text and the source training text to train the pre-trained language model inserted with the initial language component to obtain the target language component; the merging module is used to merge the target task component and the target language component to obtain the target model component; the fine-tuning module is used to insert the target model component into the pre-trained language model to fine-tune the pre-trained language model, wherein the pre-trained language model inserted with the target model component is used to process the target text data of the target language.

[0130] It should be noted that the acquisition module 92, the first training module 94, the second training module 96, the merging module 98 and the fine-tuning module 910 correspond to steps S402 to S4010 in Example 2, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may be run in the server 10 provided in Example 1.

[0131] In the above embodiment of the present application, the second training module includes: a first training unit, a second training unit and a first merging unit.

[0132] Among them, the first training unit is used to train the pre-trained language model with the first initial component inserted using the source training text to obtain the first component; the second training unit is used to train the pre-trained language model with the second initial component inserted using the target training text to obtain the second component; and the first component and the second component are numerically merged using the difference between the target training text and the source training text to obtain the target language component.

[0133] In the above embodiment of the present application, the first training unit includes: a first acquisition subunit.

[0134] Among them, the first acquisition subunit is used to obtain the second processing result of the source training text under the preset task, and use the source training text and the second processing result to train the pre-trained language model inserted with the first initial component to obtain the first component.

[0135] In the above embodiment of the present application, the second training unit includes: a second acquisition subunit.

[0136] Among them, the second acquisition subunit is used to obtain the third processing result of the target training text under the preset task, and use the target training text and the third processing result to train the pre-trained language model inserted with the second initial component to obtain the second component.

[0137] In the above embodiment of the present application, the merging module includes: a determining unit and a second merging unit.

[0138] Among them, the determination unit is used to determine the merging method based on the target type of the target task component and the target type of the target language component; the second merging unit is used to merge the parameters of the target task component and the target language component based on the merging method to obtain the target model component.

[0139] Example 7

[0140] According to an embodiment of the present application, a text processing device for implementing the above-mentioned text processing method is also provided. Fig.10 is a schematic diagram of a text processing device according to Embodiment 7 of the present application, such as Fig.10 As shown, the device includes: a first display module 1002 and a second display module 1004 .

[0141] Among them, the first display module is used to respond to the input instructions acting on the operation interface, and display the target text data of the target language on the operation interface; the second display module is used to respond to the processing instructions acting on the operation interface, and display the target processing result of the target text data under the target task on the operation interface, wherein the target processing result is obtained by processing the target text data using a pre-trained language model with a target model component inserted, the pre-trained language model is pre-trained using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted is used to process the source text data of the source language to obtain the processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted is used to process the target text data to obtain the processing result of the target text data under the preset task.

[0142] It should be noted that the first display module 1002 and the second display module 1004 correspond to steps S502 to S504 in Example 3, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may run in the server 10 provided in Example 1.

[0143] Example 8

[0144] According to an embodiment of the present application, a text processing device for implementing the above-mentioned text processing method is also provided. Fig.11 is a schematic diagram of a text processing device according to Embodiment 8 of the present application, such as Fig.11 As shown, the device includes: an acquisition module 1102 , a processing module 1104 and an output module 1106 .

[0145] Among them, the acquisition module is used to obtain the target text data of the target language by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the target text data; the processing module is used to process the target text data using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task; the output module is used to output the target processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target processing result.

[0146] It should be noted that the acquisition module 1102, the processing module 1104 and the output module 1106 correspond to steps S702 to S706 in Example 4, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may run in the server 10 provided in Example 1.

[0147] Example 9

[0148] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0149] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0150] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the text processing method: obtaining target text data in a target language; processing the target text data using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task.

[0151] Optionally, Fig.12 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.12 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1202, a memory 1204, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0152] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the text processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned text processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain target text data in the target language; use the pre-trained language model inserted with the target model component to process the target text data to obtain the target processing result of the target text data under the target task, wherein the pre-trained language model is obtained by pre-training using the source training text of the source language, the target model component is obtained by merging the target task component and the target language component, the pre-trained language model inserted with the target task component is used to process the source text data of the source language to obtain the processing result of the source text data under the target task, and the pre-trained language model inserted with the target language component is used to process the target text data to obtain the processing result of the target text data under the preset task.

[0154] Optionally, the target task component is obtained by training a pre-trained language model with an initial task component inserted therein using source training samples and a first processing result of the source training samples under the target task, and the target language component is obtained by training a pre-trained language model with an initial language component inserted therein using the difference between a target training text and a source training text in the target language.

[0155] Optionally, the target language component is obtained by merging a first component and a second component based on the difference between the target training text and the source training text, wherein the first component is trained using the source training text and the second component is trained using the target training text.

[0156] Optionally, the first component is obtained by training a pre-trained language model with the first initial component inserted using source training text and a second processing result of the source training text under a preset task, and the second component is obtained by training a pre-trained language model with the second initial component inserted using target training text and a third processing result of the target training text under a preset task.

[0157] Optionally, the processor may also execute program codes of the following steps: determining a merging method based on a target type of a target model component; and merging parameters of a target task component and a target language component based on the merging method.

[0158] Optionally, the processor may also execute the program code of the following steps: when the target type is a low-rank adaptation type, determining the merging method as element-by-element addition or subtraction; when the target type is an injection type by suppressing and amplifying internal activation, determining the merging method as element-by-element multiplication or division.

[0159] Optionally, the processor may also execute the following program code: merging parameters of the target model component and the pre-trained language model to obtain merged parameters; and processing the target text data using the merged parameters to obtain target processing results.

[0160] By adopting the embodiment of the present application, a method is provided for obtaining target text data in a target language; processing the target text data using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data in the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task. It is easy to notice that by adjusting the capabilities of the pre-trained language model through the target task component and the target language component, the processing capability of the pre-trained language model in the cross-language transfer scenario can be improved. Secondly, by processing the target text data through the pre-trained language model with the target model component inserted after the capability enhancement, the purpose of improving the cross-language transfer accuracy of the pre-trained language model is achieved, thereby achieving the technical effect of accurate cross-language transfer of large language models, and then solving the technical problem of low cross-language transfer accuracy of large language models in related technologies.

[0161] It can be understood by those skilled in the art that Fig.12 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.12 It does not limit the structure of the above electronic device. For example, the computer terminal A may also include Fig.12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.12 Different configurations shown.

[0162] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0163] Example 10

[0164] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the text processing method provided in the first embodiment.

[0165] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0166] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining target text data in a target language; processing the target text data using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is pre-trained using a source training text in a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process source text data in a source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task.

[0167] Optionally, the target task component is obtained by training a pre-trained language model with an initial task component inserted therein using source training samples and a first processing result of the source training samples under the target task, and the target language component is obtained by training a pre-trained language model with an initial language component inserted therein using the difference between a target training text and a source training text in the target language.

[0168] Optionally, the target language component is obtained by merging a first component and a second component based on the difference between the target training text and the source training text, wherein the first component is trained using the source training text and the second component is trained using the target training text.

[0169] Optionally, the first component is obtained by training a pre-trained language model with the first initial component inserted using source training text and a second processing result of the source training text under a preset task, and the second component is obtained by training a pre-trained language model with the second initial component inserted using target training text and a third processing result of the target training text under a preset task.

[0170] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: determining a merging mode based on a target type of a target model component; and merging parameters of a target task component and a target language component based on the merging mode.

[0171] Optionally, in this embodiment, the storage medium is also configured to store program codes for performing the following steps: when the target type is a low-rank adaptation type, determining the merging method to be element-by-element addition or subtraction; when the target type is an injection type by suppressing and amplifying internal activation, determining the merging method to be element-by-element multiplication or division.

[0172] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: merging parameters of the target model component and the pre-trained language model to obtain merged parameters; and processing the target text data using the merged parameters to obtain a target processing result.

[0173] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0174] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0179] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A text processing method, characterized in that: include: Obtain target text data in the target language; The target text data is processed using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task.

2. The method according to claim 1, characterized in that: The target task component is obtained by training a pre-trained language model with an initial task component inserted therein using the source training sample and a first processing result of the source training sample under the target task, and the target language component is obtained by training a pre-trained language model with an initial language component inserted therein using the difference between a target training text of the target language and the source training text.

3. The method according to claim 2, characterized in that The target language component is obtained by merging a first component and a second component based on the difference between the target training text and the source training text, wherein the first component is obtained by training using the source training text, and the second component is obtained by training using the target training text.

4. The method according to claim 3, characterized in that The first component is obtained by training a pre-trained language model with a first initial component inserted therein using the source training text and a second processing result of the source training text under a preset task, and the second component is obtained by training a pre-trained language model with a second initial component inserted therein using the target training text and a third processing result of the target training text under the preset task.

5. The method according to claim 1, characterized in that The method further comprises: Determining a merging method based on a target type of the target model component; Parameters of the target task component and the target language component are merged based on the merging method.

6. The method according to claim 5, characterized in that Determining a merging method based on the target type of the target model component includes: When the target type is a low-rank adaptive type, determining that the merging mode is element-by-element addition or subtraction; In case the target type is a type of injection by suppressing and amplifying internal activations, the merging mode is determined to be element-wise multiplication or division.

7. The method according to claim 1, characterized in that The target text data is processed using a pre-trained language model into which a target model component is inserted to obtain a target processing result of the target text data under a target task, including: Merging parameters of the target model component and the pre-trained language model to obtain merged parameters; The target text data is processed using the merged parameters to obtain the target processing result.

8. A method for fine-tuning a pre-trained language model, characterized in that: include: Obtaining a target training text in a target language and a first processing result of a source training text in a source language under a target task; Using the source training sample and the first processing result, training a pre-trained language model into which the initial task component is inserted to obtain a target task component, wherein the pre-trained language model is obtained by pre-training using the source training text; Using the difference between the target training text and the source training text, the pre-trained language model into which the initial language component is inserted is trained to obtain the target language component; Merging the target task component and the target language component to obtain a target model component; The target model component is inserted into the pre-trained language model to fine-tune the pre-trained language model, wherein the pre-trained language model into which the target model component is inserted is used to process target text data of the target language.

9. The method according to claim 8, characterized in that Using the difference between the target training text and the source training text, training the pre-trained language model into which the initial language component is inserted to obtain the target language component includes: Using the source training text to train the pre-trained language model into which the first initial component is inserted, to obtain a first component; Using the target training text to train the pre-trained language model into which the second initial component is inserted, to obtain a second component; The first component and the second component are numerically merged by utilizing the difference between the target training text and the source training text to obtain the target language component.

10. The method according to claim 9, characterized in that Using the source training text to train the pre-trained language model into which the first initial component is inserted to obtain the first component, comprising: obtaining a second processing result of the source training text under a preset task, and using the source training text and the second processing result to train the pre-trained language model into which the first initial component is inserted to obtain the first component; The pre-trained language model with the second initial component inserted is trained using the target training text to obtain the second component, including: obtaining a third processing result of the target training text under the preset task, and using the target training text and the third processing result to train the pre-trained language model with the second initial component inserted to obtain the second component.

11. The method according to claim 10, characterized in that The target task component and the target language component are merged to obtain a target model component, including: Determining a merging method based on a target type of the target task component and a target type of the target language component; The target task component and the target language component are parameter-merged based on the merging method to obtain the target model component.

12. A text processing method, characterized in that: include: In response to an input instruction acting on the operation interface, displaying target text data in a target language on the operation interface; In response to a processing instruction acting on the operation interface, a target processing result of the target text data under the target task is displayed on the operation interface, wherein the target processing result is obtained by processing the target text data using a pre-trained language model with a target model component inserted therein, the pre-trained language model is pre-trained using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language to obtain the processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain the processing result of the target text data under a preset task.

13. A text processing method, characterized in that: include: Acquire target text data in a target language by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the target text data; The target text data is processed using a pre-trained language model with a target model component inserted therein to obtain a target processing result of the target text data under a target task, wherein the pre-trained language model is obtained by pre-training using a source training text of a source language, the target model component is obtained by merging a target task component and a target language component, the pre-trained language model with the target task component inserted therein is used to process the source text data of the source language to obtain a processing result of the source text data under the target task, and the pre-trained language model with the target language component inserted therein is used to process the target text data to obtain a processing result of the target text data under a preset task; The target processing result is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target processing result.

14. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 13 when running.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 13.

Images