Model training method and apparatus, and storage medium

Abstract

The application relates to the field of transfer learning in the field of artificial intelligence technology, in particular to a model training method and device and a storage medium. The method comprises the following steps: obtaining first training data according to training data of a labeled auxiliary language; inputting the first training data into a first model to obtain a plurality of first prediction results; determining whether the first model meets a training convergence condition according to the plurality of first prediction results; in the case that the training convergence condition is not met, inputting training data of a target language into the first model to obtain pseudo-labeling, adding the training data of the target language including the pseudo-labeling into the first training data, repeatedly executing the step of obtaining the first training data according to the training data of the labeled auxiliary language and the following steps until the training convergence condition is met, and obtaining a trained first model. According to the embodiment of the application, the prediction accuracy of a target natural language understanding model on the target language can be realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus and storage medium. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Transfer learning, as an important field in artificial intelligence, is gaining increasing attention because it can leverage knowledge from similar source domains to aid in learning tasks in the target domain. Current mainstream natural language understanding models based on transfer learning perform well in resource-rich languages; however, for less common languages ​​with limited resources, the lack of sufficient labeled data hinders training, resulting in lower prediction accuracy and impacting the user experience of related products. Summary of the Invention

[0004] In view of this, a model training method, apparatus and storage medium are proposed.

[0005] In a first aspect, embodiments of this application provide a model training method, comprising: obtaining first training data based on training data of an annotated auxiliary language; inputting the first training data into a first model to obtain multiple first prediction results, wherein the first model includes multiple adapters, each adapter is connected to a network module, each adapter performs different dimensional changes internally, and each adapter and the corresponding network module outputs a first prediction result of the first training data; determining whether the first model meets the training convergence condition based on the multiple first prediction results; if the training convergence condition is not met, inputting training data of the target language into the first model to obtain pseudo-annotations, adding training data of the target language including pseudo-annotations to the first training data, and repeating the steps of obtaining the first training data based on the training data of the annotated auxiliary language and subsequent steps until the training convergence condition is met to obtain a trained first model, wherein the first model is used for natural language understanding of the target language, and the auxiliary language is different from the target language.

[0006] According to the embodiments of this application, by obtaining first training data, inputting the first training data into a first model, multiple first prediction results are obtained. Based on the multiple first prediction results, it is determined whether the first model meets the training convergence condition. If the training convergence condition is not met, the training data of the target language is input into the first model to obtain pseudo-labels. The training data of the target language including pseudo-labels is added to the first training data. The steps of obtaining the first training data based on the training data of the labeled auxiliary language and subsequent steps are repeated until the training convergence condition is met, and a trained first model is obtained. This can realize the process of self-supervised learning of the model. Through the process of self-iterative training, pseudo-labels can be obtained and the accuracy of pseudo-labels can be continuously optimized without manual labeling, reducing noise. By determining the pseudo-labels, the model can also meet the conditions of self-supervised learning, further improving the accuracy of the model. The first model includes multiple adapters, each adapter is connected to a network module, and different dimensional changes are performed inside each adapter. Each adapter and the corresponding network module output a first prediction result of the first training data. The multiple first prediction results output can be used to achieve voting, making greater use of existing training resources and improving the performance of the model.

[0007] According to the first aspect, in a first possible implementation of the model training method, different dimensional changes are performed within each adapter, including: each adapter compresses and restores the dimension of the input vector, with each adapter having a different compression factor.

[0008] According to the embodiments of this application, it is possible to achieve model outputs that are not completely identical, realize the ensemble learning of the model, make greater use of existing training resources, and improve the accuracy of the model.

[0009] According to the first aspect, in a second possible implementation of the model training method, obtaining the first training data based on the labeled auxiliary language training data includes: replacing the auxiliary language training data with a predetermined probability in the labeled auxiliary language training data with the target language training data according to the correspondence between the auxiliary language and the target language, while retaining the original auxiliary language annotations, to obtain the first training data.

[0010] According to the embodiments of this application, by replacing the training data of the auxiliary language with the training data of the target language according to a predetermined probability, while retaining the original annotations of the auxiliary language, the noise contained in the prediction results can be reduced when pseudo-annotations are obtained directly without replacement, thereby improving the robustness and accuracy of the model.

[0011] According to the first aspect or the first or second possible implementation of the first aspect, in the third possible implementation of the model training method, inputting the training data of the target language into the first model to obtain pseudo-labels includes: inputting the training data of the target language into the first model to obtain multiple second prediction results corresponding to each adapter; when the multiple second prediction results are the same and the corresponding confidence scores are all greater than the first threshold, determining the second prediction result as a pseudo-label of the input training data of the target language.

[0012] According to the embodiments of this application, pseudo-labels can be selected by voting from multiple second prediction results corresponding to the adapter, so that the determined pseudo-labels have higher accuracy and less noise, and can be better applied to the training of the model, resulting in a more accurate trained model.

[0013] According to the first aspect, in a fourth possible implementation of the model training method, the method further includes: training a second model to obtain a trained second model, wherein the second model includes multiple adapters, each adapter is connected to a network module, each adapter has different dimensional changes, and the training task of the second model is a masked language model (MLM) task; adjusting the adapters and network modules in the trained second model to adapters and network modules for natural language understanding to obtain the first model.

[0014] According to the embodiments of this application, by training the second model according to the MLM task, the model can achieve better results in both the auxiliary language and the target language. Furthermore, by adjusting the adapter and network modules in the trained second model to adapters and network modules for natural language understanding, the first model is obtained. Since the second model has been trained on the MLM task, the first model can better adapt to the training of the natural language understanding task of the target language, resulting in better performance of the trained model and saving training resources.

[0015] According to the fourth possible implementation of the first aspect, in the fifth possible implementation of the model training method, the second model further includes a feature extraction module, a linear transformation module, and a feature conversion module. The feature extraction module is used to extract feature vectors from the training data. The linear transformation module is used to convert the first labeled feature vector in the extracted feature vector to a bilingual feature space of the auxiliary language and the target language to obtain a second labeled feature vector, which represents the features of words in the training data. The feature conversion module is used to convert the second labeled feature vector and output it to each adapter.

[0016] According to the embodiments of this application, since the first model is trained on the auxiliary language and the target language, by transforming the first labeled feature vector in the extracted feature vector to the bilingual feature space of the auxiliary language and the target language, a second labeled feature vector is obtained. The labeled feature vector represents the features of words in the training data, which can make the second model perform better on the auxiliary language and the target language. Thus, after the first model is obtained based on the second model, the first model performs better on the auxiliary language and the target language. The model is more targeted during the training process, so that the finally trained model can perform better on the target language and the model's inference results are more accurate.

[0017] Secondly, embodiments of this application provide a model training apparatus, comprising: a first determining module, configured to obtain first training data based on training data of an annotated auxiliary language; a second determining module, configured to input the first training data into a first model to obtain multiple first prediction results, wherein the first model includes multiple adapters, each adapter is connected to a network module, each adapter undergoes different dimensional changes, and each adapter and corresponding network module outputs a first prediction result of the first training data; a judging module, configured to determine whether the first model satisfies the training convergence condition based on the multiple first prediction results; and a third determining module, configured to, if the training convergence condition is not met, input training data of the target language into the first model to obtain pseudo-annotations, add training data of the target language including pseudo-annotations to the first training data, and repeatedly execute the steps of obtaining the first training data based on the training data of the annotated auxiliary language and subsequent steps until the training convergence condition is met to obtain a trained first model, wherein the first model is used for natural language understanding of the target language, and the auxiliary language is different from the target language.

[0018] According to the second aspect, in a first possible implementation of the model training device, different dimensional changes are performed within each adapter, including: each adapter compresses and restores the dimension of the input vector, with each adapter having a different compression factor.

[0019] According to the second aspect, in a second possible implementation of the model training device, obtaining the first training data based on the labeled auxiliary language training data includes: replacing the auxiliary language training data with a predetermined probability in the labeled auxiliary language training data with the target language training data according to the correspondence between the auxiliary language and the target language, while retaining the original auxiliary language annotations, to obtain the first training data.

[0020] According to the second aspect or the first or second possible implementation of the second aspect, in the third possible implementation of the model training device, inputting the training data of the target language into the first model to obtain pseudo-labels includes: inputting the training data of the target language into the first model to obtain multiple second prediction results corresponding to each adapter; when the multiple second prediction results are the same and the corresponding confidence scores are all greater than the first threshold, determining the second prediction result as a pseudo-label of the input training data of the target language.

[0021] According to the second aspect, in a fourth possible implementation of the model training device, the device further includes: a training module for training a second model to obtain a trained second model, wherein the second model includes multiple adapters, each adapter is connected to a network module, and each adapter undergoes different dimensional changes, and the training task of the second model is a masked language modeling (MLM) task; and an adjustment module for adjusting the adapters and network modules in the trained second model to adapters and network modules for natural language understanding to obtain the first model.

[0022] According to the fourth possible implementation of the second aspect, in the fifth possible implementation of the model training device, the second model further includes a feature extraction module, a linear transformation module, and a feature conversion module. The feature extraction module is used to extract feature vectors from the training data. The linear transformation module is used to convert the first labeled feature vector in the extracted feature vector to a bilingual feature space of the auxiliary language and the target language to obtain a second labeled feature vector, which represents the features of words in the training data. The feature conversion module is used to convert the second labeled feature vector and output it to each adapter.

[0023] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement one or more of the model training methods described in the first aspect or various possible implementations of the first aspect when executing the instructions.

[0024] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the model training methods described in the first aspect or various possible implementations of the first aspect.

[0025] Fifthly, embodiments of this application provide a terminal device that can execute one or more of the model training methods described in the first aspect or various possible implementations of the first aspect.

[0026] Sixthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes one or more of the model training methods described in the first aspect or various possible implementations of the first aspect.

[0027] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0028] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0029] Figure 1 This diagram illustrates a structural framework for artificial intelligence.

[0030] Figure 2 A structural diagram of a model training apparatus according to an embodiment of this application is shown.

[0031] Figure 3 A flowchart illustrating a model training method in the pre-training phase according to an embodiment of this application is shown.

[0032] Figure 4 A flowchart illustrating a model training method for an iterative training phase according to an embodiment of this application is shown.

[0033] Figure 5 A flowchart illustrating a model training method according to an embodiment of this application is shown.

[0034] Figure 6 A flowchart illustrating a model training method according to an embodiment of this application is shown.

[0035] Figure 7 A flowchart illustrating a model training method according to an embodiment of this application is shown.

[0036] Figure 8 A structural diagram of a model training apparatus according to an embodiment of this application is shown.

[0037] Figure 9 A structural diagram of a model training apparatus according to an embodiment of this application is shown. Detailed Implementation

[0038] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0039] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0040] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0041] In one existing implementation, natural language understanding models perform well on rich resources, i.e., richly labeled data (such as Chinese and English). For low-resource languages, i.e., languages ​​lacking labeled data, existing natural language understanding models often adopt transfer learning methods, learning knowledge from rich-resource languages ​​through cross-language learning models to transfer to low-resource languages. However, in existing technologies, for a large amount of unlabeled corpus of target languages ​​with low resources, manual annotation is usually required to meet the conditions for model self-supervised learning. Moreover, the accuracy of the annotations generated during model self-supervised learning in existing technologies is low, resulting in excessive noise, which affects the accuracy of model prediction and leads to low model performance.

[0042] To address the aforementioned technical problems, this application provides a model training method. The model training method of this application can train a natural language understanding model to obtain a target model, thereby improving the performance of the target model in the target language and maintaining high prediction accuracy.

[0043] First, the overall workflow of the artificial intelligence system is described. Figure 1 This illustrates a structural diagram of the main framework of artificial intelligence, such as... Figure 1 As shown, the above-mentioned artificial intelligence framework will be elaborated from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the industrial ecosystem of systems.

[0044] (1) Infrastructure

[0045] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the external world is achieved through sensors; computing power is provided by intelligent chips, including but not limited to hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0046] (2) Data

[0047] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0048] (3) Data processing

[0049] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0050] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0051] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0052] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0053] (4) General ability

[0054] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, etc.

[0055] (5) Smart Products and Industry Applications

[0056] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, and safe cities.

[0057] The model trained using the model training method of this application embodiment can be applied to various fields of artificial intelligence. The trained model can be a natural language understanding model, used to perform various tasks related to natural language understanding in the target language, such as part-of-speech tagging, syntactic analysis, text classification, information extraction, question answering system tasks, etc. It should be understood that the examples given are only for the convenience of understanding the application scenarios of this application embodiment and are not exhaustive. The target language can be a low-resource language lacking labeled data, such as a minority language (Malay, Hindi, etc.), and this application does not impose any limitations on this.

[0058] Figure 2 A structural diagram of a model training apparatus according to an embodiment of this application is shown. Figure 2 As shown, the model training apparatus of one embodiment of this application can be applied to a server, which can also be a virtual machine. The server can contain one or more CPUs and one or more GPUs.

[0059] Among them, it can also be achieved through, for example Figure 2The server shown enables the invocation of the model. The model trained on the server can also be deployed on terminal devices, where it can be invoked by modules such as voice assistants. The input text and output results can be determined according to the task of the natural language understanding model. The task of the natural language understanding model may include text classification based on understanding the semantics of the text, understanding the sentiment of the text, etc., as well as tasks such as tagging the parts of speech of the text, text syntactic analysis, and text information extraction. This application does not limit the specific types of tasks that the natural language understanding model can perform. For example, if the natural language understanding model is used for text classification in the target language, the input text may be sentences or documents containing the target language, and the output result may be the classification result of the input text. For example, if the classification task is to determine the sentiment of the text, the output result may be the sentiment category label (such as negative, neutral, positive, etc.) corresponding to the input text.

[0060] In one possible implementation, the model training method of one embodiment of this application can be divided into two stages: pre-training and iterative training.

[0061] Figure 3 A flowchart illustrating a model training method during the pre-training phase according to an embodiment of this application is shown. Figure 3 As shown, in the pre-training stage, if the initial model used is not an open-source multilingual model (such as the mBert model, XLM model, etc.), the initial model can be pre-trained first. Figure 3 (a) shows a flowchart of the pre-training of the initial model, such as Figure 3 As shown in (a), the initial model may include a feature extraction module, a feature transformation module, and a downstream network module. The feature extraction module extracts features from a large amount of unlabeled training data to obtain corresponding feature vectors. These feature vectors may include token embeddings, mask embeddings, and position embeddings. Token embeddings represent the features of the input words, mask embeddings represent the features of masked words in the sentence, and position embeddings represent the positional features of the input words in the sentence. The feature transformation module transforms these feature vectors to input them into the downstream network module. The structure of the downstream network module can be determined based on the model's training task. The downstream network module outputs corresponding results based on this training task. During the pre-training phase of the initial model, the training task can be a masked language modeling (MLM) task, enabling the model to be better applied to multiple languages. Figure 3(a) The initial model shown can be pre-trained to obtain a trained initial model. This application does not restrict the method of pre-training the initial model. The multilingual unlabeled training data input to the model can be multilingual text data obtained from a database containing a large amount of text. This application also does not restrict the method of obtaining multilingual unlabeled training data.

[0062] Based on the trained initial model, its structure can be modified to obtain a natural language understanding model suitable for subsequent iterative training. This modified model can then be pre-trained to obtain a pre-trained model. By modifying the model's structure and further pre-training, the trained model can better adapt to the subsequent iterative training process.

[0063] Figure 3 (b) A flowchart illustrating the pre-training of the modified natural language understanding model is shown, such as... Figure 3As shown in (b), the modified model, based on the pre-trained initial model, also includes a linear transformation module and an adapter module. The linear transformation module can be used to perform a linear transformation on the labeled feature vectors obtained in the feature extraction module, mapping the multilingual feature space corresponding to the labeled feature vectors to the bilingual feature space of the rich resource language and the target language. This allows the labeled feature vectors to better reflect the features of the rich resource language and the target language, enabling the model to achieve better training results in these two languages. The result of the linear transformation can be provided to the feature conversion module. The linear transformation can take the form y = W*x + B, where y represents the labeled feature vector corresponding to the bilingual feature space after transformation, x represents the labeled feature vector corresponding to the multilingual feature space, and W and B represent learnable parameters. The labeled feature vectors in a multilingual feature space can reflect the features of multiple languages, including multiple resource-rich languages ​​(such as English, German, Japanese, etc.) and multiple low-resource languages ​​(such as Malay, Indonesian, etc.). The labeled feature vectors in a bilingual feature space reflect the features of two languages: a resource-rich language (such as one of English, German, Japanese, etc.) and a target language (the target language can be a low-resource language, such as one of Malay, Indonesian, etc.). A resource-rich language can be a language with sufficient labeled training data (such as English), while a target language can be a language lacking labeled training data (such as a less commonly spoken language like Malay). The resource-rich language differs from the target language. The adapter module can include multiple adapters. Each adapter receives the output of the feature transformation module and connects to its corresponding downstream network module to output the corresponding results. The adapters and downstream network modules within the adapter module can share all parameters of the feature extraction module, linear transformation module, and feature transformation module. Each adapter in the adapter module can have the same structure but different dimensional transformation methods. Each downstream network module corresponding to an adapter can have the same structure. Through different dimensional transformation methods of the adapters, the downstream network modules can learn different network parameters during training, effectively creating multiple classifiers (corresponding to multiple downstream network modules) in the model. Therefore, the modified model can be an ensemble learning network structure. For example, if the adapter module contains N adapters, these N adapters can each correspond to N downstream networks. Through the multiple adapters in this adapter module, a single input can correspond to N sets of not entirely identical output results, each corresponding to a downstream network and an adapter. This is equivalent to having N-1 additional training results. By optimizing the loss of these multiple output results, existing training resources can be utilized more effectively.

[0064] The input training data may include unlabeled training data in multiple languages, or it may include only unlabeled training data in the resource-rich language and the target language. Since the MLM task does not require labeled training data, pre-training can be completed using unlabeled training data based on the MLM task, thereby saving training resources and compressing the model size. This application does not impose any restrictions on this.

[0065] Figure 3 (c) shows a structural diagram of the adapter module. The adapter module may include at least one adapter, such as... Figure 3 As shown in (c), adapter 1, adapter 2, and adapter N can represent adapters in the adapter module. Each adapter can be a bottleneck structure. An adapter may include a hidden layer, which may include two linear transformation layers and one activation layer. The linear transformation layers can be used to compress and restore the dimension of the feature vector output by the feature transformation module. The activation layer can be used to perform nonlinear mapping on the feature vector, making the adapter a nonlinear transformation module. Through the adapter, nonlinear transformation of the feature vector can be achieved. Figure 3 (c) The hexagonal structure of the upper and lower layers in the adapter can represent the linear transformation layer, and the rectangular structure connected in the middle of the linear transformation layer can represent the activation layer. After passing through the first linear transformation layer, the dimension of the feature vector is compressed, and after passing through the second linear transformation layer, the dimension of the feature vector is restored, so that the dimension of the adapter output remains unchanged after passing through the hidden layer. For example, the input dimension and output dimension of the hidden layer in each adapter can be represented as H. The dimension compressed by the first linear transformation layer in each adapter is different. The dimensions inside the hidden layer of each adapter can satisfy H / k, H / 2k...H / nk after compression. For example, if the dimension inside the hidden layer of adapter 1 is H / k after compression, the dimension inside the hidden layer of adapter 2 can be H / 2k after compression, and the dimension inside the hidden layer of adapter N can be H / nk after compression. Here, H represents the original input dimension and output dimension of the adapter, k can represent the pre-set dimension change factor, and n can represent the number of adapters. By varying the dimensions within the hidden layers of each adapter, the model can differentiate each adapter and its corresponding downstream network module from other adapters and downstream network modules during training, allowing them to learn different parameters. This results in downstream network modules producing slightly different outputs during training. Furthermore, by using adapters to perform non-linear transformations on the feature vectors, the parameters learned by each downstream network module are not limited to a linear range, thus improving training effectiveness.

[0066] During the pre-training process of the modified model, the input unlabeled training data may include... Figure 3In (a), the unlabeled training data in multiple languages ​​input, including the training data of the resource-rich language and the target language, can be used as the training task for the downstream network modules. Since the iterative training focuses on the resource-rich language and the target language, the model can be pre-trained using the MLM task, which can enable the model to perform better in the resource-rich language and the target language. Multiple downstream network modules can output multiple corresponding prediction results, and the loss of the model can be optimized based on the corresponding prediction results, and finally a pre-trained natural language understanding model can be obtained.

[0067] It should be noted that this application does not limit the specific types of resource-rich languages ​​or target languages. Figure 3 (b) also does not impose any restrictions on the method of pre-training the natural language understanding model shown.

[0068] Figure 4 A flowchart illustrating a model training method in an iterative training phase according to an embodiment of this application is provided. Wherein, it is possible to... Figure 3 (b) Based on the pre-trained natural language understanding model with modified structure, the model is iteratively trained. For simplicity, Figure 3 (b) multiple downstream network task modules in Figure 4 (a) and Figure 4 (b) is represented by a unified downstream network module. Figure 4 (a) and Figure 4 (b) The training task adapted to the downstream network module can be determined based on the model's natural language understanding task, such as text classification, information extraction, etc., and this application does not impose any restrictions on this. Figure 4 As shown, iterative training can be performed by repeatedly executing steps (a) and (b). The iterative training process includes:

[0069] Step (a) involves randomly replacing a portion of the training data using the code-switching method to fine-tune the training model;

[0070] The input training data may include rich-resource labeled training data. This application does not limit the language type of the rich-resource training data, such as... Figure 4 As shown in (a), English training data with rich resources is used as an example for illustration.

[0071] At the start of the first iteration, if the input training data only includes labeled training data in English and not labeled training data in the target language, the prediction results obtained by the model will contain a lot of noise. The noise can be reduced by randomly replacing the labeled English training data using the code-switching method.

[0072] Therefore, in step (a), it can be based on, as follows Figure 4 (a) shows a bilingual dictionary of English and the target language. Using a code-switching method, a predetermined probability of English training data is randomly replaced, while retaining the original English annotations. The predetermined probability can be set according to the needs of the training task; for example, a predetermined probability of 10% can randomly replace 10% of the English training data. The retained annotations can be the English annotations corresponding to the training data. After the replacement, the input training data can include the unreplaced English training data and the target language training data with the original English annotations. This training data can be used to fine-tune the relevant parameters of the model. Compared to random replacement without using the code-switching method, the model trained using this method has higher accuracy.

[0073] Starting from the second iteration, since the target language training data with pseudo-annotations can be determined according to step (b) in the first iteration, the training data may also include, for example, Figure 4 The target language training data with pseudo-annotations shown in the dashed box in (a) includes new target language training data with original English annotations obtained by replacing part of the original labeled English training data using the code-switching method. This training data can be used to train the model and fine-tune its parameters. The method for determining the target language training data with pseudo-annotations will be introduced in step (b).

[0074] Step (b) involves identifying the training data with pseudo-labels and adding the pseudo-labeled training data to the training data.

[0075] Since the above-mentioned random replacement method alone cannot meet the conditions for the model to start self-supervised learning, by determining training data with pseudo-labels, the model can perform self-supervised learning and better complete relevant natural language understanding tasks in the target language.

[0076] This process involves inputting unlabeled training data of the target language into the model to obtain corresponding prediction results and confidence levels. For n downstream networks in the model, n prediction results and corresponding confidence levels can be obtained. For a given data point, if the corresponding n prediction results are consistent and the confidence levels of the n prediction results are all greater than a predetermined threshold, this prediction result is determined as a pseudo-label corresponding to the training data. After the first iteration, the training data with pseudo-labels can be added to the training data in step (a). The pseudo-labels can participate in model training as annotations for the training data. By determining pseudo-labels through voting on multiple prediction results, the accuracy of the obtained pseudo-labels can be improved, thereby reducing noise caused by labeling errors, improving the accuracy of the model, and enabling the model to perform self-supervised learning without manual labeling, reducing manpower and material resources.

[0077] In each iteration, the unlabeled data of the target language input in step (b) can include all the original unlabeled data of the target language. The training data with pseudo-labels can be re-determined in each iteration, and the training data in step (a) can be updated. The pseudo-labeled data determined in step (a) in the previous iteration is replaced by the pseudo-labeled data determined in the training data in step (a) with the pseudo-labeled data re-determined in this iteration.

[0078] Then, the iterative training process of steps (a)-(b)-(a) can be repeated. After each iteration, it is determined whether the model's loss function value meets the requirements. If it does not, the relevant parameters in the model are updated, the pseudo-annotated target language training data in the first training data is replaced, and the English training data in the annotated English training data is randomly replaced again. The iteration continues until the model's loss function converges to a small value, resulting in the iteratively trained target natural language understanding model. Through the iterative training process, the quality of the training data can be continuously improved, noise reduced, and thus the accuracy of the finally trained target model improved. It should be noted that this application does not restrict the method of loss optimization during iterative training, and the value at which the loss function converges can be preset, which is also not restricted in this application.

[0079] Figure 5 A flowchart illustrating a model training method according to an embodiment of this application is shown. Figure 5 As shown, the method includes:

[0080] Step S501: Obtain the first training data based on the training data of the labeled auxiliary language;

[0081] Step S502: Input the first training data into the first model to obtain multiple first prediction results. The first model includes multiple adapters, each adapter is connected to a network module, each adapter performs different dimensional changes, and each adapter and the corresponding network module output a first prediction result of the first training data.

[0082] Step S503: Determine whether the first model meets the training convergence condition based on the multiple first prediction results.

[0083] Step S504: If the training convergence condition is not met, input the training data of the target language into the first model to obtain pseudo-labels, add the training data of the target language including pseudo-labels to the first training data, and repeat the steps of obtaining the first training data based on the training data of the labeled auxiliary language until the training convergence condition is met to obtain the trained first model. The first model is used to perform natural language understanding of the target language, and the auxiliary language is different from the target language.

[0084] According to the embodiments of this application, by obtaining first training data, inputting the first training data into a first model, multiple first prediction results are obtained. Based on the multiple first prediction results, it is determined whether the first model meets the training convergence condition. If the training convergence condition is not met, the training data of the target language is input into the first model to obtain pseudo-labels. The training data of the target language including pseudo-labels is added to the first training data. The steps of obtaining the first training data based on the training data of the labeled auxiliary language and subsequent steps are repeated until the training convergence condition is met, and a trained first model is obtained. This can realize the process of self-supervised learning of the model. Through the process of self-iterative training, pseudo-labels can be obtained and the accuracy of pseudo-labels can be continuously optimized without manual labeling, reducing noise. By determining the pseudo-labels, the model can also meet the conditions of self-supervised learning, further improving the accuracy of the model. The first model includes multiple adapters, each adapter is connected to a network module, and different dimensional changes are performed inside each adapter. Each adapter and the corresponding network module output a first prediction result of the first training data. The multiple first prediction results output can be used to achieve voting, making greater use of existing training resources and improving the performance of the model.

[0085] Among them, the content of the annotation or pseudo-annotation is adapted to the natural language understanding task of the model. For example, when the type of training data is text, the annotation or pseudo-annotation can represent the sentiment type of the text, the word nature of the text, etc. For example, if the model is used for text sentiment type classification, the English text "I’m angry" and the annotation "angry" representing the sentiment can be used as a piece of training data and its annotation in the "training data with annotation for auxiliary language training". The network module can refer to the downstream network module in the previous text. The number of adapters in the first model can be determined according to the needs of the task. The obtained multiple first prediction results can be the same or different. The training convergence condition can be that the loss function of the model converges to a predetermined value. The auxiliary language can be a language with sufficient training data with annotation (such as the rich resource language in the previous text), such as English, Chinese, etc. The target language can be a language lacking training data with annotation (such as the low resource language in the previous text), such as small language like Malay. The present application does not limit the specific types of the auxiliary language and the target language. The present application also does not limit the specific task type of natural language understanding for which the first model is used.

[0086] Examples of steps S501 - S504 can refer to Figure 4 the process of iteratively training the model in

[0087] In a possible implementation, different dimensionality changes are performed inside each adapter, including: each adapter compresses and restores the dimension of the input vector, and the compression multiples of each adapter are different.

[0088] According to the embodiments of the present application, it is possible to achieve that the model outputs prediction results that are not completely the same, realize the ensemble learning of the model, make greater use of the existing training resources, and improve the accuracy of the model.

[0089] Among them, the compression multiple of the adapter can be determined according to the needs of the training task and the number of adapters. For example, the compression multiples between each adapter can satisfy a linear change. The present application does not limit the change amount or change method of the dimensionality change between adapters.

[0090] Examples of the above-mentioned adapter compressing and restoring the dimension and different compression multiples of different adapters can refer to Figure 3 (c) in the relevant description.

[0091] In a possible implementation, obtaining the first training data according to the training data of the auxiliary language with annotation includes: according to the correspondence between the auxiliary language and the target language, replacing the training data of the auxiliary language with a predetermined probability in the training data of the auxiliary language with annotation with the training data of the target language, and retaining the original annotation of the auxiliary language to obtain the first training data.

[0092] According to the embodiments of this application, by replacing the training data of the auxiliary language with the training data of the target language according to a predetermined probability, while retaining the original annotations of the auxiliary language, the noise contained in the prediction results can be reduced when pseudo-annotations are obtained directly without replacement, thereby improving the robustness and accuracy of the model.

[0093] The correspondence between the auxiliary language and the target language can be determined by a bilingual dictionary of the auxiliary language and the target language, or by other means. This application does not impose any restrictions on this. The predetermined probability can be set according to the needs of the task. In each iteration, the training data of the auxiliary language can be replaced, and the predetermined probability can be reset. This application does not impose any restrictions on this.

[0094] An example of the above process can be found in [reference]. Figure 4 Step (a) in the middle.

[0095] Figure 6 A flowchart illustrating a model training method according to an embodiment of this application is shown. Figure 6 As shown, the training data of the target language is input into the first model to obtain pseudo-labels, including:

[0096] Step S601: Input the training data of the target language into the first model to obtain multiple second prediction results corresponding to each adapter;

[0097] Step S602: If the multiple second prediction results are the same and the corresponding confidence levels are all greater than the first threshold, determine that the second prediction result is a pseudo-label of the training data of the input target language.

[0098] According to the embodiments of this application, pseudo-labels can be selected by voting from multiple second prediction results corresponding to the adapter, so that the determined pseudo-labels have higher accuracy and less noise, and can be better applied to the training of the model, resulting in a more accurate trained model.

[0099] The training data for the target language can be unlabeled, such as containing only text training data in the target language. Multiple second prediction results can be identical or different. The first threshold can be pre-set according to the needs of the natural language recognition task. The training data for the target language can have pseudo-labels or not. For example, if multiple second prediction results for a certain training data set of the target language are not completely identical, or if multiple second prediction results are identical but the confidence level of some second prediction results is lower than the first threshold, then the training data set of that target language can be considered to have no pseudo-labels and does not need to be added to the first training data. In each iteration, the pseudo-labels for the training data of the target language can be redefined, and the first threshold can be reset; this application does not impose any restrictions on this.

[0100] An example of the above process can be found in [reference]. Figure 4 Step (b) in the middle.

[0101] Figure 7 A flowchart illustrating a model training method according to an embodiment of this application is shown. Figure 7 As shown, the method further includes:

[0102] Step S701: Train the second model to obtain a trained second model, wherein the second model includes multiple adapters, each adapter is connected to a network module, and each adapter undergoes different dimensional changes. The training task of the second model is the Masked Language Model (MLM) task.

[0103] Step S702: Adjust the adapter and network module in the trained second model to an adapter and network module for natural language understanding to obtain the first model.

[0104] According to the embodiments of this application, by training the second model according to the MLM task, the model can achieve better results in both the auxiliary language and the target language. Furthermore, by adjusting the adapter and network modules in the trained second model to adapters and network modules for natural language understanding, the first model is obtained. Since the second model has been trained on the MLM task, the first model can better adapt to the training of the natural language understanding task of the target language, resulting in better performance of the trained model and saving training resources.

[0105] The number of adapters included in the second model can be determined according to the needs of the task. The adapters and network modules in the trained second model can be adjusted to be adapters and network modules for natural language understanding. This can be done by adjusting the structure of the adapters and network modules to adapt to the natural language understanding task of the first model.

[0106] Examples of steps S701-S702 can be found in [reference]. Figure 3 (b) describes the process of pre-training the model.

[0107] In one possible implementation, the second model further includes a feature extraction module, a linear transformation module, and a feature conversion module. The feature extraction module is used to extract feature vectors from the training data. The linear transformation module is used to convert the first labeled feature vector in the extracted feature vector to a bilingual feature space of the auxiliary language and the target language to obtain a second labeled feature vector, which represents the features of words in the training data. The feature conversion module is used to convert the second labeled feature vector and output it to each adapter.

[0108] According to the embodiments of this application, since the first model is trained on the auxiliary language and the target language, by transforming the first labeled feature vector in the extracted feature vector to the bilingual feature space of the auxiliary language and the target language, a second labeled feature vector is obtained. The labeled feature vector represents the features of words in the training data, which can make the second model perform better on the auxiliary language and the target language. Thus, after the first model is obtained based on the second model, the first model performs better on the auxiliary language and the target language. The model is more targeted during the training process, so that the finally trained model can perform better on the target language and the model's inference results are more accurate.

[0109] The bilingual feature space can be the feature spaces of the auxiliary language and the target language corresponding to the feature vectors. The feature vectors extracted by the feature extraction module from the training data can be from the multilingual feature space. The feature transformation module can also transform other feature vectors besides the second labeled feature vector and output them to each adapter; this application does not impose any restrictions on this.

[0110] The schematic structure of the feature extraction module, linear transformation module, and feature conversion module can be found in the following reference. Figure 3 The corresponding module in the model in (b) is shown.

[0111] Figure 8 A structural diagram of a model training apparatus according to an embodiment of this application is shown. Figure 8 As shown, the device includes:

[0112] The first determining module 801 is used to obtain the first training data based on the training data of the labeled auxiliary language;

[0113] The second determining module 802 is used to input the first training data into the first model to obtain multiple first prediction results. The first model includes multiple adapters, each adapter is connected to a network module, each adapter performs different dimensional changes, and each adapter and the corresponding network module output a first prediction result of the first training data.

[0114] The judgment module 803 is used to determine whether the first model meets the training convergence condition based on the plurality of first prediction results.

[0115] The third determining module 804 is used to input the training data of the target language into the first model to obtain pseudo-labels when the training convergence condition is not met, add the training data of the target language including the pseudo-labels to the first training data, and repeat the steps of obtaining the first training data based on the training data of the labeled auxiliary language until the training convergence condition is met and the trained first model is obtained. The first model is used to perform natural language understanding of the target language, and the auxiliary language is different from the target language.

[0116] According to the embodiments of this application, by obtaining first training data, inputting the first training data into a first model, multiple first prediction results are obtained. Based on the multiple first prediction results, it is determined whether the first model meets the training convergence condition. If the training convergence condition is not met, the training data of the target language is input into the first model to obtain pseudo-labels. The training data of the target language including pseudo-labels is added to the first training data. The steps of obtaining the first training data based on the training data of the labeled auxiliary language and subsequent steps are repeated until the training convergence condition is met, and a trained first model is obtained. This can realize the process of self-supervised learning of the model. Through the process of self-iterative training, pseudo-labels can be obtained and the accuracy of pseudo-labels can be continuously optimized without manual labeling, reducing noise. By determining the pseudo-labels, the model can also meet the conditions of self-supervised learning, further improving the accuracy of the model. The first model includes multiple adapters, each adapter is connected to a network module, and different dimensional changes are performed inside each adapter. Each adapter and the corresponding network module output a first prediction result of the first training data. The multiple first prediction results output can be used to achieve voting, making greater use of existing training resources and improving the performance of the model.

[0117] In one possible implementation, each adapter performs different dimensionality changes, including: each adapter compresses and restores the dimensions of the input vector, with each adapter having a different compression factor.

[0118] According to the embodiments of this application, it is possible to achieve model outputs that are not completely identical, realize the ensemble learning of the model, make greater use of existing training resources, and improve the accuracy of the model.

[0119] In one possible implementation, obtaining the first training data based on the labeled auxiliary language training data includes: replacing the auxiliary language training data with a predetermined probability in the labeled auxiliary language training data with the target language training data according to the correspondence between the auxiliary language and the target language, while retaining the original auxiliary language annotations, to obtain the first training data.

[0120] According to the embodiments of this application, by replacing the training data of the auxiliary language with the training data of the target language according to a predetermined probability, while retaining the original annotations of the auxiliary language, the noise contained in the prediction results can be reduced when pseudo-annotations are obtained directly without replacement, thereby improving the robustness and accuracy of the model.

[0121] In one possible implementation, inputting training data of the target language into a first model to obtain pseudo-labels includes: inputting training data of the target language into the first model to obtain multiple second prediction results corresponding to each adapter; and determining the second prediction result as a pseudo-label of the input training data of the target language when the multiple second prediction results are the same and the corresponding confidence scores are all greater than a first threshold.

[0122] According to the embodiments of this application, pseudo-labels can be selected by voting from multiple second prediction results corresponding to the adapter, so that the determined pseudo-labels have higher accuracy and less noise, and can be better applied to the training of the model, resulting in a more accurate trained model.

[0123] In one possible implementation, the device further includes: a training module for training a second model to obtain a trained second model, wherein the second model includes multiple adapters, each adapter is connected to a network module, and each adapter undergoes different dimensional changes, and the training task of the second model is a masked language modeling (MLM) task; and an adjustment module for adjusting the adapters and network modules in the trained second model to adapters and network modules for natural language understanding to obtain the first model.

[0124] According to the embodiments of this application, by training the second model according to the MLM task, the model can achieve better results in both the auxiliary language and the target language. Furthermore, by adjusting the adapter and network modules in the trained second model to adapters and network modules for natural language understanding, the first model is obtained. Since the second model has been trained on the MLM task, the first model can better adapt to the training of the natural language understanding task of the target language, resulting in better performance of the trained model and saving training resources.

[0125] In one possible implementation, the second model further includes a feature extraction module, a linear transformation module, and a feature conversion module. The feature extraction module is used to extract feature vectors from the training data. The linear transformation module is used to convert the first labeled feature vector in the extracted feature vector to a bilingual feature space of the auxiliary language and the target language to obtain a second labeled feature vector, which represents the features of words in the training data. The feature conversion module is used to convert the second labeled feature vector and output it to each adapter.

[0126] According to the embodiments of this application, since the first model is trained on the auxiliary language and the target language, by transforming the first labeled feature vector in the extracted feature vector to the bilingual feature space of the auxiliary language and the target language, a second labeled feature vector is obtained. The labeled feature vector represents the features of words in the training data, which can make the second model perform better on the auxiliary language and the target language. Thus, after the first model is obtained based on the second model, the first model performs better on the auxiliary language and the target language. The model is more targeted during the training process, so that the finally trained model can perform better on the target language and the model's inference results are more accurate.

[0127] Figure 9 A structural diagram of a model training apparatus according to an embodiment of this application is shown. Figure 9 As shown, the device 40 includes at least one processor 1801, at least one memory 1802, and at least one communication interface 1803. Furthermore, the device may also include general-purpose components such as antennas, which will not be detailed here.

[0128] Processor 1801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs in the above scheme. Processor 1801 may include one or more processing units, such as: processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0129] The communication interface 1803 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), core network, Wireless Local Area Networks (WLAN), etc.

[0130] The memory 1802 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0131] The memory 1802 stores the application code for executing the above scheme, and its execution is controlled by the processor 1801. The processor 1801 executes the application code stored in the memory 1802.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0133] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0135] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] An embodiment of this application provides a model training apparatus, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0138] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0139] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0140] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0141] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0142] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0143] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0144] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0145] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0147] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0148] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0149] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that, The method comprises: obtaining first training data according to training data of a labeled auxiliary language; inputting the first training data into a first model to obtain a plurality of first prediction results, wherein the first model comprises a plurality of adapters, each adapter is connected with a network module, different dimensional changes are performed in each adapter, and each adapter and the corresponding network module output a first prediction result of the first training data; determining whether the first model meets a training convergence condition according to the plurality of first prediction results, in the case that the training convergence condition is not met, inputting training data of a target language into the first model to obtain pseudo-labeling, adding training data of the target language with pseudo-labeling into the first training data, repeatedly performing the step of obtaining first training data according to training data of a labeled auxiliary language and the following steps until the training convergence condition is met, obtaining a trained first model, and the first model is used for natural language understanding of the target language, and the auxiliary language is different from the target language; the step of obtaining first training data according to training data of a labeled auxiliary language comprises: replacing training data of the auxiliary language with a predetermined probability in the training data of the labeled auxiliary language with training data of the target language according to the correspondence between the auxiliary language and the target language, retaining the label of the original auxiliary language, and obtaining the first training data.

2. The method according to claim 1, characterized in that different dimensional changes are performed in each adapter, comprising: each adapter compresses and restores the dimension of an input vector, and the compression multiples of the adapters are different.

3. The method according to claim 1 or 2, characterized in that, inputting training data of a target language into a first model to obtain pseudo-labeling comprises: inputting the training data of the target language into the first model to obtain a plurality of second prediction results corresponding to each adapter; in the case that the plurality of second prediction results are the same and the corresponding confidence values are all greater than a first threshold value, determining that the second prediction result is the pseudo-labeling of the input training data of the target language.

4. The method of claim 1, wherein, The method further comprises: training a second model to obtain a trained second model, wherein the second model comprises a plurality of adapters, each adapter is connected with a network module, different dimensional changes are performed in each adapter, and the training task of the second model is a masked language model (MLM) task; adjusting the adapters and network modules in the trained second model to adapters and network modules used for natural language understanding to obtain the first model.

5. The method of claim 4, wherein, The second model further comprises a feature extraction module, a linear transformation module and a feature conversion module, the feature extraction module is used for extracting a feature vector of training data; the linear transformation module is used for converting a first marked feature vector in the extracted feature vector to a bilingual feature space of an auxiliary language and a target language to obtain a second marked feature vector, and the marked feature vector represents the features of a word in the training data; the feature conversion module is used for converting the second marked feature vector and outputting to each adapter.

6. A model training apparatus characterized by comprising: The device comprises: a first determination module configured to obtain first training data according to training data of a labeled auxiliary language; The second determining module is configured to input the first training data into a first model to obtain a plurality of first prediction results, wherein the first model comprises a plurality of adapters, each adapter is connected to a network module, different dimensional changes are performed in each adapter, and each adapter and the corresponding network module output a first prediction result of the first training data; The judging module is configured to determine whether the first model meets a training convergence condition according to the plurality of first prediction results, The third determining module is configured to, in the case where the training convergence condition is not met, input training data of a target language into the first model to obtain pseudo-labeling, add the training data of the target language with the pseudo-labeling into the first training data, repeatedly execute the steps of obtaining the first training data from the training data of the labeled auxiliary language and the subsequent steps until the training convergence condition is met, and obtain a trained first model, wherein the first model is used for natural language understanding of the target language, and the auxiliary language is different from the target language. The first determining module is configured to: According to the correspondence between the auxiliary language and the target language, replace training data of the auxiliary language with a predetermined probability in the training data of the labeled auxiliary language with training data of the target language, and retain the label of the original auxiliary language to obtain the first training data.

7. A model training apparatus characterized by comprising: It comprises: A processor; A memory for storing processor-executable instructions; When the processor is configured to execute the instructions, the method of any one of claims 1-5 is implemented.

8. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1-5.

9. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in an electronic device, a processor in the electronic device executes the method of any one of claims 1-5.

Images