Text processing method and device, storage medium and computer program product

Abstract

The embodiment of the invention discloses a text processing method and device, a storage medium and a computer program product, and the method comprises the steps: integrating different language models with the same structure and different language text processing capabilities through a router, and obtaining an integrated model; each language model comprises a word list, a word embedding layer, an attention layer, a forward layer and a generator which are cascaded, and the router is connected between the forward layer and the attention layer of different language models; and respectively inputting a to-be-processed text into each language model in the integrated model, and in the text processing process of each language model, distributing a forward layer for a hidden layer state generated by an attention layer of each language model from each connected forward layer through the router so as to carry out forward calculation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a text processing method, device, storage medium, and computer program product. Background Art

[0002] In recent years, language big models have become a hot topic in the natural language processing research community and the artificial intelligence industry. Their outstanding language understanding and generation capabilities have made many researchers realize that the development of today's artificial intelligence is entering the era of general artificial intelligence. Multilingual capabilities are one of the directions that cannot be ignored in the development of language big models. Multilingual big models not only need to have strong performance in multiple languages, but also have strong cross-language understanding and generation capabilities. However, the training data available for different languages ​​varies greatly in quantity, quality, and fields, which makes the establishment of a multilingual big model system not a one-time thing.

[0003] At present, on the one hand, multilingual expansion can be performed based on a large model of a single language or a small number of languages. The language expansion method generally adopted requires expanding the vocabulary and then continuing to pre-train the model. In this process, it is necessary not only to collect data of the language to be expanded, but also to sample the original language data and match it with the data to be expanded. However, even so, it may not be possible to support the new language well while avoiding serious catastrophic forgetting. On the other hand, a large multilingual model can be directly built. In this case, large-scale multilingual training data needs to be prepared. However, the dilemma is that the data collected for different languages ​​is difficult to be unified in terms of quantity, quality and field. The screening, cleaning and preprocessing strategies of different language data will also be different, and the requirements for manpower will also increase with the scarcity of the language. This means that data in different languages ​​need to go through different iterations to achieve a good quality, which greatly increases the iteration cost of the model. Therefore, directly building a large multilingual model is an extremely time-consuming and labor-intensive task.

[0004] Based on the above content, it can be seen that the current multilingual large model data is difficult to build, and the performance of processing texts in different languages ​​varies greatly. Summary of the invention

[0005] The embodiments of the present application provide a text processing method, device, storage medium and computer program product, which integrate different language models with different language text processing capabilities in a model integration manner, and use the integrated model to perform text processing, which can not only save a large amount of computing resources and make full use of existing language models, but also show good performance in processing texts in different languages.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application embodiment provides a text processing method, including:

[0008] A router is used to integrate different language models with the same structure and different language text processing capabilities to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models;

[0009] The text to be processed is input into each language model in the integrated model respectively. During the text processing process of each language model, the router allocates a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers to perform forward calculation.

[0010] In the above method, allocating a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers by the router includes:

[0011] By means of the router, the probability values ​​of the first hidden layer state being assigned to each connected forward layer are calculated;

[0012] And, sorting the forward layers in descending order of probability values, and assigning the first K forward layers to the first hidden layer state for respectively performing forward calculations;

[0013] Wherein, K is a natural number greater than or equal to 1, and the first hidden layer state is a hidden layer state generated by the attention layer of any language model.

[0014] In the above method, the integrated model also includes a collector, which is connected between the forward layer and the generator of different language models. The method also includes:

[0015] During the text processing of each language model, the forward calculation results of the hidden layer state generated by the attention layer of each language model are collected and sorted through the collector, and passed to the next structure of the same language model.

[0016] In the above method, in the integrated model, different adapters are connected between the router and the forward layers of different language models, and different reducers are respectively connected between the collector and the generators of different language models. The method further includes:

[0017] During the text processing of each language model, each adapter is used to adapt the passed hidden layer state to the connected forward layer, and each reducer is used to adapt and restore the passed forward calculation results.

[0018] In the above method, the integrated model also includes a scoring model, and the scoring model is connected to the generator of each language model in the integrated model. The method also includes:

[0019] The scoring model is used to score the text processing results of the text to be processed generated by the generator of each language model, and the text processing result with the highest score is output.

[0020] In the above method, it also includes:

[0021] By means of the scoring model, for each language model, a corresponding optimization weight is set for scoring the text processing result of the to-be-processed text generated by the respective generator, so as to determine the loss function of the integrated model;

[0022] The model loss function of the integrated model is the weighted sum of the loss functions of different language models and the corresponding optimization weights.

[0023] In the above method, it also includes:

[0024] Using a control variable method, respectively configure the parameters of the router, each adapter and each reducer in the integrated model;

[0025] After configuring the parameters, the integrated model is trained, and during the training process, the parameters of each language model in the integrated model are kept unchanged, and the parameters of the router, each adapter and each reducer are optimized.

[0026] In the above method, the control variable method is used to respectively determine the parameters of the router, each adapter and each reducer in the integrated model, including:

[0027] After removing each adapter and each reducer in the integrated model, the parameters of the router are used as variables, while the parameters of each language model are kept unchanged, the sample text is processed, and the parameters when the processing result of the sample text is optimal are configured for the router;

[0028] After completing the router parameter configuration, each adapter and each reducer are re-added to the integrated model, and the parameters of each adapter and each reducer are used as variables. At the same time, the parameters of each language model and the router are kept unchanged, and text processing is performed on the sample text. The parameters that produce the optimal processing result of the sample text are configured for each adapter and each reducer.

[0029] The present application provides a text processing device, including:

[0030] A model integration module, which is used to integrate different language models with the same structure and different language text processing capabilities using a router to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models;

[0031] The text processing module is used to input the text to be processed into each language model in the integrated model respectively; during the text processing process of each language model, the router is used to allocate a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers to perform forward calculation.

[0032] The embodiment of the present application provides a text processing device, including: a processor, a memory and a communication bus;

[0033] The communication bus is used to realize the communication connection between the processor and the memory;

[0034] The processor is used to execute one or more computer programs stored in the memory to implement the above-mentioned text processing method.

[0035] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned text processing method is implemented.

[0036] An embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned text processing method when executed.

[0037] The embodiment of the present application provides a text processing method, device, storage medium and computer program product, the method includes: using a router to integrate different language models with the same structure and different language text processing capabilities to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models; the text to be processed is input into each language model in the integrated model respectively, and in the process of text processing in each language model, the forward layer is allocated to the hidden layer state generated by the attention layer of each language model from the connected forward layers through the router to perform forward calculation. The technical solution provided by the embodiment of the present application integrates different language models with different language text processing capabilities in a model integration manner, and uses the integrated model for text processing, which can not only save a lot of computing resources and make full use of existing language models, but also show good performance in the processing of texts in different languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1A schematic diagram of the structure of an exemplary hybrid expert system provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of a text processing method provided in an embodiment of the present application;

[0040] Figure 3 An exemplary model integration diagram provided in the embodiment of the present application Figure 1 ;

[0041] Figure 4 A schematic diagram of text processing of an exemplary integrated model provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of an exemplary deployment of an adapter and a reducer provided in an embodiment of the present application;

[0043] Figure 6 An exemplary model integration diagram provided in the embodiment of the present application Figure 2 ;

[0044] Figure 7 An exemplary model integration diagram provided in the embodiment of the present application Figure 3 ;

[0045] Figure 8 A structural diagram of a text processing device provided in an embodiment of the present application Figure 1 ;

[0046] Fig. 9 A structural diagram of a text processing device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The following will specifically describe the technical solution of the present application and how the technical solution of the present application solves the above technical problems through embodiments and in conjunction with the accompanying drawings. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0049] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0050] The core idea of ​​the technical solution of this application is to avoid the collection of large-scale multilingual data and pre-training of multilingual models when building a multilingual model, but to use the existing single language or a small number of language large models that specialize in different languages ​​to integrate into an integrated model. The integration method of this application is to use a hybrid expert system to achieve model integration. The structure of the hybrid expert system is as follows Figure 1 As shown, specifically, in this application, a router is used to integrate the forward layers of different language models, and the forward layers are shared between different language models, thereby realizing forward calculation of the same input across different models. Since there are large differences in parameter distribution between different language models, adapters and reducers can also be added internally to enable parameter sharing between different models, which is described in detail below.

[0051] The embodiment of the present application provides a text processing method, which is implemented by a text processing device. The text processing device can specifically be an electronic device such as a mobile phone, a tablet computer, etc., and the embodiment of the present application is not limited thereto. Figure 2 A flowchart of a text processing method provided in an embodiment of the present application is shown below. Figure 2 As shown, in the embodiment of the present application, the text processing method mainly includes the following steps:

[0052] S101. Using a router, different language models with the same structure and different language text processing capabilities are integrated to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models.

[0053] In an embodiment of the present application, the text processing device may first utilize a router to integrate different language models having the same structure and the ability to process texts in different languages ​​to obtain an integrated model.

[0054] It is understandable that in the embodiments of the present application, different language models have different language text processing capabilities. For example, one language model has Chinese text processing capabilities, and another language model has English text processing capabilities. The integrated model obtained by integrating the two language models becomes a model with Chinese and English bilingual text processing capabilities. The specific language text processing capabilities of each language model are not limited in the embodiments of the present application.

[0055] It should be noted that in the embodiments of the present application, the type of each language model can be selected according to actual needs. For example, one language model can use the Large Language Model AI (LLaMA) and another model can use Tongyi Qianwen (Qwen). Each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, wherein the forward layer and the attention layer of each language model can be stacked L, see Figure 3 As shown, L is a positive integer, and the specific structure of each language model is not limited in this embodiment of the application.

[0056] It should be noted that in the embodiment of the present application, the forward layer and the attention layer of different language models can be connected by using a router, so that the integration of different language models can be achieved. Figure 3 , the stacked L forward layers and attention layers of different language models can be connected using routers respectively. Of course, according to actual needs, only some of them can be selected for connection, and the parts not connected through the router are not included in the integrated model.

[0057] S102. Input the text to be processed into each language model in the integrated model respectively. During the text processing process of each language model, a forward layer is allocated to the hidden layer state generated by the attention layer of each language model from the connected forward layers through a router to perform forward calculation.

[0058] In an embodiment of the present application, after the text processing device integrates the integrated model, it can input the text to be processed into each language model in the integrated model for processing. Since a router is introduced in the integration process, based on this, the router can assign a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers to perform forward calculation.

[0059] It should be noted that in the embodiments of the present application, in the integrated model of multiple language models, while retaining the forward layer of each language model, the respective word segmenter, word embedding layer, generator, attention layer, etc. in each language model are also retained. This is because the parameters between the internal modules of the language model are highly coupled, and the forward layer of each language model can only be cascaded with the modules of the same language model. When the model input is assigned to the forward layer of a non-current model, corresponding adaptation is required, which will be described in detail later.

[0060] It should be noted that, in this process, the router can generate a hidden layer state for the attention layer of each language model and assign a forward layer that is better at processing the language to which it belongs. The assigned forward layer may belong to the same language model as the attention layer that generates the hidden layer state, or it may belong to a different language model. In addition, the assigned forward layer may be one or more, which is not limited in the embodiments of the present application.

[0061] For example, in the embodiments of the present application, Figure 4 As shown in the figure, the integrated model that combines the two language models Qwen with Chinese text processing capabilities and LLaMA with English text processing capabilities is used for translation tasks as an example to demonstrate the integrated model and its internal connection method and data transmission method. Among them, the input data is a simple combination of Chinese and English, so it is necessary to have both Chinese and English text processing capabilities. When the model output is required to translate back to Chinese, the model also needs to have Chinese generation capabilities. The processing method of the model input in the forward process is not much different from that of the general language model. First, word segmentation is performed to convert the input sentence into multiple word units, and then it is mapped to the distributed vector representation space through the word embedding layer. After passing through multiple layers of superimposed attention layers and forward layers, it is output to the generator to predict the next generated word. The main difference from the general language model in the forward calculation process is that each text to be processed needs to go through the forward transmission process of each language model in the integrated model. Therefore, one input will become multiple inputs corresponding to each language model. When the input is passed to the superimposed forward layer, the hidden state corresponding to each input word is transmitted to the forward layer that is better at processing such words under the distribution of the router. For example, the hidden state corresponding to the Chinese word is assigned to the forward layer of Qwen, and the hidden state corresponding to the English word is assigned to the forward layer of LLaMA.

[0062] It can be understood that in the embodiments of the present application, Figure 4 A multilingual application scenario was demonstrated, in which Qwen was used to understand the Chinese content of the text to be processed, and LLaMA was used to understand the English content of the text to be processed. However, in actual large-scale model application scenarios, more complex multilingual scenarios need to be dealt with. For example, Qwen itself already has good Chinese and English bilingual understanding capabilities, so a language model with more complementary language capabilities can be used for integration, and the capabilities of Qwen and LLaMA can also be complemented in multilingual tasks of small languages.

[0063] In an embodiment of the present application, a forward layer is allocated to the hidden layer state generated by the attention layer of each language model from the connected forward layers through a router, including: calculating the probability value of the first hidden layer state allocated to each connected forward layer through the router; and sorting the forward layers in order from large to small according to the probability value, and allocating the first K forward layers to the first hidden layer state for forward calculation respectively; wherein K is a natural number greater than or equal to 1, and the first hidden layer state is the hidden layer state generated by the attention layer of any language model.

[0064] It can be understood that in the embodiment of the present application, the router can be in any hidden layer state, that is, the probability value of its allocation to each forward layer connected to the router is calculated, so as to select the forward layer allocation with a higher probability value.

[0065] It should be noted that, in the embodiment of the present application, the router can calculate the probability value of any hidden layer state being assigned to the forward layer i to which the router is connected according to the following formulas (1) and (2):

[0066] h(x)=Wx (1)

[0067] p i (x) = Softmax(h i (x)) (2)

[0068] Among them, x represents the hidden layer state, w represents the weight mapping matrix used by the router, and finally the probability p assigned to the forward layer i is calculated by the normalized exponential function Softmax. i (x).

[0069] In the examples of this application, see Figure 4 The integrated model also includes a collector, which is connected between the forward layer and the generator of different language models. The method also includes: in the process of text processing by each language model, the forward calculation results of the hidden layer state generated by the attention layer of each language model are collected and sorted through the collector, and passed to the next structure of the same language model.

[0070] It should be noted that in the embodiment of the present application, for each hidden layer state, the forward layer assigned by the router may not belong to the same language model as the attention layer that generates it. When the collector transmits the forward calculation result, it needs to return the forward calculation result to the next structure of the language model to which the attention layer that generates it belongs. Since the layers in the language model are stacked together and the last layer is connected to the generator, the specific next structure can be the next layer or the generator.

[0071] It should be noted that, in the embodiment of the present application, it can be seen from the above router that assigns the forward layer to the hidden layer state that each hidden layer state can be assigned to K forward layers. Through the forward calculation of the K forward layers, if K is greater than 1, the collector can perform weighted summing according to the following formula (3) for each forward calculation result generated by the K forward layers for the same hidden layer state, and obtain its final forward calculation result y for further transmission:

[0072] y=∑ i∈k p i (x)E i (x) (3)

[0073] Among them, i represents the index of the selected K forward layers, E i (x) represents the forward calculation result of the hidden layer state x after the forward layer i.

[0074] It can be understood that in the embodiment of the present application, if K is equal to 1, that is, the router only selects one forward layer for the hidden layer state to perform forward calculation, and the collector directly transmits its forward calculation result.

[0075] In the examples of this application, see Figure 5 In the integrated model, different adapters are connected between the router and the forward layers of different language models, and different reducers are respectively connected between the collector and the generators of different language models. The method also includes: in the text processing process of each language model, each adapter is used to adapt the passed hidden layer state to the connected forward layer, and each reducer is used to adapt and restore the passed forward calculation results.

[0076] It should be noted that, in the embodiments of the present application, since each forward layer comes from a different language model, the parameter distribution between the forward layers is very different. When a forward layer directly performs forward calculations on the hidden state from other language models, the result obtained will be incompatible with the language model to which the attention layer that generates this hidden state belongs. Therefore, before being passed to the forward layer, the hidden state is first passed to the adapter for hidden state adaptation, and then the forward calculation of the forward layer is performed. Finally, the calculation result of the forward layer must be de-adapted, that is, adapted and restored, so a restorer is set after the forward layer.

[0077] In the examples of this application, see Figure 4 The integrated model also includes a scoring model, which is connected to the generator of each language model in the integrated model. The method also includes: using the scoring model, scoring the text processing results of the text to be processed generated by the generator of each language model, and outputting the text processing result with the highest score.

[0078] It is understandable that in the embodiments of the present application, the integrated model includes multiple language models, all of which can generate text processing results, so it is necessary to select the output results of multiple language models. Specifically, a scoring model can be used to select the text processing result with the highest score for output. The scoring model can specifically use a confidence scoring method for scoring, which is not limited in the embodiments of the present application.

[0079] In an embodiment of the present application, the text processing device may further perform the following steps: through a scoring model, for each language model, based on the score of the text processing result of the text to be processed generated by its respective generator, a corresponding optimization weight is set to determine the loss function of the integrated model; wherein the model loss function of the integrated model is the weighted sum of the loss functions of different language models and the corresponding optimization weights.

[0080] It should be noted that in the embodiment of the present application, during the model training phase, the scoring model needs to balance the output of each language model so as to optimize the entire integrated model in a targeted manner. Therefore, the loss function of the training integrated model is the weighted sum of the loss functions of different language models, as shown in the following formula (4):

[0081]

[0082] Among them, L i Represents the loss function of the i-th language model, W i Represents the optimization weight corresponding to the i-th language model.

[0083] It should be noted that, in the embodiment of the present application, the optimization weight can be set for each language model in the manner shown in the following formula (5):

[0084]

[0085] Among them, the first method uses a constant value of 1 as the weight, which means that all language models are optimized indiscriminately during the optimization period, and is more suitable for use in the pre-training stage. The second method uses the one-hot vector method to optimize the model only for the selected single model. The third method is the above-mentioned optimization weight corresponding to the score setting based on the text processing result, where the score of the text processing result can be obtained by calculating the confidence (Confidence):

[0086]

[0087] Among them, T represents the sequence length of the text processing result, p t It represents the probability prediction value of the language model when generating the word at position t, S iIt represents the average probability of each generated word in the process of generating text processing results. The higher the value, the higher the confidence of the text generation result of the language model. Therefore, the confidence can be used as the score of the text generation result, and the optimization weight corresponding to the language model can be set based on this. Model optimization can promote the model to handle tasks that it is better at. At the same time, models that are not good at the current task will also be optimized because their confidence is non-zero. According to the above design, the confidence method used in method 3 is more suitable for use in the instruction fine-tuning stage of the model.

[0088] It should be noted that in the embodiments of the present application, the confidence calculation result of the text processing result generated by each language model is used as its score, and the corresponding optimization weight is set based on the score, that is, the confidence calculation result. This is only an optional implementation method. Of course, other methods can also be used to score the text processing results, and the corresponding optimization weights can be set based on the scores. It is also possible to select different methods to set the optimization weights of the language model at different stages as shown in the three methods above, and the embodiments of the present application are not limited. In addition, a single language model can also be selected for text processing in the reasoning stage, and the text processing results can also be scored based on the above confidence calculation method to make a selection.

[0089] In an embodiment of the present application, a text processing device can perform model training to obtain an integrated model with better performance, as follows: using a control variable method, respectively configure the parameters of the router, each adapter, and each reducer in the integrated model; after configuring the parameters, train the integrated model, and keep the parameters of each language model in the integrated model unchanged during the training process, and optimize the parameters of the router, each adapter, and each reducer.

[0090] It is understandable that in the embodiment of the present application, the model mainly uses fine-tuning to fix the parameters in the language model during the training phase and only updates the parameters in the router, adapter and reducer. This method retains the original capabilities of each language model used to the greatest extent, thereby achieving the purpose of preventing catastrophic forgetting. On the other hand, this can make the expansion of the model more convenient. When a new large language model is added, only some parameters need to be adjusted.

[0091] It should be noted that in the embodiments of the present application, in the ideal initial state, the router should assign the hidden state from each language model to its respective forward layer, for example, assign the hidden state from LLaMA to the forward layer of LLaMA. After fine-tuning, the router gradually learns that the hidden state should be assigned to the forward layer that is more suitable for it, for example, the hidden state corresponding to most of the English words in the entire English input is assigned to LLaMA, and the hidden state corresponding to a small number of Chinese words is assigned to Qwen. However, in general, the weight mapping matrix in the router is randomly initialized, and this model method will make the training process of the model more difficult to converge. Therefore, this proposal proposes a solution of first using the control variable method to adjust the parameters step by step, and then training the model as a whole.

[0092] In an embodiment of the present application, a control variable method is used to determine the parameters of a router, each adapter, and each reducer in an integrated model, respectively, including: after removing each adapter and each reducer in the integrated model, taking the parameters of the router as variables, while keeping the parameters of each language model unchanged, performing text processing on a sample text, and configuring the parameters of the router when the processing result of the sample text is optimal; after completing the router parameter configuration, re-adding each adapter and each reducer to the integrated model, taking the parameters of each adapter and each reducer as variables, while keeping the parameters of each language model and router unchanged, performing text processing on the sample text, and configuring the parameters of the sample text when the processing result is optimal for each adapter and each reducer.

[0093] It should be noted that in the embodiment of the present application, the adapter and the reducer are removed, the router is retained, and then all language model parameters are fixed, only the parameters of the router are kept adjustable, and then the parameters are configured. In this process, since only the parameters of the router are adjustable and the parameters between the language models cannot be adapted, the optimization direction of the model can only be that the router redistributes the hidden layer states from each model to the forward layer of the model corresponding to it, thereby maintaining the compatibility between the parameters.

[0094] It should be noted that in an embodiment of the present application, on the basis of executing the above-mentioned router parameter configuration, the parameters of all models and routers are fixed and cannot be adjusted, two parts, an adapter and a reducer, are added, and the parameters of the adapter and the reducer are kept adjustable for configuration. The purpose of executing this step is to make the initial states of the adapter and the reducer correspond to the adaptation of the original language model. This step can also be chosen not to be performed after the model is integrated, and separate adaptation training can be performed in advance for each language model.

[0095] It should be noted that in the embodiments of the present application, on the basis of the above-mentioned configuration parameters of the router, adapter and reducer, the parameters of the language model are further fixed, and then the overall training of the integrated model is started. The main purpose is to perform multi-language training. By adjusting the router, adapter and reducer, different language models can be better integrated into the integrated model.

[0096] It should be noted that in the embodiments of the present application, since there are few parameters that need to be adjusted in the model training phase, only a small amount of data is needed for training. In view of the aforementioned configuration of the parameters of the router, adapter and reducer using the control variable method, the data required for the initialization step is less. Finally, the training of the entire integrated model, that is, the fine-tuning of the models of different languages, requires the most data. Therefore, the data can be divided in a 1:3:6 data allocation method, which is used for the router parameter configuration, the adapter and reducer parameter configuration, and the steps of the overall training of the integrated model.

[0097] The following describes in detail the differences between the integrated model provided in the embodiments of the present application and other types of multilingual models.

[0098] Figure 6 The following is a scheme to directly extend the general language model to a multilingual model, which is similar to Figure 3 Compared with the overall solution of integrating different language models into an integrated model provided by the present application, Figure 6 The model structure shown is to expand a single language model into a multi-language model, for example, to expand LLaMA into LLaMA-MoE. It only needs to use the existing method to copy the forward layer of LLaMA multiple times and then continue pre-training so that different forward layers have different parameter differentiation. However, this method cannot integrate multiple different language models, for example, integrating the two large language models of LLaMA and Qwen into a multi-language large model system with Chinese and English bilingual capabilities. Although these two large language models have the same model structure, they are trained separately, so the parameter distribution of the two is quite different, so the forward layers of different models cannot be directly combined.

[0099] Figure 7 Another solution for integrating multiple language models into a model system is shown. This solution is very different from the method adopted by the present application in achieving the integration of different language models: the present application and Figure 7 The main difference between the schemes is that the integration method of this application is based on the hybrid expert system structure, which integrates the forward layers of multiple models together using routers, while Figure 7The solution uses the cross attention mechanism to transfer the hidden vector encoding knowledge in the auxiliary model (Augmenting Model) to the main model (Anchor Model). Figure 7 The solution uses a master-slave structure to integrate multiple models. The output of the model is generated by the master model. Generally, the master model uses a large model with a larger scale and stronger reasoning ability, such as the 70B scale LLaMA. The slave model is mainly used for rapid updates when new knowledge is introduced, so a small model with relatively easier parameter adjustment is generally used, such as the 7B scale LLaMA. When reasoning about languages ​​or tasks that are not good at, the master model can use the updated knowledge in the slave model to realize cross-model external knowledge utilization.

[0100] It should be noted that the scheme using the main-auxiliary structure is largely limited by the ability of the auxiliary model when reasoning about new languages ​​or tasks, so the parameter scale of the auxiliary model should not be too small. However, the integration scheme using this main-auxiliary structure is mainly for updating knowledge more quickly, so it is not appropriate to use a larger-scale auxiliary model. If the scale of the auxiliary model is comparable to that of the main model, the model integration in this way will lose its meaning. The problem to be solved by this application is essentially different from the scheme of the main-auxiliary structure. This application emphasizes that multiple language models with different language text processing capabilities share language capabilities through model integration. The integrated model has the ability to process multi-language texts, and each language model in the integrated model still has independent generation capabilities.

[0101] Based on the above detailed description, it can be seen that the technical solution provided by the present application has the following advantages: First, instead of large-scale multilingual pre-training, a model integration method is adopted, mainly to fine-tune a small number of parameters of the integrated module, which can save a lot of computing resources and make full use of existing resources; Second, the use of adapters and reducers to combine parameters from different language models solves the problem of not being able to integrate multiple different language models; Third, in general multilingual models, there is actually no clear division of language, function, field, etc. between the forward layers. Although the parameters of the forward layers are different, they belong to the same distribution and have basically the same functions, which mainly has the advantage of increasing the width of the model. In the present application, each forward layer comes from a different language model with a different parameter distribution, but each language model is responsible for the corresponding language, so the function is clearer; Fourth, compared with the language expansion method, the integrated model of the present application only needs to adjust a small number of parameters to achieve parameter adaptation during the training stage, and does not need to adjust the parameters of the language model, thereby avoiding catastrophic forgetting, and can more conveniently add new language models, with good scalability.

[0102] An embodiment of the present application provides a text processing device. Figure 8A structural diagram of a text processing device provided in an embodiment of the present application Figure 1 .like Figure 8 As shown, in an embodiment of the present application, the text processing device includes:

[0103] The model integration module 201 is used to integrate different language models with the same structure and different language text processing capabilities using a router to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models;

[0104] The text processing module 202 is used to input the text to be processed into each language model in the integrated model respectively; during the text processing process of each language model, the router is used to allocate a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers to perform forward calculation.

[0105] In one embodiment of the present application, the router is used to calculate the probability value of the first hidden layer state being assigned to each connected forward layer; and, sorting the forward layers in order of probability value from large to small, and assigning the first K forward layers to the first hidden layer state for respectively performing forward calculations; wherein K is a natural number greater than or equal to 1, and the first hidden layer state is the hidden layer state generated by the attention layer of any language model.

[0106] In one embodiment of the present application, the integrated model also includes a collector, which is connected between the forward layer and the generator of different language models, and is used to collect and organize the forward calculation results of the hidden layer state generated by the attention layer of each language model during the text processing process of each language model, and pass them to the next structure of the same language model.

[0107] In one embodiment of the present application, in the integrated model, different adapters are connected between the router and the forward layers of different language models, and different reducers are respectively connected between the collector and the generators of different language models;

[0108] In the text processing process of each language model, each adapter is used to adapt the passed hidden layer state to the connected forward layer, and each reducer is used to adapt and restore the passed forward calculation results.

[0109] In one embodiment of the present application, the integrated model also includes a scoring model, which is connected to the generator of each language model in the integrated model. The scoring model is used to score the text processing results of the text to be processed generated by the generator of each language model, and output the text processing result with the highest score.

[0110] In one embodiment of the present application, the scoring model is further used to set corresponding optimization weights for the scores of the text processing results of the to-be-processed text generated by the respective generators for each language model, so as to determine the loss function of the integrated model;

[0111] The model loss function of the integrated model is the weighted sum of the loss functions of different language models and the corresponding optimization weights.

[0112] In one embodiment of the present application, the model integration module 201 is also used to adopt a control variable method to respectively configure the parameters of the router, each adapter and each reducer in the integrated model; after configuring the parameters, the integrated model is trained, and during the training process, the parameters of each language model in the integrated model are kept unchanged to optimize the parameters of the router, each adapter and each reducer.

[0113] In one embodiment of the present application, the model integration module 201 is used to remove each adapter and each reducer in the integrated model, use the parameters of the router as variables, while keeping the parameters of each language model unchanged, perform text processing on the sample text, and configure the parameters of the router when the processing result of the sample text is optimal; after completing the configuration of the router parameters, re-add each adapter and each reducer to the integrated model, use the parameters of each adapter and each reducer as variables, while keeping the parameters of each language model and the router unchanged, perform text processing on the sample text, and configure the parameters of the sample text when the processing result is optimal for each adapter and each reducer.

[0114] Fig. 9 A structural diagram of a text processing device provided in an embodiment of the present application Figure 2 .like Fig. 9 As shown, the text processing device includes: a processor 301, a memory 302 and a communication bus 303;

[0115] The communication bus 303 is used to realize the communication connection between the processor 301 and the memory 302;

[0116] The processor 301 is used to execute one or more computer programs stored in the memory 302 to implement the above text processing method.

[0117] An embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned text processing method when executed.

[0118] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above-mentioned text processing method when executed by a processor. The computer-readable storage medium can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or it can be a respective device including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0120] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the utility application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A text processing method, characterized in that: include: A router is used to integrate different language models with the same structure and different language text processing capabilities to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models; The text to be processed is input into each language model in the integrated model respectively. During the text processing process of each language model, the router allocates a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers to perform forward calculation.

2. The method according to claim 1, characterized in that The router allocates a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers, comprising: By means of the router, the probability values ​​of the first hidden layer state being assigned to each connected forward layer are calculated; And, sorting the forward layers in descending order of probability values, and assigning the first K forward layers to the first hidden layer state for respectively performing forward calculations; Wherein, K is a natural number greater than or equal to 1, and the first hidden layer state is a hidden layer state generated by the attention layer of any language model.

3. The method according to claim 1, characterized in that The integrated model further includes a collector, which is connected between the forward layers and generators of different language models. The method further includes: During the text processing of each language model, the forward calculation results of the hidden layer state generated by the attention layer of each language model are collected and sorted through the collector, and passed to the next structure of the same language model.

4. The method according to claim 3, characterized in that In the integrated model, different adapters are connected between the router and the forward layers of different language models, and different reducers are respectively connected between the collector and the generators of different language models. The method further includes: During the text processing of each language model, each adapter is used to adapt the passed hidden layer state to the connected forward layer, and each reducer is used to adapt and restore the passed forward calculation results.

5. The method according to claim 1, characterized in that: The integrated model further includes a scoring model, wherein the scoring model is connected to the generator of each language model in the integrated model, and the method further includes: The scoring model is used to score the text processing results of the text to be processed generated by the generator of each language model, and the text processing result with the highest score is output.

6. The method according to claim 5, characterized in that The method further comprises: By means of the scoring model, for each language model, a corresponding optimization weight is set for scoring the text processing result of the to-be-processed text generated by the respective generator, so as to determine the loss function of the integrated model; The model loss function of the integrated model is the weighted sum of the loss functions of different language models and the corresponding optimization weights.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Using a control variable method, respectively configure the parameters of the router, each adapter and each reducer in the integrated model; After configuring the parameters, the integrated model is trained, and during the training process, the parameters of each language model in the integrated model are kept unchanged, and the parameters of the router, each adapter and each reducer are optimized.

8. The method according to claim 7, characterized in that The control variable method is used to respectively determine the parameters of the router, each adapter and each reducer in the integrated model, including: After removing each adapter and each reducer in the integrated model, the parameters of the router are used as variables, while the parameters of each language model are kept unchanged, the sample text is processed, and the parameters when the processing result of the sample text is optimal are configured for the router; After completing the router parameter configuration, each adapter and each reducer are re-added to the integrated model, and the parameters of each adapter and each reducer are used as variables. At the same time, the parameters of each language model and the router are kept unchanged, and text processing is performed on the sample text. The parameters that produce the optimal processing result of the sample text are configured for each adapter and each reducer.

9. A text processing device, characterized in that: include: A model integration module, which is used to integrate different language models with the same structure and different language text processing capabilities using a router to obtain an integrated model; each language model includes a cascaded vocabulary, a word embedding layer, an attention layer, a forward layer and a generator, and the router is connected between the forward layer and the attention layer of different language models; The text processing module is used to input the text to be processed into each language model in the integrated model respectively; during the text processing process of each language model, the router is used to allocate a forward layer to the hidden layer state generated by the attention layer of each language model from the connected forward layers to perform forward calculation.

10. A text processing device, characterized in that: include: processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute one or more computer programs stored in the memory to implement the text processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the text processing method according to any one of claims 1 to 8 is implemented.

Images