Information processing method and device

Abstract

The embodiment of the invention provides an information processing method and device, and the method comprises the steps: obtaining problem information corresponding to a natural language processing task, and inputting the problem information into a hybrid expert model; selecting a first expert model for the question information in the hybrid expert model, and determining a second expert model which is not selected in the hybrid expert model; constructing an embedded vector of the second expert model, and processing the embedded vector through a knowledge migration unit in the hybrid expert model to obtain a knowledge projection matrix; and updating the first expert model based on the knowledge projection matrix, and processing the question information by using the updated first expert model to obtain answer information.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of machine learning technology, and more particularly to information processing methods and devices. Background Art

[0002] With the development of computer technology, the Mixture of Experts (MoE) model has received widespread attention and application as an effective way to expand model capacity. Traditional dense models, such as GPT-3 and BERT, have performed well in natural language processing (NLP) tasks, but as the model size continues to increase, the computational requirements and resource consumption of these models have also increased exponentially. To solve this problem, the Sparse Activation model came into being, which can only activate a part of the parameters in the model when processing each input, thereby reducing the computational cost. The Mixture of Experts model is a representative method in the Sparse Activation model. The basic idea of ​​the MoE model is to dynamically select a specific subset of experts for each input for processing by introducing experts and gating mechanisms, thereby improving the processing capacity of the model without significantly increasing computing resources. In the prior art, the gating mechanism in the MoE model usually adopts a Top-k selection strategy, that is, the top k experts with the highest responses are selected to participate in the calculation. This strategy guarantees sparsity to a certain extent, but also limits the amount of available expert knowledge, thus affecting the overall performance of the model. Especially when dealing with complex tasks, selecting only a few experts may not be enough to fully utilize the capabilities of the model. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide an information processing method. One or more embodiments of this specification also relate to an information processing apparatus, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided an information processing method, including: Obtaining question information corresponding to a natural language processing task, and inputting the question information into a hybrid expert model; Selecting a first expert model in the hybrid expert model for the problem information, and determining a second expert model that is not selected in the hybrid expert model; Constructing an embedding vector of the second expert model, and processing the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The first expert model is updated based on the knowledge projection matrix, and the question information is processed using the updated first expert model to obtain answer information.

[0005] Optionally, selecting a first expert model in the hybrid expert model for the question information, and determining a second expert model not selected in the hybrid expert model, comprises: Calculating the matching degree between the question information and each expert model through the routing unit in the hybrid expert model; Compare the matching degree corresponding to each expert model with the preset matching degree threshold; According to the comparison result, an expert model with a matching degree greater than the preset matching degree threshold is selected as the first expert model, and expert models other than the first expert model are used as unselected second expert models.

[0006] Optionally, constructing the embedding vector of the second expert model includes: Determining embedding information and selection information corresponding to the second expert model; Inputting the embedding information and the selection information into the multi-layer perceptron network in the hybrid expert model for processing to obtain a selection embedding vector; The layer embedding vector corresponding to the second expert model and the selected embedding vector are concatenated to obtain an embedding vector.

[0007] Optionally, the processing the embedding vector by a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix includes: Inputting the embedding vector into the knowledge transfer unit in the hybrid expert model for processing to obtain an upper projection matrix and a lower projection matrix; The upper projection matrix and the lower projection matrix are used as knowledge projection matrices.

[0008] Optionally, the updating the first expert model based on the knowledge projection matrix, and processing the question information using the updated first expert model to obtain answer information includes: updating the first expert model based on the knowledge projection matrix, and obtaining a third expert model according to the updating result; Inputting the question information into the third expert model, and processing the question information through the original processing unit and the knowledge sharing unit in the third expert respectively to obtain first answer information and second answer information; Answer information corresponding to the question information is generated according to the first answer information and the second answer information.

[0009] Optionally, generating answer information corresponding to the question information according to the first answer information and the second answer information includes: Determine a first weight coefficient corresponding to the original processing unit and a second weight coefficient corresponding to the knowledge sharing unit; Performing a weighted sum calculation on the first answer information and the first weight coefficient, and on the second answer information and the second weight coefficient; The answer information corresponding to the question information is generated according to the weighted sum calculation result.

[0010] Optionally, the training of the hybrid expert model includes: Select sample question information in the sample set; Inputting the sample question information into the initial hybrid expert model for processing to obtain predicted answer information; Training the initial hybrid expert model based on the sample answer information corresponding to the sample question information and the predicted answer information until the hybrid expert model that meets the training stop condition is obtained; Wherein, the unit parameters corresponding to the knowledge transfer unit in the hybrid expert model are updated through reverse propagation.

[0011] According to a second aspect of an embodiment of this specification, another information processing method is provided, which is applied to a server, including: Receiving question information submitted by a client, and inputting the question information into the hybrid expert model; Selecting a first expert model in the hybrid expert model for the problem information, and determining a second expert model that is not selected in the hybrid expert model; Constructing an embedding vector of the second expert model, and processing the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The first expert model is updated based on the knowledge projection matrix, and the question information is processed using the updated first expert model to obtain answer information; The answer information is fed back to the client.

[0012] According to a third aspect of the embodiments of this specification, there is provided an information processing device, including: An acquisition module is configured to acquire question information corresponding to a natural language processing task and input the question information into a hybrid expert model; A selection module is configured to select a first expert model in the mixed expert model for the question information, and determine a second expert model that is not selected in the mixed expert model; A construction module is configured to construct an embedding vector of the second expert model, and process the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The updating module is configured to update the first expert model based on the knowledge projection matrix, and use the updated first expert model to process the question information to obtain answer information.

[0013] According to a fourth aspect of the embodiments of this specification, another information processing device is provided, which is applied to a server, including: An information receiving module is configured to receive question information submitted by a client and input the question information into the hybrid expert model; A model selection module is configured to select a first expert model in the mixed expert model for the question information, and determine a second expert model that is not selected in the mixed expert model; A vector construction module is configured to construct an embedding vector of the second expert model, and process the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; an updating model module, configured to update the first expert model based on the knowledge projection matrix, and use the updated first expert model to process the question information to obtain answer information; The information sending module is configured to feed back the answer information to the client.

[0014] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned information processing method are implemented.

[0015] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned information processing method are implemented.

[0016] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned information processing method when executed by a processor.

[0017] The information processing method provided in this embodiment, in order to optimize the balance between the sparsity of expert model selection and knowledge utilization, can first input the question information into the hybrid expert model after obtaining the question information corresponding to the natural language processing task, at this time, first select the first expert model for the question information in the hybrid expert model, and determine the second expert model that is not selected in the hybrid expert model; on this basis, the embedding vector of the second expert model can be first constructed, and the embedding vector can be processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix; on this basis, the first expert model can be updated based on the knowledge projection matrix, so that the knowledge information of the unselected expert model can be supplemented to the selected expert model, and finally the updated first expert model can be used to process the question information to obtain the answer information. In the processing process, the sparsity of expert selection can be guaranteed by knowledge transfer, while the availability of expert model knowledge can be improved, thereby improving the performance of the model in complex tasks, and when transferring knowledge, only positive knowledge is selected for transfer, avoiding the transmission of negative knowledge, thereby reducing the prediction risk brought by negative knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an information processing method provided by an embodiment of this specification; Figure 2 is a flow chart of an information processing method provided by an embodiment of this specification; Figure 3 is a flow chart of another information processing method provided by an embodiment of this specification; Figure 4 is a processing flow chart of an information processing method provided by an embodiment of this specification; Figure 5 is a structural diagram of an information processing device provided by an embodiment of this specification; Figure 6 is a schematic diagram of the structure of another information processing device provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0019] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0020] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0022] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0023] First, the terms involved in one or more embodiments of this specification are explained.

[0024] The Mixture of Experts (MoE) model is an efficient neural network architecture that improves overall performance by combining multiple specialized sub-models (i.e., "experts"). The core idea of ​​MoE is to decompose complex tasks into multiple sub-tasks and assign them to different expert models for processing, while using a gating network to dynamically select and assign tasks to appropriate expert networks.

[0025] In this specification, an information processing method is provided. This specification also relates to an information processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0026] See also Figure 1As shown in the schematic diagram, the information processing method provided by this embodiment can optimize the balance between the sparsity of expert model selection and knowledge utilization. After obtaining the problem information corresponding to the natural language processing task, the problem information can be first input into the hybrid expert model. At this time, the first expert model is first selected for the problem information in the hybrid expert model, and the second expert model not selected in the hybrid expert model is determined; on this basis, the embedding vector of the second expert model can be first constructed, and the embedding vector is processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix; on this basis, the first expert model is updated based on the knowledge projection matrix, and the knowledge information of the unselected expert model can be supplemented to the selected expert model. Finally, the updated first expert model can be used to process the problem information to obtain the answer information. In the processing process, the sparsity of expert selection can be guaranteed by knowledge transfer, while the availability of expert model knowledge can be improved, thereby improving the performance of the model in complex tasks. At the same time, when transferring knowledge, only positive knowledge is selected for transfer, avoiding the transmission of negative knowledge, thereby reducing the prediction risk brought by negative knowledge.

[0027] See also Figure 2 , Figure 2 A flowchart of an information processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0028] Step S202, obtaining question information corresponding to the natural language processing task, and inputting the question information into the hybrid expert model.

[0029] The information processing method provided in this embodiment can be applied to the use scenarios of hybrid expert models in any field, such as processing natural language processing tasks (such as text classification, text generation, and machine translation) through hybrid expert models, processing computer vision tasks (such as image classification, target detection, and image generation) through hybrid expert models, processing recommendation tasks (such as personalized recommendations and advertising placement) through hybrid expert models, and processing audio tasks (such as speech recognition and noise elimination) through hybrid expert models. This embodiment takes the application of hybrid expert models in natural language processing tasks as an example to illustrate the information processing method. The descriptions in other scenarios can refer to the descriptions in this embodiment, and this embodiment will not be repeated in detail here.

[0030] Specifically, question information refers to question information triggered by users or upstream tasks in natural language processing tasks, which is used to generate information corresponding to the question information after being input into the hybrid expert model, and then fed back to the user or used by downstream tasks. For example, when the user inputs the question information "Who is the author of ancient poem A", after processing it through the hybrid expert model, the answer information "The author of ancient poem A is A, A was born in ** year..." can be obtained. Correspondingly, the hybrid expert model is a MoE model used to generate answer information for question information in natural language processing tasks, which contains multiple expert models for selection in the answer generation stage after the question information is input, and the model also contains a knowledge transfer unit, which is used to transfer the relevant knowledge of the unselected expert model to the selected expert model for joint calculation without occupying more computing resources.

[0031] In practical applications, the hybrid expert model can select only a small number of experts for each input through multiple parallel expert models and a gating network. This design not only greatly reduces the computational cost, but also achieves performance comparable to that of dense models on specific tasks. However, when the gating network selects the expert model, negative knowledge transfer may occur due to incomplete or misleading information. This negative knowledge transfer will weaken the performance of the model, especially in multi-task learning scenarios, where mutual interference between different tasks may further aggravate this problem. At the same time, the expert selection and calculation processes between different layers of the hybrid expert model are independent. Although this can simplify the implementation of the model, it also leads to the isolation of information between different layers, limiting the parameter efficiency and computational efficiency of the model. Therefore, an effective solution is urgently needed to solve the above problems.

[0032] Based on this, in order to optimize the balance between the sparsity of expert model selection and knowledge utilization, after obtaining the problem information of the corresponding natural language processing task, the problem information can be first input into the hybrid expert model. At this time, the first expert model is first selected for the problem information in the hybrid expert model, and the second expert model that is not selected in the hybrid expert model is determined; on this basis, the embedding vector of the second expert model can be first constructed, and the embedding vector is processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix; on this basis, the first expert model is updated based on the knowledge projection matrix, and the knowledge information of the unselected expert model can be supplemented to the selected expert model. Finally, the updated first expert model can be used to process the problem information to obtain the answer information. In the processing process, the knowledge transfer method can ensure the sparsity of expert selection while improving the availability of expert model knowledge, thereby improving the performance of the model in complex tasks. At the same time, when transferring knowledge, only positive knowledge is selected for transfer, avoiding the transfer of negative knowledge, thereby reducing the prediction risk brought by negative knowledge. Furthermore, by sharing information between selected experts and unselected experts, it can not only improve the prediction accuracy of the model, but also improve the performance of the model in processing multi-level complex information by sharing information between layers. This makes it easier to use in actual business scenarios and provides more accurate and stable prediction results.

[0033] Step S204: selecting a first expert model in the hybrid expert model according to the question information, and determining a second expert model that is not selected in the hybrid expert model.

[0034] Specifically, after the problem information is input into the hybrid expert model, in order to achieve knowledge sharing between the selected experts and the unselected experts, the first expert model can be selected in the hybrid expert model for the problem information, and the second expert model that is not selected in the hybrid expert model for the problem information processing task can be determined. This facilitates the subsequent knowledge transfer operation. The first expert model specifically refers to the expert model that needs to be calculated in the hybrid expert model for the task processing, and the second expert model specifically refers to the expert model that does not need to participate in the calculation in the hybrid expert model for the task processing.

[0035] Furthermore, when determining the first expert model and the second expert model, it can be accomplished by a routing unit of a hybrid expert model. In this embodiment, the specific implementation is as follows: The matching degree between the question information and each expert model is calculated by a routing unit in the hybrid expert model; the matching degree corresponding to each expert model is compared with a preset matching degree threshold; and an expert model having a matching degree greater than the preset matching degree threshold is selected as a first expert model according to the comparison result, and expert models other than the first expert model are selected as unselected second expert models.

[0036] Specifically, the routing unit specifically refers to the gated network in the hybrid expert model. Correspondingly, the matching degree specifically refers to the numerical value of the matching degree between the problem information and each expert model calculated by the routing unit. The matching degree reflects the correlation between the problem information and each expert model. The higher the matching degree, the stronger the correlation and the higher the probability of being selected. Correspondingly, the preset matching degree threshold specifically refers to the threshold for screening the problem information processing according to actual needs. It can be set according to actual needs, and this embodiment does not make any limitation here.

[0037] Based on this, after the question information is input into the hybrid expert model, the matching degree between the question information and each expert model can be calculated through the routing unit in the hybrid expert model; at this time, the matching degree corresponding to each expert model can be compared with the preset matching degree threshold; by selecting an expert model with a matching degree greater than the preset matching degree threshold as the first expert model, the answer information can be completed more accurately, and at the same time, other expert models except the first expert model can be used as unselected second expert models, so as to realize subsequent knowledge transfer operations.

[0038] In practical applications, when selecting the first expert model, in order to improve prediction efficiency and accuracy, the number of first expert models can be determined according to actual needs, and this embodiment does not impose any restrictions thereon. The number of unselected second expert models will depend on the number of remaining unselected expert models.

[0039] In specific implementation, the routing unit can extract and analyze the features of the input question information to understand the content and context of the question, and then calculate a probability distribution based on the features corresponding to the question information. This distribution reflects the suitability of each expert model for the current input question information. Then, one or more expert models with the highest matching degree can be selected and activated according to the probability distribution as the first expert model. The remaining ones are used as the second expert model for subsequent knowledge transfer operations.

[0040] For example, the user inputs the question text "Who is the author of ancient poem A?" After inputting the question text into the MoE model (including 3 expert models), the routing unit in the MoE model can calculate the matching degree between the question text and the 3 expert models respectively, and select expert model 1 and expert model 2 with a higher matching degree with the question text according to the calculation results. At this time, expert model 1 and expert model 2 can be selected as the selected expert models, and the remaining unselected expert model 3 can be used as the unselected expert model, so as to improve the accuracy of answer generation to the question text through subsequent knowledge transfer.

[0041] In summary, by selecting the expert model with a higher matching degree as the first expert model and the remaining expert models as the second expert model, it is possible to divide the expert models that participate in the calculation and those that do not participate in the calculation, thereby facilitating the subsequent knowledge transfer operation of the unselected expert model.

[0042] Step S206: construct an embedding vector of the second expert model, and process the embedding vector through the knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix.

[0043] Specifically, after the first expert model and the second expert model are determined as described above, further, in order to ensure that the unselected expert model can assist in the generation of answer information corresponding to the question information while ensuring sparsity, the embedding vector of the second expert model can be constructed first, and then the embedding vector is processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix, thereby realizing the conversion of the knowledge information of the unselected second expert model into the knowledge projection matrix. After sharing it as the first expert model, it can be used together in the answer information generation stage, so that the selected expert model can integrate the knowledge information of the unselected expert model, and the second expert model can be prevented from directly participating in the calculation, thereby ensuring that the model prediction accuracy is improved without increasing the computing resource overhead.

[0044] Among them, the embedding vector specifically refers to the embedding vector constructed for the second expert model, which is used to contain the predictive knowledge information possessed by the second expert model. Correspondingly, the knowledge transfer unit specifically refers to the network in the hybrid expert model used to transfer the knowledge information of the unselected expert model to the selected expert model. Its main purpose is to enable information sharing between expert models, and to transfer the knowledge of the unselected expert model to the selected expert model through the conditional generation mechanism. Correspondingly, the knowledge projection matrix specifically refers to the weight matrix that can be used in the prediction stage of the first expert model after the knowledge transfer processing is performed on the embedding vector corresponding to the second expert model, and the projection matrix contains the knowledge information of the second expert model. Converting it into a projection matrix can facilitate the calculation and use of the first expert model.

[0045] Furthermore, when constructing the embedding vector of the second expert model, in order to complete the knowledge transfer operation through the embedding vector, the layer embedding of the second expert model and the embedding concatenation can be selected to obtain the embedding vector. In this embodiment, the specific implementation method is as follows: Determine the embedding information and selection information corresponding to the second expert model; input the embedding information and the selection information into the multilayer perceptron network in the hybrid expert model for processing to obtain a selection embedding vector; concatenate the layer embedding vector corresponding to the second expert model and the selection embedding vector to obtain an embedding vector.

[0046] Specifically, the embedding information specifically refers to the initial embedding vector corresponding to the second expert model, and correspondingly, the selection information specifically refers to a binary scalar reflecting whether the second expert model is selected. Correspondingly, the selection embedding vector specifically refers to the selection vector expression obtained by combining the unselected scalar of the second expert model and the initial embedding vector through a multi-layer perceptron network. Correspondingly, the layer embedding vector specifically refers to the vector expression corresponding to the layer of the second expert model in the hybrid expert model.

[0047] Based on this, before performing knowledge transfer, it is necessary to first process the knowledge information of the second expert model into a data structure that can be used by the first expert model in the prediction stage. Therefore, the embedding information and selection information corresponding to the second expert model can be determined first; at this time, the embedding information and selection information can be first input into the multilayer perceptron network in the mixed expert model for processing, so that after processing it through the multilayer perceptron network, the selection embedding vector corresponding to the second expert model can be obtained; on this basis, by splicing the layer embedding vector and the selection embedding vector corresponding to the second expert model, the embedding vector that fully reflects the knowledge information of the second expert model can be obtained, and the embedding vector can be processed later to obtain the projection matrix needed to be used in the application stage of the first expert model.

[0048] In practical applications, when constructing the embedding vector corresponding to the second expert model, the multi-layer perceptron network MLP in the hybrid expert model can be used to process the embedding information of the unselected second expert model to generate the selected embedding vector. Specifically, assuming that there are N unselected expert models, and for the i-th input token, the calculation of its selected embedding vector can be obtained by the following formula (1): (1) Among them, p i represents the selection of embedding vector, s j represents the initial embedding vector of the jth expert model, z i，j A binary scalar indicating whether the j-th expert model is selected. MLP is a multi-layer perceptron network.

[0049] After obtaining the selection embedding vector corresponding to the second expert model, the selection embedding vector can be concatenated with the layer embedding vector corresponding to the second expert model to obtain the embedding vector corresponding to the second expert model, so as to facilitate the subsequent conversion of the embedding vector into a projection matrix for fusion on the first expert model to generate answer information.

[0050] In summary, by concatenating the selected embedding vector and the layer embedding vector of the second expert model, the embedding vector corresponding to the knowledge information corresponding to the second expert model is constructed, which can facilitate the subsequent construction of the knowledge projection matrix to ensure that the matrix can be used in the prediction stage of the first expert model, thereby achieving knowledge sharing without increasing the computational cost.

[0051] Furthermore, in order to integrate the knowledge information of the second expert model into the first expert model for this calculation, a knowledge projection matrix can be constructed in combination with the knowledge transfer unit. In this embodiment, the specific implementation method is as follows: The embedding vector is input into the knowledge transfer unit in the hybrid expert model for processing to obtain an upper projection matrix and a lower projection matrix; and the upper projection matrix and the lower projection matrix are used as knowledge projection matrices.

[0052] Based on this, after obtaining the embedding vector corresponding to the second expert model, the embedding vector can be input into the knowledge transfer unit in the hybrid expert model for processing, so as to realize the projection matrix construction through the knowledge transfer unit, and obtain the upper projection matrix and the lower projection matrix; thereafter, the upper projection matrix and the lower projection matrix can be used as the knowledge projection matrix for subsequent generation of answer information in combination with the first expert model.

[0053] In practical applications, after the embedding vector corresponding to the second expert model is input into the knowledge transfer unit, the upper projection matrix and the lower projection matrix can be generated by the following formula (2): (2) Among them, He (·) is the knowledge transfer unit, W D and W U Represents the matrix used to generate the lower projection and the projection matrix The weight matrix of , and then the two are used as the knowledge projection matrix, which can be used to generate answer information in combination with the first expert model.

[0054] Using the above example, after determining that the selected expert models are expert model 1 and expert model 2, and that the unselected expert model is expert model 3, the selected embedding vector and layer embedding vector corresponding to expert model 3 can be further determined by the above formula (1). Then, by concatenating the selected embedding vectors and layer embedding vectors corresponding to them and processing them through a multi-layer perceptron network, the embedding vector VT3 corresponding to expert model 3 can be obtained. Then, the embedding vector corresponding to expert model 3 can be input into the hypernetwork (knowledge transfer unit) in the MoE model for processing. Then, the lower projection matrix and upper projection matrix corresponding to expert model 3 can be generated through the hypernetwork. After that, the lower projection matrix and upper projection matrix are calculated together with expert models 1 and 2 to obtain the prediction result fused into the selected expert model.

[0055] In summary, by converting the embedding vector of the unselected expert model into a knowledge projection matrix, the first expert model can be combined with the matrix to complete the calculation in the application stage, thereby achieving the purpose of integrating the knowledge information of the second expert model for prediction, thereby effectively improving the prediction accuracy and completing the knowledge sharing between expert models.

[0056] Step S208: updating the first expert model based on the knowledge projection matrix, and processing the question information using the updated first expert model to obtain answer information.

[0057] Specifically, after obtaining the knowledge projection matrix, in order to enable the first expert model to complete the prediction process in combination with the knowledge projection matrix in the prediction stage, the first expert model can be updated based on the knowledge projection matrix to integrate the knowledge information of the second expert model into the first expert model, and then the updated first expert model can be used to process the question information, so as to obtain the answer information according to the processing result, and then feed it back to the user or use it downstream. The answer information is the answer text generated after processing the question information.

[0058] Furthermore, after the first expert model is updated using the knowledge projection matrix, the knowledge information of the unselected experts can be migrated to the selected expert model. On this basis, in order to realize the processing of the question information, the knowledge information of the second expert model can be integrated, and the integration of the answer information can be completed by predicting the first answer information and the second answer information. In this embodiment, the specific implementation method is as follows: The first expert model is updated based on the knowledge projection matrix, and a third expert model is obtained according to the update result; the question information is input into the third expert model, and the question information is processed by the original processing unit and the knowledge sharing unit in the third expert respectively to obtain first answer information and second answer information; answer information corresponding to the question information is generated according to the first answer information and the second answer information.

[0059] Specifically, the third expert model specifically refers to the expert model obtained after integrating the knowledge projection matrix, which can use the knowledge information of the second expert model and the prediction ability of the first expert model to jointly generate the answer information in the prediction stage. Correspondingly, the original processing unit specifically refers to the processing unit corresponding to the prediction ability of the first expert model in the third expert model, and the knowledge sharing unit specifically refers to the processing unit corresponding to the knowledge projection matrix of the second expert model in the third expert model. Correspondingly, the first answer information is the answer information output corresponding to the prediction ability of the first expert model. Correspondingly, the second answer information is specifically the answer information obtained by the knowledge projection matrix corresponding to the second expert model.

[0060] Based on this, after updating the first expert model based on the knowledge projection matrix, the third expert model can be obtained according to the update result; on this basis, the question information can be input into the third expert model. In order to enable the expert model to combine the knowledge information of the unselected expert models in the prediction stage, the prediction fusion method can be used to complete the generation of the answer information.

[0061] That is to say, the original processing unit and the knowledge sharing unit in the third expert can process the question information respectively, and the first answer information corresponding to the original processing unit and the second answer information corresponding to the knowledge sharing unit can be obtained. On this basis, by fusing the first answer information and the second answer information, a more accurate answer information can be obtained.

[0062] In summary, by using the original processing unit and the knowledge sharing unit to process respectively and fuse the output answer information to construct the answer information corresponding to the question information, the knowledge fusion of the unselected experts and the selected experts can be achieved, thereby ensuring sparsity while improving the prediction accuracy.

[0063] On this basis, considering that the main processing model of the question information is the first expert model, and the knowledge information of the second expert model plays an auxiliary role, the answer information can be weighted according to the prediction sources of different answer information. In this embodiment, the specific implementation method is as follows: Determine a first weight coefficient corresponding to the original processing unit and a second weight coefficient corresponding to the knowledge sharing unit; perform weighted sum calculation on the first answer information and the first weight coefficient, and the second answer information and the second weight coefficient; and generate answer information corresponding to the question information based on the weighted sum calculation result.

[0064] Specifically, the first weight coefficient specifically refers to the weight value corresponding to the original processing unit. Correspondingly, the second weight coefficient specifically refers to the weight value corresponding to the knowledge sharing unit. In practical applications, the first weight coefficient and the second weight coefficient can be set according to actual needs, and this embodiment does not make any limitation here.

[0065] Based on this, after obtaining the first answer information and the second answer information, the answer information corresponding to the question information can be determined by weighted sum. Specifically, the first weight coefficient corresponding to the original processing unit and the second weight coefficient corresponding to the knowledge sharing unit can be determined; on this basis, the first answer information and the first weight coefficient, as well as the second answer information and the second weight coefficient can be weighted and calculated; thus, the answer information corresponding to the question information can be generated according to the weighted and calculated result.

[0066] In practical applications, after the projection matrix is ​​obtained by the above formula (2), it can be used together with the selected first expert model for calculation, wherein the calculation formula of the knowledge sharing unit corresponding to the second answer information can be implemented by the following formula (3): (3) Among them, x i Indicates the token corresponding to the question information.

[0067] On this basis, the output of the hybrid expert model is the weighted sum of the output of the selected expert model and the output of the knowledge sharing unit, which can be achieved by the following formula (4): (4) where G(·) represents the output of the gating mechanism, is the output of the selected expert model, It is the output of the knowledge sharing unit.

[0068] In addition, in order to further improve the parameter efficiency and computational efficiency of the model, the knowledge transfer unit can be shared between multiple Transformer layers. This method can effectively reduce the number of model parameters while allowing information flow between different layers, thereby effectively improving the overall performance of the model. During the Transformer calculation process of each layer, the knowledge transfer unit generates the knowledge projection matrix of the corresponding layer based on the selected embedding and layer embedding, thereby realizing cross-layer information sharing and knowledge transfer. Through the knowledge transfer unit, the model can effectively transfer useful expert knowledge between different layers, avoiding the problem of information isolation.

[0069] Using the above example, after obtaining the lower projection matrix and upper projection matrix corresponding to expert model 3, the lower projection matrix and upper projection matrix can be used together with expert models 1 and 2 to participate in the calculation. Through the above formulas (3) and (4), expert models 1 and 2 are combined with the knowledge information of expert model 3 that is not selected. After processing the question text "Who is the author of ancient poem A", the answer text "The author of ancient poem A is A, who was born in ** year, and his representative work is..." can be obtained, and finally it can be fed back to the user.

[0070] In addition, in order to enable the hybrid expert model to have a strong prediction ability and to realize knowledge transfer, it is necessary to reversely broadcast and update the knowledge transfer unit during the model training phase. In this embodiment, the training of the hybrid expert model includes: Select sample question information from a sample set; input the sample question information into an initial hybrid expert model for processing to obtain predicted answer information; train the initial hybrid expert model based on the sample answer information corresponding to the sample question information and the predicted answer information until the hybrid expert model that meets the training stop condition is obtained; wherein the unit parameters corresponding to the knowledge transfer unit in the hybrid expert model are updated by reverse propagation.

[0071] Based on this, in the model training stage, sample question information can be first selected from the sample set; at this time, the sample question information can be input into the initial hybrid expert model, and on this basis, an expert model can be first selected for the sample question information in the initial hybrid expert model, and the expert models not selected in the initial hybrid expert model can be determined; then an embedding vector of the unselected expert model can be constructed, and the embedding vector can be processed through the knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; on this basis, the selected expert model can be updated based on the knowledge projection matrix, and finally the updated expert model can be used to process the sample question information to obtain the predicted answer information.

[0072] Further, the sample answer information corresponding to the sample question information is determined, and then the initial hybrid expert model can be trained based on the sample answer information and the predicted answer information corresponding to the sample question information, and then it can be detected whether the trained model meets the training stop condition. If not, a new sample can be selected from the sample set to continue training until a hybrid expert model that meets the training stop condition is obtained. During the model training process, the unit parameters corresponding to the knowledge transfer unit in the hybrid expert model are updated through reverse propagation.

[0073] That is to say, during the training process, the model selects the expert model to participate in the calculation through the routing unit, and also generates the embedding vector of the unselected expert model. After that, the knowledge transfer unit generates the projection matrix according to the embedding vector and performs forward propagation with the selected expert model. The model parameters of the knowledge transfer unit will be updated through back propagation, so as to achieve the purpose of optimizing the knowledge transfer unit.

[0074] The information processing method provided in this embodiment, in order to optimize the balance between the sparsity of expert model selection and knowledge utilization, can first input the question information into the hybrid expert model after obtaining the question information corresponding to the natural language processing task, at this time, first select the first expert model for the question information in the hybrid expert model, and determine the second expert model that is not selected in the hybrid expert model; on this basis, the embedding vector of the second expert model can be first constructed, and the embedding vector can be processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix; on this basis, the first expert model can be updated based on the knowledge projection matrix, so that the knowledge information of the unselected expert model can be supplemented to the selected expert model, and finally the updated first expert model can be used to process the question information to obtain the answer information. In the processing process, the sparsity of expert selection can be guaranteed by knowledge transfer, while the availability of expert model knowledge can be improved, thereby improving the performance of the model in complex tasks, and when transferring knowledge, only positive knowledge is selected for transfer, avoiding the transmission of negative knowledge, thereby reducing the prediction risk brought by negative knowledge.

[0075] See also Figure 3 , Figure 3 A flowchart of another information processing method provided according to an embodiment of the present specification is shown. The method is applied to a server and specifically includes the following steps.

[0076] Step S302, receiving question information submitted by the client, and inputting the question information into the hybrid expert model; Step S304, selecting a first expert model in the hybrid expert model according to the question information, and determining a second expert model that is not selected in the hybrid expert model; Step S306, constructing an embedding vector of the second expert model, and processing the embedding vector through the knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; Step S308, updating the first expert model based on the knowledge projection matrix, and processing the question information using the updated first expert model to obtain answer information; Step S310: Feedback the answer information to the client.

[0077] The above is an illustrative scheme of another information processing method of this embodiment. It should be noted that the technical scheme of this information processing method and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the information processing method can be referred to the description of the technical scheme of the above information processing method.

[0078] The following combination Figure 4 , taking the application of the information processing method provided in this specification in the intelligent question-answering scenario as an example, the information processing method is further described. Figure 4 A processing flow chart of an information processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0079] Step S402: Obtain question information corresponding to the natural language processing task, and input the question information into the hybrid expert model.

[0080] Step S404: Calculate the matching degree between the question information and each expert model through the routing unit in the hybrid expert model.

[0081] Step S406: compare the matching degree corresponding to each expert model with a preset matching degree threshold.

[0082] Step S408: selecting an expert model with a matching degree greater than a preset matching degree threshold as a first expert model according to the comparison result, and treating expert models other than the first expert model as unselected second expert models.

[0083] Step S410: Determine the embedding information and selection information corresponding to the second expert model.

[0084] Step S412: input the embedding information and the selection information into the multi-layer perceptron network in the hybrid expert model for processing to obtain a selection embedding vector.

[0085] Step S414, concatenating the layer embedding vector corresponding to the second expert model and the selected embedding vector to obtain an embedding vector.

[0086] Step S416: input the embedding vector into the knowledge transfer unit in the hybrid expert model for processing to obtain an upper projection matrix and a lower projection matrix.

[0087] Step S418: Use the upper projection matrix and the lower projection matrix as the knowledge projection matrix.

[0088] Step S420: updating the first expert model based on the knowledge projection matrix, and obtaining a third expert model according to the updating result.

[0089] Step S422: input the question information into the third expert model, and process the question information through the original processing unit and the knowledge sharing unit in the third expert respectively to obtain the first answer information and the second answer information.

[0090] Step S424, determining a first weight coefficient corresponding to the original processing unit and a second weight coefficient corresponding to the knowledge sharing unit.

[0091] Step S426: performing a weighted sum calculation on the first answer information and the first weight coefficient, and the second answer information and the second weight coefficient.

[0092] Step S428, generating answer information corresponding to the question information according to the weighted sum calculation result.

[0093] In summary, in order to optimize the balance between the sparsity of expert model selection and knowledge utilization, after obtaining the problem information of the corresponding natural language processing task, the problem information can be first input into the hybrid expert model. At this time, the first expert model is first selected for the problem information in the hybrid expert model, and the second expert model that is not selected in the hybrid expert model is determined; on this basis, the embedding vector of the second expert model can be first constructed, and the embedding vector is processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix; on this basis, the first expert model is updated based on the knowledge projection matrix, and the knowledge information of the unselected expert model can be supplemented to the selected expert model. Finally, the updated first expert model can be used to process the problem information to obtain the answer information. In the processing process, the knowledge transfer method can ensure the sparsity of expert selection while improving the availability of expert model knowledge, thereby improving the performance of the model in complex tasks. At the same time, when transferring knowledge, only positive knowledge is selected for transfer, avoiding the transfer of negative knowledge, thereby reducing the prediction risk brought by negative knowledge.

[0094] Corresponding to the above method embodiment, this specification also provides an information processing device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of an information processing device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises: An acquisition module 502 is configured to acquire question information corresponding to a natural language processing task and input the question information into a hybrid expert model; A selection module 504 is configured to select a first expert model in the hybrid expert model for the question information, and determine a second expert model that is not selected in the hybrid expert model; A construction module 506 is configured to construct an embedding vector of the second expert model, and process the embedding vector through the knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The updating module 508 is configured to update the first expert model based on the knowledge projection matrix, and use the updated first expert model to process the question information to obtain answer information.

[0095] In an optional embodiment, the selection module 504 is further configured to: The matching degree between the question information and each expert model is calculated by a routing unit in the hybrid expert model; the matching degree corresponding to each expert model is compared with a preset matching degree threshold; and an expert model having a matching degree greater than the preset matching degree threshold is selected as a first expert model according to the comparison result, and expert models other than the first expert model are selected as unselected second expert models.

[0096] In an optional embodiment, the construction module 506 is further configured to: Determine the embedding information and selection information corresponding to the second expert model; input the embedding information and the selection information into the multilayer perceptron network in the hybrid expert model for processing to obtain a selection embedding vector; concatenate the layer embedding vector corresponding to the second expert model and the selection embedding vector to obtain an embedding vector.

[0097] In an optional embodiment, the construction module 506 is further configured to: The embedding vector is input into the knowledge transfer unit in the hybrid expert model for processing to obtain an upper projection matrix and a lower projection matrix; and the upper projection matrix and the lower projection matrix are used as knowledge projection matrices.

[0098] In an optional embodiment, the updating module 508 is further configured to: The first expert model is updated based on the knowledge projection matrix, and a third expert model is obtained according to the update result; the question information is input into the third expert model, and the question information is processed by the original processing unit and the knowledge sharing unit in the third expert respectively to obtain first answer information and second answer information; answer information corresponding to the question information is generated according to the first answer information and the second answer information.

[0099] In an optional embodiment, the updating module 508 is further configured to: Determine a first weight coefficient corresponding to the original processing unit and a second weight coefficient corresponding to the knowledge sharing unit; perform weighted sum calculation on the first answer information and the first weight coefficient, and the second answer information and the second weight coefficient; and generate answer information corresponding to the question information based on the weighted sum calculation result.

[0100] In an optional embodiment, the training of the hybrid expert model includes: Select sample question information from a sample set; input the sample question information into an initial hybrid expert model for processing to obtain predicted answer information; train the initial hybrid expert model based on the sample answer information corresponding to the sample question information and the predicted answer information until the hybrid expert model that meets the training stop condition is obtained; wherein the unit parameters corresponding to the knowledge transfer unit in the hybrid expert model are updated by reverse propagation.

[0101] The information processing device provided in this embodiment can optimize the balance between the sparsity of expert model selection and knowledge utilization. After obtaining the problem information corresponding to the natural language processing task, the problem information can be first input into the hybrid expert model. At this time, the first expert model is first selected for the problem information in the hybrid expert model, and the second expert model not selected in the hybrid expert model is determined; on this basis, the embedding vector of the second expert model can be first constructed, and the embedding vector is processed by the knowledge transfer unit in the hybrid expert model to obtain the knowledge projection matrix; on this basis, the first expert model is updated based on the knowledge projection matrix, and the knowledge information of the unselected expert model can be supplemented to the selected expert model. Finally, the updated first expert model can be used to process the problem information to obtain the answer information. In the processing process, the sparsity of expert selection can be guaranteed by knowledge transfer, while the availability of expert model knowledge can be improved, thereby improving the performance of the model in complex tasks. At the same time, when transferring knowledge, only positive knowledge is selected for transfer, avoiding the transfer of negative knowledge, thereby reducing the prediction risk brought by negative knowledge.

[0102] The above is a schematic scheme of an information processing device of this embodiment. It should be noted that the technical scheme of the information processing device and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the information processing device can be referred to the description of the technical scheme of the above information processing method.

[0103] Corresponding to the above method embodiment, this specification also provides another information processing device embodiment, Figure 6 FIG. 2 shows a schematic diagram of the structure of another information processing device provided by an embodiment of the present specification. Figure 6 As shown, the device is applied to the server, including: The information receiving module 602 is configured to receive question information submitted by the client and input the question information into the hybrid expert model; A model selection module 604 is configured to select a first expert model in the mixed expert model for the question information, and determine a second expert model that is not selected in the mixed expert model; A vector construction module 606 is configured to construct an embedding vector of the second expert model, and process the embedding vector through the knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; An updating model module 608 is configured to update the first expert model based on the knowledge projection matrix, and use the updated first expert model to process the question information to obtain answer information; The information sending module 610 is configured to feed back the answer information to the client.

[0104] The above is a schematic scheme of another information processing device of this embodiment. It should be noted that the technical scheme of the information processing device and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the information processing device can be referred to the description of the technical scheme of the above information processing method.

[0105] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0106] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0107] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0108] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0109] The processor 720 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned information processing method when executed by the processor.

[0110] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above information processing method.

[0111] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned information processing method when executed by a processor.

[0112] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above information processing method.

[0113] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned information processing method when executed by a processor.

[0114] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above information processing method.

[0115] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0117] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. An information processing method, characterized in that: include: Obtaining question information corresponding to a natural language processing task, and inputting the question information into a hybrid expert model; Selecting a first expert model in the hybrid expert model for the problem information, and determining a second expert model that is not selected in the hybrid expert model; Constructing an embedding vector of the second expert model, and processing the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The first expert model is updated based on the knowledge projection matrix, and the question information is processed using the updated first expert model to obtain answer information.

2. The information processing method according to claim 1, characterized in that: The selecting a first expert model in the mixed expert model for the problem information, and determining a second expert model not selected in the mixed expert model, comprises: Calculating the matching degree between the question information and each expert model through the routing unit in the hybrid expert model; Compare the matching degree corresponding to each expert model with the preset matching degree threshold; According to the comparison result, an expert model with a matching degree greater than the preset matching degree threshold is selected as the first expert model, and expert models other than the first expert model are used as unselected second expert models.

3. The information processing method according to claim 1, characterized in that: The constructing the embedding vector of the second expert model includes: Determining embedding information and selection information corresponding to the second expert model; Inputting the embedding information and the selection information into the multi-layer perceptron network in the hybrid expert model for processing to obtain a selection embedding vector; The layer embedding vector corresponding to the second expert model and the selected embedding vector are concatenated to obtain an embedding vector.

4. The information processing method according to claim 1, characterized in that: The step of processing the embedding vector by the knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix includes: Inputting the embedding vector into the knowledge transfer unit in the hybrid expert model for processing to obtain an upper projection matrix and a lower projection matrix; The upper projection matrix and the lower projection matrix are used as knowledge projection matrices.

5. The information processing method according to claim 1, characterized in that: The updating of the first expert model based on the knowledge projection matrix and processing the question information using the updated first expert model to obtain answer information includes: updating the first expert model based on the knowledge projection matrix, and obtaining a third expert model according to the updating result; Inputting the question information into the third expert model, and processing the question information through the original processing unit and the knowledge sharing unit in the third expert respectively to obtain first answer information and second answer information; Answer information corresponding to the question information is generated according to the first answer information and the second answer information.

6. The information processing method according to claim 5, characterized in that: The generating the answer information corresponding to the question information according to the first answer information and the second answer information includes: Determine a first weight coefficient corresponding to the original processing unit and a second weight coefficient corresponding to the knowledge sharing unit; Performing a weighted sum calculation on the first answer information and the first weight coefficient, and on the second answer information and the second weight coefficient; The answer information corresponding to the question information is generated according to the weighted sum calculation result.

7. The information processing method according to any one of claims 1 to 6, characterized in that: The training of the hybrid expert model includes: Select sample question information in the sample set; Inputting the sample question information into the initial hybrid expert model for processing to obtain predicted answer information; Training the initial hybrid expert model based on the sample answer information corresponding to the sample question information and the predicted answer information until the hybrid expert model that meets the training stop condition is obtained; Wherein, the unit parameters corresponding to the knowledge transfer unit in the hybrid expert model are updated through reverse propagation.

8. An information processing method, characterized in that: Applied to the server, including: Receiving question information submitted by a client, and inputting the question information into the hybrid expert model; Selecting a first expert model in the hybrid expert model for the problem information, and determining a second expert model that is not selected in the hybrid expert model; Constructing an embedding vector of the second expert model, and processing the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The first expert model is updated based on the knowledge projection matrix, and the question information is processed using the updated first expert model to obtain answer information; The answer information is fed back to the client.

9. An information processing device, characterized in that: include: An acquisition module is configured to acquire question information corresponding to a natural language processing task and input the question information into a hybrid expert model; A selection module is configured to select a first expert model in the mixed expert model for the question information, and determine a second expert model that is not selected in the mixed expert model; A construction module is configured to construct an embedding vector of the second expert model, and process the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; The updating module is configured to update the first expert model based on the knowledge projection matrix, and use the updated first expert model to process the question information to obtain answer information.

10. An information processing device, characterized in that: Applied to the server, including: An information receiving module is configured to receive question information submitted by a client and input the question information into the hybrid expert model; A model selection module is configured to select a first expert model in the mixed expert model for the question information, and determine a second expert model that is not selected in the mixed expert model; A vector construction module is configured to construct an embedding vector of the second expert model, and process the embedding vector through a knowledge transfer unit in the hybrid expert model to obtain a knowledge projection matrix; an updating model module, configured to update the first expert model based on the knowledge projection matrix, and process the question information using the updated first expert model to obtain answer information; The information sending module is configured to feed back the answer information to the client.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images