Model optimization method and device and information processing method and device
By calculating the target entropy value of word segmentation units in the mixed expert model and dividing it into global and local partial word units, the problem of router selection uncertainty in the model is solved, the model robustness and prediction accuracy are improved, and the expert sub-model is used in application stage efficiency.
Patent Information
- Application Number
- CN202411191438.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-19
AI Technical Summary
The existing hybrid expert model has uncertainty in the expert selection process, resulting in high computational costs and model performance impacts, especially when the router score is close, affecting prediction accuracy.
By calculating the target entropy value of each word segmentation unit, it is divided into global word segmentation unit and local word segmentation unit, and broadcasting the global word segmentation unit to all expert submodels for processing, and broadcasting the local word segmentation unit to match expert submodel for processing Processing, implement model optimization and reduce router selection uncertainty.
Improve model robustness and prediction accuracy, ensuring that the expert sub-model has better robustness during the application stage and can give accurate prediction results when encountering uncertain tokens.
Smart Images

Figure CN118981528A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of machine learning technology, and in particular to model optimization methods and devices, and information processing methods and devices. Background Art
[0002] With the rapid growth of computing power and data volume, large language models (LLMs) have been widely developed and applied. These models have shown excellent performance in natural language processing, computer vision, recommendation systems and other fields. Its core technologies include Transformer architecture, attention mechanism and large-scale pre-training. However, as the scale of model parameters continues to expand, the computational cost and energy consumption of the model are becoming increasingly prominent. In order to solve this problem, a mixture of experts (MoE) model is proposed. The MoE model achieves computational sparsity by dynamically selecting and activating some experts. Specifically, the MoE model consists of a router and multiple experts, and only a small number of experts are activated for each input, which significantly reduces the computational cost. In the prior art, although the MoE model performs well in reducing computational costs, there is an uncertainty problem in the expert selection process. These uncertainties are mainly manifested in that some tokens have close scores when selecting experts, resulting in increased randomness in the selection, thereby affecting the model performance. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention
[0003] In view of this, an embodiment of this specification provides a model optimization method. One or more embodiments of this specification also relate to an information processing method, a model optimization device, an information processing device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.
[0004] According to a first aspect of an embodiment of this specification, a model optimization method is provided, including:
[0005] Inputting a plurality of word segmentation units into a hybrid expert model, and calculating a target entropy value corresponding to each word segmentation unit through a router in the hybrid expert model;
[0006] Dividing the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value;
[0007] Broadcasting the global word segmentation unit to all expert sub-models in the hybrid expert model for processing, and broadcasting the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing;
[0008] The hybrid expert model is adjusted according to the processing results until a target hybrid expert model that meets the optimization stop condition is obtained.
[0009] Optionally, the calculation of a target entropy value corresponding to any one of the multiple word segmentation units includes:
[0010] Calculating, by means of a router in the hybrid expert model, a selection score of each expert sub-model in the hybrid expert model selected by a word segmentation unit;
[0011] An initial entropy value of the selection score corresponding to each expert sub-model is determined, and a target entropy value corresponding to the word segmentation unit is obtained by normalizing the initial entropy value.
[0012] Optionally, dividing the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value includes:
[0013] Compare the target entropy value corresponding to each word segmentation unit with a preset entropy value threshold, and select the word segmentation unit greater than the preset entropy value threshold as the global word segmentation unit according to the comparison result, and select the word segmentation unit less than or equal to the preset entropy value threshold as the local word segmentation unit; or,
[0014] The multiple word segmentation units are sorted according to the target entropy value corresponding to each word segmentation unit to obtain a word segmentation unit sequence, a global word segmentation unit is selected from the word segmentation unit sequence according to preset proportion information, and the remaining word segmentation units except the global word segmentation unit are used as local word segmentation units.
[0015] Optionally, broadcasting the local word segmentation unit to an expert sub-model matching the local word segmentation unit in the hybrid expert model for processing includes:
[0016] Calculate the target selection score of each expert sub-model in the hybrid expert model for selecting the local word segment unit;
[0017] A local expert sub-model matching the local word segmentation unit is determined in the hybrid expert model according to the target selection score, and the local word segmentation unit is broadcast to the local expert sub-model for processing.
[0018] Optionally, adjusting the parameters of the hybrid expert model according to the processing result until a target hybrid expert model that meets the optimization stop condition is obtained includes:
[0019] Adjusting the parameters of the hybrid expert model according to the processing result, and detecting whether the hybrid expert model after the adjustment satisfies the optimization stop condition;
[0020] If not, select multiple candidate word segmentation units as the multiple word segmentation units, and perform the step of inputting the multiple word segmentation units into the hybrid expert model, and calculating the target entropy value corresponding to each word segmentation unit through the router in the hybrid expert model;
[0021] If so, the adjusted hybrid expert model is used as the target hybrid expert model.
[0022] Optionally, before the step of inputting the plurality of word segmentation units into the hybrid expert model is executed, the step further includes:
[0023] Acquire original sample data associated with the hybrid expert model, preprocess the original sample data, and obtain a plurality of initial word segmentation units;
[0024] The multiple initial word segmentation units are respectively embedded to obtain multiple word segmentation units.
[0025] According to a second aspect of the embodiments of this specification, there is provided an information processing method, including:
[0026] Obtaining question information submitted by a client, constructing a plurality of question word segmentation units according to the question information and inputting the units into a target hybrid expert model, wherein the target hybrid expert model is obtained by the above method;
[0027] Selecting a target expert sub-model matching each question word segmentation unit in the target hybrid expert model to process the multiple question word segmentation units;
[0028] The prediction results of the target expert sub-model matched to each question word segmentation unit are merged to obtain answer information, and the answer information is fed back to the client.
[0029] According to a third aspect of the embodiments of this specification, a model optimization device is provided, including:
[0030] A calculation module is configured to input a plurality of word segmentation units into a hybrid expert model, and calculate a target entropy value corresponding to each word segmentation unit through a router in the hybrid expert model;
[0031] A division module is configured to divide the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value;
[0032] A processing module is configured to broadcast the global word segmentation unit to all expert sub-models in the hybrid expert model for processing, and broadcast the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing;
[0033] The parameter adjustment module is configured to adjust the parameters of the hybrid expert model according to the processing results until a target hybrid expert model that meets the optimization stop condition is obtained.
[0034] According to a fourth aspect of the embodiments of this specification, there is provided an information processing device, including:
[0035] An information acquisition module is configured to acquire question information submitted by a client, construct a plurality of question word segmentation units according to the question information, and input the units into a target hybrid expert model, wherein the target hybrid expert model is obtained by the above method;
[0036] A word segmentation processing module is configured to select a target expert sub-model matching each question word segmentation unit in the target mixed expert model to process the multiple question word segmentation units;
[0037] The feedback information module is configured to merge the prediction results of the target expert sub-model matched with each question segmentation unit, obtain answer information, and feed back the answer information to the client.
[0038] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:
[0039] Memory and processor;
[0040] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned model optimization method or information processing method are implemented.
[0041] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned model optimization method or information processing method are implemented.
[0042] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned model optimization method or information processing method when executed by a processor.
[0043] The model optimization method provided in this embodiment can reduce the uncertainty of router selection, improve the robustness of the model, and thus ensure the prediction accuracy. In the model optimization stage, after inputting multiple word segmentation units into the hybrid expert model, the target entropy value corresponding to each word segmentation unit can be calculated using the router in the hybrid expert model; on this basis, the multiple word segmentation units can be divided into global word segmentation units and local word segmentation units according to the target entropy value; then the global word segmentation unit can be broadcast to all expert sub-models in the hybrid expert model for processing, and the local word segmentation unit can be broadcast to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; in the model optimization stage, the uncertain word segmentation unit can be broadcast to all expert sub-models for learning, while the determined word segmentation unit is still used for matching the learning of the expert sub-model, so as to adjust the parameters of the hybrid expert model according to the processing results to obtain the target hybrid expert model that meets the optimization stop condition. In the fine-tuning process of the hybrid expert model, all expert sub-models are allowed to learn the relevant knowledge of the uncertain word segmentation unit to ensure that the expert sub-model has better robustness in the application stage. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is a schematic diagram of a model optimization method provided by an embodiment of this specification;
[0045] Figure 2 is a flow chart of a model optimization method provided by an embodiment of this specification;
[0046] Figure 3 is a flow chart of an information processing method provided by an embodiment of this specification;
[0047] Figure 4 is a processing flow chart of an information processing method provided by an embodiment of this specification;
[0048] Figure 5 is a structural schematic diagram of a model optimization device provided by an embodiment of this specification;
[0049] Figure 6 is a structural diagram of an information processing device provided by an embodiment of this specification;
[0050] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0051] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0052] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0053] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0054] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0055] First, the terms involved in one or more embodiments of this specification are explained.
[0056] Mixture of Experts (MoE) model: The full name is Mixture of Experts. It is an integrated learning method that combines multiple specialized sub-models (i.e. "experts") to form an overall model, where each "expert" contributes in his or her area of expertise.
[0057] LLMs: (Large Language Models) refers to large language models, which are an important type of model in the current field of natural language processing (NLP). These models are usually based on deep learning technology, especially the Transformer architecture, such as extensions or variants of models such as BERT and GPT. LLMs can process and understand complex natural language texts, and perform generative tasks (such as text generation and dialogue generation) and analytical tasks (such as sentiment analysis and information extraction).
[0058] In this specification, a model optimization method is provided. One or more embodiments of this specification also relate to an information processing method, a model optimization device, an information processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.
[0059] In practical applications, the standard MoE model selects several experts with the highest scores based on the router scores for calculation. However, in practical applications, it is common for router scores to be close, especially when processing complex data. The scores of these uncertain tokens are close when selecting experts, resulting in the inability to determine the best expert, which may result in the selection of the wrong expert and affect the model performance. Therefore, an effective solution is urgently needed to solve the above problem.
[0060] See also Figure 1 As shown in the schematic diagram, the model optimization method provided by this embodiment can reduce the uncertainty of router selection, improve the robustness of the model, and thus ensure the prediction accuracy. In the model optimization stage, after inputting multiple word segmentation units into the hybrid expert model, the target entropy value corresponding to each word segmentation unit can be calculated using the router in the hybrid expert model; on this basis, the multiple word segmentation units can be divided into global word segmentation units and local word segmentation units according to the target entropy value; then the global word segmentation unit can be broadcast to all expert sub-models in the hybrid expert model for processing, and the local word segmentation unit can be broadcast to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; in the model optimization stage, the uncertain word segmentation unit can be broadcast to all expert sub-models for learning, while the determined word segmentation unit is still used for matching the learning of the expert sub-model, so as to adjust the parameters of the hybrid expert model according to the processing results to obtain the target hybrid expert model that meets the optimization stop condition. In the fine-tuning process of the hybrid expert model, all expert sub-models are allowed to learn the relevant knowledge of the uncertain word segmentation unit to ensure that the expert sub-model has better robustness in the application stage.
[0061] See also Figure 2 , Figure 2A flow chart of a model optimization method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0062] Step S202: input a plurality of word segmentation units into a hybrid expert model, and calculate a target entropy value corresponding to each word segmentation unit through a router in the hybrid expert model.
[0063] The model optimization method provided in this embodiment is applied to the fine-tuning task of the hybrid expert model (MoE model). After the hybrid expert model is optimized, if the router calculation scores are close, even if the problem of being unable to determine the best expert sub-model occurs, it can still be guaranteed that the selected expert sub-model gives a more accurate prediction result, thereby effectively improving the model robustness. Among them, the hybrid expert model can be a MoE model applied to any field, such as the field of natural language processing (hybrid expert model is used for text classification, text generation, translation, etc.), the field of computer vision (hybrid expert model is used for image classification, target detection, image generation, etc.), recommendation system (hybrid expert model is used for personalized recommendation), etc., and this embodiment does not make any limitation here. This embodiment uses the hybrid expert model for sentiment analysis of input text as an example to illustrate the model optimization method. The descriptions in other scenarios can refer to the same or corresponding descriptions in this embodiment, and this embodiment will not be repeated in detail here.
[0064] Specifically, the multiple word segmentation units specifically refer to multiple tokens obtained after word segmentation processing of the sample in the fine-tuning stage of the hybrid expert model, wherein the token can be a character unit and / or a word unit in the sample. Correspondingly, the hybrid expert model specifically refers to a MoE model including a router and multiple expert sub-models, wherein the router is used to calculate the expert sub-model matched by each word segmentation unit, so that the word segmentation unit can be submitted to the matching expert sub-model for processing in the application stage. Correspondingly, the target entropy value specifically refers to the result obtained after using the router to calculate the entropy value of each word segmentation unit. The target entropy value is used to measure the certainty of the word segmentation unit selecting the expert sub-model. The higher the entropy value, the worse the certainty, that is: after the word segmentation unit selects the expert sub-model, the possibility that the expert sub-model is the best matching word segmentation unit in the mixed expert model is uncertain. It may be the best or may not be. Therefore, it is necessary to broadcast this type of word segmentation unit to all expert sub-models for relevant knowledge learning through the model optimization method provided by this embodiment, so as to eliminate the problem of decreased prediction accuracy caused by uncertainty, and ensure that in the application stage, if an uncertain token is encountered, after assigning an expert sub-model to it, it can be guaranteed that the expert sub-model can still give accurate prediction results for the uncertain token.
[0065] Based on this, in order to reduce the uncertainty of router selection, improve the robustness of the model, and thus ensure the accuracy of prediction, in the model optimization stage, after inputting multiple word segmentation units into the hybrid expert model, the target entropy value corresponding to each word segmentation unit can be calculated using the router in the hybrid expert model; the certainty of selecting the expert sub-model for each word segmentation unit can be measured by the target entropy value, which facilitates the subsequent division of word segmentation unit types, so as to fine-tune the hybrid expert model to a target hybrid expert model that meets usage requirements.
[0066] Furthermore, in order to ensure that the hybrid expert model meets the requirements after fine-tuning, the word segmentation units input into the hybrid expert model for learning relevant knowledge can be obtained by processing the original sample data. In this embodiment, the specific implementation method is as follows:
[0067] Original sample data associated with the hybrid expert model is obtained, and the original sample data is preprocessed to obtain a plurality of initial word segmentation units; and the plurality of initial word segmentation units are respectively embedded to obtain a plurality of word segmentation units.
[0068] Specifically, the original sample data specifically refers to the text samples used to fine-tune the hybrid expert model. Correspondingly, the preprocessing specifically refers to the process of performing word segmentation, encoding, etc. on the original sample data. Correspondingly, the embedding processing specifically refers to the process of converting the initial word segmentation unit into a vector expression.
[0069] Based on this, after obtaining the original sample data of the associated hybrid expert model, the original sample data can be preprocessed first to generate multiple initial word segmentation units according to the preprocessing results; on this basis, in order to be able to process after inputting into the hybrid expert model, the multiple initial word segmentation units can be embedded separately to obtain multiple word segmentation units, so that the hybrid expert model can be fine-tuned in combination with multiple word segmentation units later.
[0070] In summary, by preprocessing the original sample data, it can be ensured that the sample data is converted into multiple word segmentation units that meet the input requirements of the hybrid expert model, thereby achieving the purpose of model fine-tuning.
[0071] Furthermore, when calculating the target entropy value corresponding to the word segmentation unit, in order to accurately reflect the certainty of the word segmentation unit selecting each expert sub-model through the target entropy value, it can be achieved in the following way. In this embodiment, the calculation of the target entropy value corresponding to any one of the multiple word segmentation units includes:
[0072] Through the router in the hybrid expert model, the selection score of each expert sub-model in the hybrid expert model selected by the word segmentation unit is calculated; the initial entropy value of the selection score corresponding to each expert sub-model is determined, and the target entropy value corresponding to the word segmentation unit is obtained by normalizing the initial entropy value.
[0073] Specifically, the selection score refers to the selection possibility score of each expert sub-model selected by the word segmentation unit calculated by the router. The higher the score, the higher the probability of the expert sub-model being selected. Correspondingly, the initial entropy value refers to the entropy value calculated based on the selection score, which is used to indicate the certainty of the word segmentation unit selecting each expert sub-model. By normalizing it, the target entropy value reflecting the certainty of the word segmentation unit selecting the expert sub-model can be obtained.
[0074] Based on this, for any word segmentation unit, the selection score of each expert sub-model in the hybrid expert model selected by the word segmentation unit can be calculated through the router in the hybrid expert model; on this basis, the initial entropy value of the selection score corresponding to each expert sub-model can be determined first, and then the initial entropy value can be normalized to merge all the initial entropy values, so as to obtain the target entropy value that reflects the word segmentation unit selection expert sub-model, so as to facilitate the subsequent division of multiple word segmentation units.
[0075] In practical applications, after the router calculates the selection score corresponding to the word segmentation unit, it can calculate the initial entropy value by the following formula (1):
[0076] H(x)=-∑ i gilog(gi) (1)
[0077] Among them, x represents the word segmentation unit, gi represents the selection score of the word segmentation unit selecting the i-th expert sub-model, and H(x) represents the initial entropy value corresponding to the selection score.
[0078] After obtaining the initial entropy value, the initial entropy values corresponding to all expert sub-models can be normalized to obtain the normalized entropy value, i.e., the target entropy value, which can be calculated by the following formula (2):
[0079] H norm (x) = [H (x)] / [log (N)] (2)
[0080] Among them, H norm (x) represents the target entropy value, and N represents the total number of expert sub-models.
[0081] For example, in order to make the MoE model of text sentiment analysis have a stronger prediction effect, the MoE model will be fine-tuned; among them, the MoE model contains 3 expert sub-models, expert sub-model a is good at processing texts containing positive words and outputs positive sentiment prediction results; expert sub-model b is good at processing texts containing negative words and outputs negative sentiment prediction results; expert sub-model c is good at processing neutral text and outputs neutral prediction results. On this basis, N tokens can be obtained and input into the MoE model. The router in the MoE model calculates the score of each token selecting 3 expert sub-models, and determines that the score of token1 selecting expert sub-model a is Sa1, the score of token1 selecting expert sub-model b is Sb1, and the score of token1 selecting expert sub-model c is Sc1; the score of token2 selecting expert sub-model a is Sa2... The score of tokenN selecting expert sub-model c is ScN. Afterwards, the above formulas (1) and (2) can be combined to calculate the normalized entropy value H1 of token1 relative to all expert sub-models...the normalized entropy value HN of tokenN relative to all expert sub-models, so as to subsequently combine the normalized entropy value corresponding to each token to complete the distinction of different types of tokens and complete the fine-tuning of the MoE model.
[0082] In summary, by calculating the entropy value to reflect the certainty of selecting the expert sub-model for each word segmentation unit, it is possible to filter out uncertain tokens, which makes it easier to broadcast them to all expert sub-models for relevant knowledge learning in the fine-tuning stage, making the model more robust.
[0083] Step S204: dividing the multiple word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value.
[0084] Specifically, after obtaining the target entropy value corresponding to each word segmentation unit as mentioned above, considering that the target entropy value is used to reflect the certainty of selecting the expert sub-model for each word segmentation unit, the word segmentation unit types can be divided according to the target entropy value corresponding to each word segmentation unit, so as to select the word segmentation unit with high uncertainty as the global word segmentation unit, and select the word segmentation unit with low uncertainty as the local word segmentation unit, so as to facilitate the subsequent fine-tuning of the hybrid expert model.
[0085] Among them, the global word segmentation unit specifically refers to the word segmentation unit with stronger uncertainty among multiple word segmentation units, that is, this type of word segmentation unit has stronger uncertainty when selecting its matching expert sub-model through the router. Correspondingly, the local word segmentation unit specifically refers to the word segmentation unit that can accurately select its matching expert sub-model among multiple word segmentation units, and this type of word segmentation unit has weaker uncertainty.
[0086] Furthermore, after obtaining the target entropy value corresponding to each word segmentation unit, when dividing the local word segmentation unit and the global word segmentation unit, the certainty and uncertainty of the word segmentation unit are actually divided according to the entropy value. In specific implementation, it can be achieved in the following two ways:
[0087] (1) Compare the target entropy value corresponding to each word segmentation unit with a preset entropy value threshold, and select the word segmentation unit whose value is greater than the preset entropy value threshold as the global word segmentation unit according to the comparison result, and select the word segmentation unit whose value is less than or equal to the preset entropy value threshold as the local word segmentation unit.
[0088] (2) Sort the multiple word segmentation units according to the target entropy value corresponding to each word segmentation unit to obtain a word segmentation unit sequence, select a global word segmentation unit from the word segmentation unit sequence according to preset proportion information, and use the remaining word segmentation units except the global word segmentation unit as local word segmentation units.
[0089] Specifically, the preset entropy value threshold refers to the threshold for screening uncertain segmentation units and determining segmentation units, which can be set according to actual needs, and this embodiment does not make any restrictions here. Correspondingly, the segmentation unit sequence specifically refers to the sequence obtained by sorting multiple segmentation units according to the target entropy value, and the preset proportion information specifically refers to the proportion value of the number of segmentation units, which can be a specific number or a percentage value, and this embodiment does not make any restrictions here.
[0090] Based on this, after obtaining the target entropy value that can reflect the certainty of the word segmentation unit selection expert sub-model, on the one hand, the target entropy value corresponding to each word segmentation unit can be compared with the preset entropy value threshold, and the word segmentation unit greater than the preset entropy value threshold can be selected as the global word segmentation unit according to the comparison result, and the word segmentation unit less than or equal to the preset entropy value threshold can be selected as the local word segmentation unit, so that the global word segmentation unit and the local word segmentation unit can be combined to complete the fine-tuning of the hybrid expert model.
[0091] On the other hand, multiple word segmentation units can be sorted according to the target entropy value corresponding to each word segmentation unit to obtain a word segmentation unit sequence; on this basis, the global word segmentation unit can be selected in the word segmentation unit sequence according to the preset proportion information, and in the selection process, it can be completed according to the arrangement order of the word segmentation units in the sequence, and finally the remaining word segmentation units except the global word segmentation unit can be used as local word segmentation units.
[0092] In specific implementation, when sorting multiple word segmentation units, they can be sorted in order from high to low according to the target entropy value to form a word segmentation unit sequence. On this basis, the global word segmentation unit is selected, that is, the word segmentation unit corresponding to the preset proportion information is selected in order from high to low as the global word segmentation unit.
[0093] In practical applications, when dividing the word segmentation units according to the target entropy value, the processing adapted to the current scene can be selected according to actual needs, and this embodiment does not make any limitation here.
[0094] Using the above example, after obtaining the normalized entropy values corresponding to N tokens, we can further sort the N tokens from high to low according to the normalized entropy values, and then select the top 5% of the entropy values as uncertain tokens, including {token1, token2, token4...}, and then take the other tokens of the N tokens except the uncertain tokens as certain tokens, including {token3, token5, token6...}. Subsequently, we can combine the uncertain tokens and certain tokens to complete the fine-tuning of the MoE model.
[0095] In summary, by choosing different ways to complete the division of global word segmentation units and local word segmentation units, it can be ensured that in the subsequent model fine-tuning stage, the model can choose different strategies for different types of word segmentation units to complete the optimization, thereby making the model have stronger prediction capabilities.
[0096] Step S206, broadcasting the global word segmentation unit to all expert sub-models in the hybrid expert model for processing, and broadcasting the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing.
[0097] Specifically, after the global word segmentation unit and the local word segmentation unit are determined as described above, further, in order to solve the problem of inaccurate prediction results caused by the poor selection of the expert sub-model by the router for the word segmentation unit, in the model fine-tuning stage, the global word segmentation unit can be broadcast to all the expert sub-models in the hybrid expert model for learning related knowledge, so that all the expert sub-models in the hybrid expert model can learn the relevant knowledge of the global word segmentation unit, thereby ensuring that in the application stage, even when encountering a word segmentation unit that cannot accurately match the expert sub-model, the selected expert sub-model can give an accurate prediction result. At the same time, the local word segmentation unit can be broadcast to the expert sub-model that matches the local word segmentation unit in the hybrid expert model for processing, so as to achieve normal fine-tuning of the expert sub-model that matches the word segmentation unit.
[0098] The expert sub-model matching the local word segment unit specifically refers to an expert sub-model that can accurately determine the expert sub-model for processing the local word segment unit after calculating the expert sub-model selection score for the local word segment unit through the router.
[0099] Furthermore, when broadcasting the local word segment unit to its matching expert sub-model, it can be implemented by calculating the selection score. In this embodiment, the specific implementation method is as follows:
[0100] Calculate the target selection score of each expert sub-model in the hybrid expert model for selecting the local word segmentation unit; determine the local expert sub-model matching the local word segmentation unit in the hybrid expert model according to the target selection score, and broadcast the local word segmentation unit to the local expert sub-model for processing.
[0101] Specifically, the target selection score specifically refers to the score of each expert sub-model selected by the local word unit using the router in the hybrid expert model. The higher the score, the higher the probability of being selected. Correspondingly, the local expert sub-model specifically refers to the expert sub-model matching the local word unit, which is used to process the local word unit.
[0102] Based on this, when selecting an expert sub-model for a local word segmentation unit, the standard MoE method can be used. That is, the target selection score of each expert sub-model in the hybrid expert model for selecting a local word segmentation unit can be calculated first; then, the local expert sub-model that matches the local word segmentation unit can be determined in the hybrid expert model according to the target selection score, and the local word segmentation unit can be broadcasted to the local expert sub-model for processing.
[0103] Using the above example, after obtaining uncertain tokens including {token1, token2, token4...} and determined tokens including {token3, token5, token6...}, the uncertain tokens can be sent to all expert sub-models a, b, and c in the MoE model for learning using a broadcast strategy, while the determined tokens can be selected using the standard MoE method to select the expert sub-models that match each token for learning, and the selection can be completed according to the scores calculated above. This allows the learning results of all expert sub-models to be integrated later and the model parameters to be updated.
[0104] In summary, by broadcasting the global word segmentation unit to all expert sub-models and broadcasting the local word segmentation unit to the matching expert sub-model for processing, all expert sub-models can learn the relevant knowledge of uncertainty, and thus the hybrid expert model has more accurate prediction capabilities.
[0105] Step S208, adjusting the parameters of the hybrid expert model according to the processing result until a target hybrid expert model that meets the optimization stop condition is obtained.
[0106] Specifically, based on the above-mentioned use of all expert sub-models to process the global word segmentation unit, and the use of the expert sub-model that matches the local word segmentation unit to process the local word segmentation unit function, the parameters of the mixed expert model can be adjusted according to the processing results to obtain the target mixed expert model that meets the optimization stop condition.
[0107] Furthermore, when adjusting the parameters of the hybrid expert model, the target hybrid expert model that meets the requirements can be determined by comparing the optimization stop conditions. In this embodiment, the specific implementation method is as follows:
[0108] The hybrid expert model is adjusted according to the processing result, and it is detected whether the hybrid expert model after the adjustment satisfies the optimization stop condition; if not, multiple candidate word segmentation units are selected as the multiple word segmentation units, and the multiple word segmentation units are input into the hybrid expert model, and the target entropy value corresponding to each word segmentation unit is calculated through the router in the hybrid expert model; if so, the hybrid expert model after the adjustment is used as the target hybrid expert model.
[0109] Based on this, when adjusting the parameters of the hybrid expert model, it is possible to detect whether the hybrid expert model meets the optimization stop condition at each parameter adjustment stage; if not, it means that the model still needs to continue training, so multiple candidate word segmentation units can be selected as multiple word segmentation units, and the multiple word segmentation units can be input into the hybrid expert model, and the target entropy value corresponding to each word segmentation unit can be calculated through the router in the hybrid expert model; until a certain parameter adjustment stage, when the model after the parameter adjustment meets the optimization stop condition, the hybrid expert model after the parameter adjustment can be used as the target hybrid expert model to be deployed in business scenarios.
[0110] Using the above example, after all expert sub-models process the uncertain token and the expert sub-model that determines the token is matched, the learning results of all expert sub-models a, b and c can be merged to update the parameters of the MoE model. When the parameter-updated MoE model meets the business deployment requirements, you can stop fine-tuning and deploy it in the business scenario. After receiving the sentiment text to be analyzed submitted by the client, you can input it into the fine-tuned MoE model for processing, and then determine the sentiment information corresponding to the sentiment text to be analyzed, so as to facilitate downstream business use.
[0111] The model optimization method provided in this embodiment can reduce the uncertainty of router selection, improve the robustness of the model, and thus ensure the prediction accuracy. In the model optimization stage, after inputting multiple word segmentation units into the hybrid expert model, the target entropy value corresponding to each word segmentation unit can be calculated using the router in the hybrid expert model; on this basis, the multiple word segmentation units can be divided into global word segmentation units and local word segmentation units according to the target entropy value; then the global word segmentation unit can be broadcast to all expert sub-models in the hybrid expert model for processing, and the local word segmentation unit can be broadcast to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; in the model optimization stage, the uncertain word segmentation unit can be broadcast to all expert sub-models for learning, while the determined word segmentation unit is still used for matching the learning of the expert sub-model, so as to adjust the parameters of the hybrid expert model according to the processing results to obtain the target hybrid expert model that meets the optimization stop condition. In the fine-tuning process of the hybrid expert model, all expert sub-models are allowed to learn the relevant knowledge of the uncertain word segmentation unit to ensure that the expert sub-model has better robustness in the application stage.
[0112] See also Figure 3 , Figure 3 A flowchart of an information processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0113] Step S302, obtaining question information submitted by the client, constructing multiple question word segmentation units according to the question information and inputting them into a target hybrid expert model, wherein the target hybrid expert model is obtained by the above method;
[0114] Step S304, selecting a target expert sub-model matching each question word segmentation unit in the target hybrid expert model to process the multiple question word segmentation units;
[0115] Step S306: merge the prediction results of the target expert sub-model matched to each question word segmentation unit to obtain answer information, and feed back the answer information to the client.
[0116] The information processing method provided in this embodiment and the training of the target mixed expert model can all refer to the same or corresponding description content in the above embodiments, and this embodiment will not be described in detail here.
[0117] The following combination Figure 4 , taking the application of the information processing method provided in this specification in the text sentiment analysis scenario as an example, the information processing method is further explained. Figure 4 A processing flow chart of an information processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.
[0118] Step S402: input multiple word segmentation units into the hybrid expert model, and calculate the target entropy value corresponding to each word segmentation unit through the router in the hybrid expert model.
[0119] Among them, the calculation of the target entropy value corresponding to any one of the multiple word segmentation units includes: calculating the selection score of each expert sub-model in the hybrid expert model selected by the word segmentation unit through the router in the hybrid expert model; determining the initial entropy value of the selection score corresponding to each expert sub-model, and obtaining the target entropy value corresponding to the word segmentation unit by normalizing the initial entropy value.
[0120] Step S404: compare the target entropy value corresponding to each word segmentation unit with a preset entropy value threshold, and select the word segmentation unit with a value greater than the preset entropy value threshold as the global word segmentation unit according to the comparison result.
[0121] Step S406: Select, according to the comparison result, a word segmentation unit whose entropy value is less than or equal to a preset entropy value threshold as a local word segmentation unit.
[0122] Step S408: broadcast the global word segmentation unit to all expert sub-models in the hybrid expert model for processing.
[0123] Step S410, calculating the target selection score of each expert sub-model in the local word segment unit selection hybrid expert model.
[0124] Step S412, determining a local expert sub-model that matches the local word segmentation unit in the hybrid expert model according to the target selection score, and broadcasting the local word segmentation unit to the local expert sub-model for processing.
[0125] Step S414, adjusting the parameters of the hybrid expert model according to the processing results until a target hybrid expert model that meets the optimization stop condition is obtained.
[0126] Step S416: upon receiving the question information submitted by the client, a plurality of question word segmentation units are constructed according to the question information and input into the target hybrid expert model.
[0127] Step S418: Select a target expert sub-model that matches each question word segmentation unit in the target mixed expert model to process the multiple question word segmentation units.
[0128] Step S420: merge the prediction results of the target expert sub-model matched to each question word segmentation unit to obtain answer information, and feed back the answer information to the client.
[0129] In summary, in order to reduce the uncertainty of router selection and improve the robustness of the model, thus ensuring the accuracy of prediction, in the model optimization stage, after inputting multiple word segmentation units into the hybrid expert model, the target entropy value corresponding to each word segmentation unit can be calculated using the router in the hybrid expert model; on this basis, multiple word segmentation units can be divided into global word segmentation units and local word segmentation units according to the target entropy value; then the global word segmentation unit can be broadcast to all expert sub-models in the hybrid expert model for processing, and the local word segmentation unit can be broadcast to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; in the model optimization stage, the uncertain word segmentation unit can be broadcast to all expert sub-models for learning, while the certain word segmentation unit is still used for matching the learning of the expert sub-model, so as to adjust the parameters of the hybrid expert model according to the processing results to obtain the target hybrid expert model that meets the optimization stop condition. In the fine-tuning process of the hybrid expert model, all expert sub-models can learn the relevant knowledge of the uncertain word segmentation unit to ensure that the expert sub-model has better robustness in the application stage.
[0130] Corresponding to the above method embodiment, this specification also provides a model optimization device embodiment, Figure 5 FIG. 1 shows a schematic diagram of a model optimization device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:
[0131] The calculation module 502 is configured to input a plurality of word segmentation units into the hybrid expert model, and calculate a target entropy value corresponding to each word segmentation unit through a router in the hybrid expert model;
[0132] A division module 504 is configured to divide the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value;
[0133] The processing module 506 is configured to broadcast the global word segmentation unit to all expert sub-models in the hybrid expert model for processing, and broadcast the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing;
[0134] The parameter adjustment module 508 is configured to adjust the parameters of the hybrid expert model according to the processing results until a target hybrid expert model that meets the optimization stop condition is obtained.
[0135] In an optional embodiment, the calculation of the target entropy value corresponding to any one of the multiple word segmentation units includes:
[0136] Through the router in the hybrid expert model, the selection score of each expert sub-model in the hybrid expert model selected by the word segmentation unit is calculated; the initial entropy value of the selection score corresponding to each expert sub-model is determined, and the target entropy value corresponding to the word segmentation unit is obtained by normalizing the initial entropy value.
[0137] In an optional embodiment, the division module 504 is further configured to:
[0138] The target entropy value corresponding to each word segmentation unit is compared with a preset entropy value threshold, and the word segmentation unit whose value is greater than the preset entropy value threshold is selected as the global word segmentation unit according to the comparison result, and the word segmentation unit whose value is less than or equal to the preset entropy value threshold is selected as the local word segmentation unit; or, the multiple word segmentation units are sorted according to the target entropy value corresponding to each word segmentation unit to obtain a word segmentation unit sequence, and a global word segmentation unit is selected from the word segmentation unit sequence according to preset proportion information, and the remaining word segmentation units except the global word segmentation unit are used as local word segmentation units.
[0139] In an optional embodiment, the processing module 506 is further configured to:
[0140] Calculate the target selection score of each expert sub-model in the hybrid expert model for selecting the local word segmentation unit; determine the local expert sub-model matching the local word segmentation unit in the hybrid expert model according to the target selection score, and broadcast the local word segmentation unit to the local expert sub-model for processing.
[0141] In an optional embodiment, the parameter adjustment module 508 is further configured to:
[0142] The hybrid expert model is adjusted according to the processing result, and it is detected whether the hybrid expert model after the adjustment satisfies the optimization stop condition; if not, multiple candidate word segmentation units are selected as the multiple word segmentation units, and the multiple word segmentation units are input into the hybrid expert model, and the target entropy value corresponding to each word segmentation unit is calculated through the router in the hybrid expert model; if so, the hybrid expert model after the adjustment is used as the target hybrid expert model.
[0143] In an optional embodiment, the device further comprises:
[0144] The word segmentation module is configured to obtain original sample data associated with the hybrid expert model, preprocess the original sample data to obtain multiple initial word segmentation units; and embed the multiple initial word segmentation units respectively to obtain multiple word segmentation units.
[0145] The model optimization device provided in this embodiment can reduce the uncertainty of router selection, improve the robustness of the model, and thus ensure the prediction accuracy. In the model optimization stage, after inputting multiple word segmentation units into the hybrid expert model, the target entropy value corresponding to each word segmentation unit can be calculated using the router in the hybrid expert model; on this basis, the multiple word segmentation units can be divided into global word segmentation units and local word segmentation units according to the target entropy value; then the global word segmentation unit can be broadcast to all expert sub-models in the hybrid expert model for processing, and the local word segmentation unit can be broadcast to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; in the model optimization stage, the uncertain word segmentation unit can be broadcast to all expert sub-models for learning, while the determined word segmentation unit is still used for matching the learning of the expert sub-model, so as to adjust the parameters of the hybrid expert model according to the processing results to obtain the target hybrid expert model that meets the optimization stop condition. In the fine-tuning process of the hybrid expert model, all expert sub-models are allowed to learn the relevant knowledge of the uncertain word segmentation unit to ensure that the expert sub-model has better robustness in the application stage.
[0146] The above is a schematic scheme of a model optimization device of this embodiment. It should be noted that the technical scheme of the model optimization device and the technical scheme of the above-mentioned model optimization method belong to the same concept, and the details not described in detail in the technical scheme of the model optimization device can be referred to the description of the technical scheme of the above-mentioned model optimization method.
[0147] Corresponding to the above method embodiment, this specification also provides an information processing device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of an information processing device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:
[0148] The information acquisition module 602 is configured to acquire question information submitted by the client, construct a plurality of question word segmentation units according to the question information, and input the units into the target hybrid expert model, wherein the target hybrid expert model is obtained by the above method;
[0149] The word segmentation processing module 604 is configured to select a target expert sub-model matching each question word segmentation unit in the target mixed expert model to process the multiple question word segmentation units;
[0150] The feedback information module 606 is configured to merge the prediction results of the target expert sub-model matched with each question word segmentation unit, obtain answer information, and feed back the answer information to the client.
[0151] The above is a schematic scheme of an information processing device of this embodiment. It should be noted that the technical scheme of the information processing device and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the information processing device can be referred to the description of the technical scheme of the above information processing method.
[0152] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.
[0153] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0154] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0155] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.
[0156] The processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned model optimization method or information processing method.
[0157] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned model optimization method or information processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned model optimization method or information processing method.
[0158] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model optimization method or information processing method.
[0159] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned model optimization method or information processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned model optimization method or information processing method.
[0160] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned model optimization method or information processing method when executed by a processor.
[0161] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned model optimization method or information processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned model optimization method or information processing method.
[0162] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0163] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0164] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0165] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0166] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.
Claims
1. A model optimization method, characterized in that: include: Inputting a plurality of word segmentation units into a hybrid expert model, and calculating a target entropy value corresponding to each word segmentation unit through a router in the hybrid expert model; Dividing the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value; Broadcasting the global word segmentation unit to all expert sub-models in the hybrid expert model for processing, and broadcasting the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; The hybrid expert model is adjusted according to the processing results until a target hybrid expert model that meets the optimization stop condition is obtained.
2. The model optimization method according to claim 1, characterized in that: The calculation of the target entropy value corresponding to any one of the multiple word segmentation units includes: Calculating, by means of a router in the hybrid expert model, a selection score of each expert sub-model in the hybrid expert model selected by a word segmentation unit; An initial entropy value of the selection score corresponding to each expert sub-model is determined, and a target entropy value corresponding to the word segmentation unit is obtained by normalizing the initial entropy value.
3. The model optimization method according to claim 1, characterized in that: The dividing the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value comprises: Compare the target entropy value corresponding to each word segmentation unit with a preset entropy value threshold, and select the word segmentation unit greater than the preset entropy value threshold as the global word segmentation unit according to the comparison result, and select the word segmentation unit less than or equal to the preset entropy value threshold as the local word segmentation unit; or, The multiple word segmentation units are sorted according to the target entropy value corresponding to each word segmentation unit to obtain a word segmentation unit sequence, a global word segmentation unit is selected from the word segmentation unit sequence according to preset proportion information, and the remaining word segmentation units except the global word segmentation unit are used as local word segmentation units.
4. The model optimization method according to claim 1, characterized in that: The broadcasting of the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing includes: Calculate the target selection score of each expert sub-model in the hybrid expert model for selecting the local word segment unit; A local expert sub-model matching the local word segmentation unit is determined in the hybrid expert model according to the target selection score, and the local word segmentation unit is broadcast to the local expert sub-model for processing.
5. The model optimization method according to claim 1, characterized in that: The step of adjusting the parameters of the hybrid expert model according to the processing result until a target hybrid expert model that meets the optimization stop condition is obtained includes: Adjusting the parameters of the hybrid expert model according to the processing result, and detecting whether the hybrid expert model after the adjustment satisfies the optimization stop condition; If not, select multiple candidate word segmentation units as the multiple word segmentation units, and perform the step of inputting the multiple word segmentation units into the hybrid expert model, and calculating the target entropy value corresponding to each word segmentation unit through the router in the hybrid expert model; If so, the adjusted hybrid expert model is used as the target hybrid expert model.
6. The model optimization method according to claim 1, characterized in that: Before the step of inputting the plurality of word segmentation units into the hybrid expert model is executed, the method further comprises: Acquire original sample data associated with the hybrid expert model, preprocess the original sample data, and obtain a plurality of initial word segmentation units; The multiple initial word segmentation units are respectively embedded to obtain multiple word segmentation units.
7. An information processing method, characterized in that: include: Obtaining question information submitted by a client, constructing a plurality of question word segmentation units according to the question information and inputting the units into a target hybrid expert model, wherein the target hybrid expert model is obtained by the method according to any one of claims 1 to 7; Selecting a target expert sub-model matching each question word segmentation unit in the target hybrid expert model to process the multiple question word segmentation units; The prediction results of the target expert sub-model matched to each question word segmentation unit are merged to obtain answer information, and the answer information is fed back to the client.
8. A model optimization device, characterized in that: include: A calculation module is configured to input a plurality of word segmentation units into a hybrid expert model, and calculate a target entropy value corresponding to each word segmentation unit through a router in the hybrid expert model; A division module is configured to divide the plurality of word segmentation units into global word segmentation units and local word segmentation units according to the target entropy value; A processing module is configured to broadcast the global word segmentation unit to all expert sub-models in the hybrid expert model for processing, and broadcast the local word segmentation unit to the expert sub-model matching the local word segmentation unit in the hybrid expert model for processing; The parameter adjustment module is configured to adjust the parameters of the hybrid expert model according to the processing results until a target hybrid expert model that meets the optimization stop condition is obtained.
9. An information processing device, characterized in that: include: an information acquisition module, configured to acquire question information submitted by a client, construct a plurality of question word segmentation units according to the question information, and input the units into a target hybrid expert model, wherein the target hybrid expert model is obtained by the method according to any one of claims 1 to 7; A word segmentation processing module is configured to select a target expert sub-model matching each question word segmentation unit in the target mixed expert model to process the multiple question word segmentation units; The feedback information module is configured to merge the prediction results of the target expert sub-model matched with each question segmentation unit, obtain answer information, and feed back the answer information to the client.
10. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.
11. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 7.
12. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.
Citation Information
Cited By
Model optimization method and apparatus, and information processing method and apparatus
WO2026045029A1
