Prediction method and related device

Abstract

The invention provides a prediction method and a related device, and relates to the technical field of natural language processing. The prediction method can comprise the following steps: determining a current scene, and obtaining a target scene dictionary corresponding to the current scene; activating a target expert network corresponding to the target scene dictionary in the hybrid expert model; wherein the target expert network comprises a personalized expert network corresponding to the current scene; and through the target expert network, based on the target scene dictionary, predicting candidate information matched with model input information. The technical scheme provided by the invention is used for solving the problems of high screening rule maintenance difficulty and poor scene adaptability in the existing scene personalized candidate information recommendation.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a prediction method and related apparatus. Background Technology

[0002] With the development of natural language processing technology, the function of predicting and matching candidate information based on user input information has been widely used in scenarios such as intelligent input methods, chat assistants, and text editing.

[0003] In existing technologies, candidate information can generally be predicted using predictive models (such as language models) based on contextual information. Since users have different word usage habits in different scenarios, to reflect scenario personalization, in addition to model prediction, candidate information matching the scenario can be filtered from the user's personalized dictionary (such as a custom thesaurus, commonly used words, etc.) based on preset rules (such as the domain of the word, word usage frequency, etc.). Finally, the model prediction results and rule-based filtering results are combined to output candidate information that includes the personalized characteristics of the scenario.

[0004] However, the above-mentioned screening rules rely on manual setting. The more rules there are, the more difficult it is to maintain. Moreover, the number of rules set manually is limited, which can easily lead to poor adaptability to different scenarios. Summary of the Invention

[0005] Based on the defects and shortcomings of the existing technology, this application proposes a prediction method and related apparatus to solve the problems of high difficulty in maintaining screening rules and poor scene adaptability in the current personalized candidate information recommendation.

[0006] According to a first aspect of this application, a prediction method is provided, the method comprising:

[0007] Determine the current scene and obtain the target scene dictionary corresponding to the current scene;

[0008] Activate the target expert network in the hybrid expert model that corresponds to the target scene dictionary; wherein, the target expert network includes a personalized expert network corresponding to the current scene; the correspondence between the scene dictionary and the expert network is obtained through model training;

[0009] The target expert network predicts candidate information that matches the model input information based on the target scene dictionary.

[0010] This application utilizes a scene-aware expert selection mechanism to achieve deeply personalized text generation, providing an accurate, efficient, and scene-adaptive solution for applications such as input methods and chat assistants. Furthermore, this eliminates the need for model prediction followed by selecting scene-specific candidate information from the user's personalized dictionary based on personalized filtering rules, thus overcoming the problems of difficult filtering rule maintenance and poor scene adaptability.

[0011] In some optional embodiments, when the candidate information consists of statements and there are multiple statements, the method further includes:

[0012] Analyze the semantic accuracy of each candidate piece of information separately;

[0013] Candidate information with semantic accuracy greater than or equal to a preset value will be used as the final candidate information.

[0014] In this application, when the candidate information is a sentence, the prediction model can also understand the complete semantics of the whole sentence, analyze the semantic accuracy of each candidate information, and select the candidate information with higher semantic accuracy as the final candidate information, which can improve the accuracy of the candidate information.

[0015] In some optional embodiments, the target scenario dictionary includes user-personified terms for the current scenario;

[0016] The step of predicting candidate information that matches the model input information based on the target scene dictionary through the target expert network includes:

[0017] The target expert network predicts candidate information that matches the model input information, based at least on user-personalized words in the target scene dictionary.

[0018] In this application, when predicting candidate information, a target expert network can be used to predict candidate information that matches the model input information, based at least on user-personalized words in the target scene dictionary. This enables personalized prediction for different users in the same scene and meets the personalized needs of users.

[0019] In some alternative embodiments, the target expert network further includes a general expert network for predicting candidate information for the general scenario portion.

[0020] In this application, the general expert network can be used to predict text candidate information in general scenarios, that is, to handle general tasks in the input data that are not covered by experts in a specific domain, and to ensure the model’s adaptability and robustness to diverse inputs.

[0021] In some alternative embodiments, when the current scenario cannot be determined, the method further includes:

[0022] Obtain a general dictionary for specific scenarios;

[0023] The gating network in the hybrid expert network model is used to activate the general expert network corresponding to the general dictionary of the scene.

[0024] The general expert network, based on the general scene dictionary, predicts candidate information that matches the current input information.

[0025] In this application, in order to ensure the model output prediction results when the current scene cannot be determined, a general expert network can be activated. The general expert network predicts candidate information that matches the current input information based on a general scene dictionary, thereby improving the reliability of the model.

[0026] In some alternative embodiments, determining the current scenario includes:

[0027] Obtain target information; wherein, the target information includes at least one of the following: the name of the foreground application that calls the prediction model, and the input context information;

[0028] Based on the target information, the current scene is determined.

[0029] In this application, the current scene is automatically determined by calling the target information such as the front-end application name of the prediction model and the input context information, which can improve the intelligence of the prediction method.

[0030] According to a second aspect of this application, a prediction apparatus is provided for use with a prediction model equipped with a hybrid expert network model, the apparatus comprising:

[0031] The scene processing module is used to determine the current scene and obtain the target scene dictionary corresponding to the current scene;

[0032] An expert activation module is used to activate the target expert network in the hybrid expert network model that corresponds to the target scene dictionary; wherein, the target expert network includes a personalized expert network corresponding to the current scene;

[0033] The prediction module is used to predict candidate information that matches the current input information based on the target scene dictionary through the target expert network.

[0034] In some alternative embodiments, when the candidate information consists of statements and there are multiple statements, the apparatus may further include:

[0035] The speech analysis module is used to analyze the semantic accuracy of each candidate piece of information.

[0036] The determination module is used to select candidate information whose semantic accuracy is greater than or equal to a preset value as the final candidate information.

[0037] In some optional embodiments, the target scenario dictionary includes user-personified terms for the current scenario;

[0038] The prediction module is specifically used for:

[0039] The target expert network predicts candidate information that matches the model input information, based at least on user-personalized words in the target scene dictionary.

[0040] In some alternative embodiments, the target expert network further includes a general expert network for predicting candidate information for the general scenario portion.

[0041] In some optional embodiments, when the current scene cannot be determined, the scene processing module is further configured to obtain a general scene dictionary; the expert activation module is further configured to activate a general expert network corresponding to the general scene dictionary through a gating network in the hybrid expert network model; the prediction module can also be configured to predict candidate information matching the current input information based on the general scene dictionary through the general expert network.

[0042] In some optional embodiments, the scene processing module is specifically used for:

[0043] Obtain target information; determine the current scene based on the target information. The target information includes at least one of the following: the name of the foreground application that invokes the prediction model, and the input context information.

[0044] According to a third aspect of this application, an electronic device is provided, comprising: a memory and a processor;

[0045] The memory is connected to the processor and is used to store programs;

[0046] The processor is used to implement the prediction method as described in the first aspect by running a program in the memory.

[0047] According to a fourth aspect of this application, a storage medium is provided that stores a computer program, which, when executed by a processor, implements the prediction method as described in the first aspect.

[0048] According to a fifth aspect of this application, a computer program product or computer program is provided, the computer program product including the computer program, wherein a processor of a computer device executes the computer program to implement the steps in the prediction method as described in the first aspect. Optionally, the computer program may be stored on a readable storage medium of the computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0049] The technical solution provided in this application utilizes a prediction model incorporating a hybrid expert model to perform personalized scenario prediction of candidate information based on the current scenario. Specifically, based on a target scenario dictionary corresponding to the current scenario, the gating network in the hybrid expert model activates the corresponding target expert network, which then predicts candidate information matching the current scenario. In this hybrid expert model, different expert networks are trained to predict candidate information for different scenarios, thus enabling personalized scenario prediction. This eliminates the need to filter personalized scenario candidate information from the user's personalized dictionary based on personalized filtering rules after model prediction, overcoming the difficulty of maintaining filtering rules. Furthermore, the model possesses strong predictive reasoning capabilities and can flexibly adapt to various scenarios, thereby improving the scenario adaptability of the prediction results. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the structure of a prediction model provided in an embodiment of this application;

[0052] Figure 2 A flowchart illustrating a prediction method provided in an embodiment of this application;

[0053] Figure 3 A block diagram of a prediction device provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0056] Application Overview

[0057] In existing application scenarios such as intelligent input methods, chat assistants, and text editing, predicting and recommending candidate information based on the information already input by the user is a key technology to improve the user experience and input efficiency. For example, in the scenario of an intelligent input method, based on the pinyin string input by the user, the corresponding words are predicted and the recommended candidate words are output; for another example, in the scenario of a chat assistant, based on the chat content being input by the user, the complete chat content is predicted and the recommended candidate sentences are output. For another example, in the scenario of text editing, specifically in the order evaluation scenario of an e-commerce platform, based on the already input evaluation content, the candidate content for subsequent evaluation can be recommended.

[0058] In the prior art, generally, candidate information can be predicted through a prediction model (such as a language model) based on context information, etc. Since users have different language usage habits in different scenarios, for example, in the daily chat scenario, they generally tend to use relaxed and casual language, while in the work scenario, they generally tend to use relatively formal language. For example, in the daily chat scenario, when expressing hunger and wanting to eat to a friend, the user may input "I'm hungry and need to be fed". In the work scenario, when expressing hunger and wanting to eat to a leader, the user may input "It's meal time. Can I go to have a meal?". For another example, in the daily chat scenario, after inputting the pinyin string "nihao" through the input method, the user may select "拟好" from the candidate words, while in the work scenario, after inputting the pinyin string "nihao" through the input method, the user generally selects "你好" from the candidate words.

[0059] Therefore, in order to reflect the personalization of the scenario, on the basis of model prediction, candidate information matching the scenario can also be obtained from the user's personalized dictionary (such as a custom word library, commonly used words, etc.) based on preset rules (such as the field to which the word belongs, the usage frequency of the word, etc.). Finally, the candidate information including the personalized characteristics of the scenario is output by integrating the model prediction results and the rule screening results. For example, the prediction model outputs candidate words such as "此种" and "辞职" based on the input pinyin string "ciz". The current scenario is the thesis editing scenario, and the word "磁轴" is frequently used in this scenario in the user's personalized dictionary. Therefore, the finally output candidate information can include: "此种", "辞职", "磁轴", etc..

[0060] However, this existing solution, which relies on preset rules to filter candidate information for personalized scenarios, has obvious limitations, as described below:

[0061] Rule maintenance is difficult because the formulation, adjustment, and optimization of rules usually require manual intervention. As user input habits evolve or application scenarios expand, established rules may quickly become invalid or fail to cover new situations. Maintaining an efficient, universal, and dynamically adaptable rule system is costly and difficult to automate.

[0062] Poor scenario adaptability: Preset rules are essentially human-defined pattern matching logic, which lacks flexibility and generalization ability. When faced with complex and ever-changing natural language contexts and highly personalized user expressions (such as domain-specific terminology, emerging internet slang, user-created vocabulary, or special sentence structures), fixed rules often fail to comprehensively and accurately capture all relevant features, resulting in limited coverage and accuracy of personalized recommendations.

[0063] In summary, existing personalized candidate information recommendation methods suffer from challenges such as the difficulty in maintaining filtering rules and poor scenario adaptability.

[0064] To address the aforementioned technical problems, this application provides a prediction technique that uses a prediction model incorporating a hybrid expert model to perform personalized prediction of candidate information based on the current scene. Specifically, based on a target scene dictionary corresponding to the current scene, the gating network in the hybrid expert model activates the corresponding target expert network, which then predicts candidate information matching the current scene. In this hybrid expert model, different expert networks are trained to predict candidate information for different scenes, thus enabling personalized prediction of candidate information. This eliminates the need to filter personalized candidate information from a user-personalized dictionary based on personalized filtering rules after model prediction, overcoming the difficulty of maintaining filtering rules. Furthermore, the model has strong predictive reasoning capabilities and can flexibly adapt to various scenes, thereby improving the scene adaptability of the prediction results.

[0065] Exemplary methods

[0066] This application provides a prediction method applied to electronic devices with text prediction capabilities. The electronic device can be a terminal device, such as a mobile phone, tablet computer, desktop computer, personal digital assistant device, or a server.

[0067] The prediction method provided in this application embodiment can be implemented by a prediction model installed in an electronic device. The prediction model can be installed in the electronic device along with the corresponding application, such as along with an input method application, a chat software application, etc., or it can be installed in the electronic device in other ways.

[0068] The prediction model can be a Transformer model, whose core function is to transform the input data matrix. This transformation is achieved by calculating the relationships between the vectors within the matrix, removing invalid information and enhancing valid information. During this process, the values ​​of the vectors that make up the matrix change, but their shapes remain unchanged. This transformation allows the model to better solve the final mathematical problem and obtain the optimal solution.

[0069] like Figure 1 As shown, the model structure of this prediction model may include: a multi-head attention layer, a hybrid expert model, and a normalization layer.

[0070] The multi-head attention layer is the core component of this prediction model. By computing multiple independent attention mechanisms in parallel, the model can capture diverse semantic associations of the input from different perspectives. This mechanism segments the input features into multiple 'heads,' each of which learns attention weights independently. Finally, the information is integrated through concatenation and linear transformation, which can improve sequence modeling capabilities.

[0071] Among them, the Mixture of Experts (MFA) model is a machine learning method that improves computational efficiency and inference performance by dividing the model into multiple sub-networks (i.e., expert networks), each of which specializes in a specific subset of the input data.

[0072] In traditional Transformer models, each encoder and / or decoder layer contains a position-independent feedforward neural network (FFN). This feedforward network typically consists of two linear transformations and an activation function, using the same parameter set for all input tokens and all scenarios. While this "one-size-fits-all" network design results in a large number of parameters, it struggles to adequately capture the specific language habits required for highly personalized or scenario-specific text generation tasks. For example, the expected vocabulary and expressions of users in work emails and social chat scenarios may differ drastically.

[0073] In this embodiment, the feedforward neural network in the traditional Transformer model is replaced with a hybrid expert model. By training the hybrid expert model, different expert networks can be used to predict candidate information for different scenarios.

[0074] like Figure 1 As shown, the hybrid expert model can include a gating network and multiple expert networks.

[0075] The gating network is responsible for dynamically assigning weights to each expert network based on the input and current scene information, which can be represented by the following equation (1):

[0076] G(x,s)=Softmax(W g ×[x;E(S)]+b g (1)

[0077] Here, x represents the hidden state of the current token. The hidden state is the result of the neural network layers (such as self-attention layers, feedforward layers, etc.) calculating the input token, and is usually a high-dimensional vector. E(S) is the embedding representation of the scene S. Embedding representation is the process of mapping high-dimensional, sparse raw data (such as text, images, audio, etc.) to a low-dimensional, dense vector space, with the aim of capturing the inherent features and semantic relationships of the data. [x; E(S)] represents vector concatenation. W g and b g These are the weights and bias parameters of the gating network. The Softmax() function is a widely used activation function in machine learning and deep learning, primarily used to transform a real-valued vector into a probability distribution.

[0078] Each expert network (FFN) can maintain the same structure as the FFN in the traditional Transformer model, but its parameters are independent. Each expert network can be represented by the following equation (2):

[0079] FFN i (x)=W 2i ×GELU(W 1i ×x+b 1i )+b 2i (2)

[0080] Among them, W 1i Let b represent the weights of the In Projection in the i-th expert network FFN. 1i This represents the bias value of the In Projection in the i-th expert network FFN. The In Projection is the first linear transformation in each expert network of the FFN, responsible for mapping the input features to a high-dimensional space, preparing for subsequent nonlinear transformations.

[0081] W 2i b represents the weight of the Out Projection in the i-th expert network FFN.2i This represents the bias value of the Out Projection in the i-th expert network FFN. The Out Projection is used to map high-dimensional features back to the original dimension. Gelu() is an activation function, short for Gaussian Error Linear Unit, used for nonlinear transformations.

[0082] Each expert network in the hybrid expert model is trained to specialize in a specific text generation task. For example, expert network 1 is used for text generation in everyday chat scenarios; expert network 2 is used for text generation in work email scenarios; expert network 3 is used for text generation in technical document scenarios; and expert network 4 is used for text generation in literary creation scenarios, etc. The expert networks can be trained according to specific needs.

[0083] The hybrid expert model in this application embodiment can be represented by the following equation (3):

[0084] MoE-FFN(x)=∑ i G(x,S) i ×FFN i (x) (3)

[0085] Here, x represents the input vector, which comes from the output of the multi-head attention layer. G(x, S) i This represents the weight allocation of the gating network to the i-th expert, where S represents the current scene information; FFN i Let i represent the feedforward network of the i-th expert network.

[0086] The normalization layer is used to scale and normalize the input vector, thereby improving computational efficiency while maintaining model performance.

[0087] Optionally, the normalization layer mentioned here can be a root mean square layer normalization (RMSNorm) layer. The core idea of ​​RMSNorm is to replace the standard deviation with the root mean square and omit mean centering, adaptively adjusting the feature distribution through learnable parameters. RMSNorm simplifies the normalization process, significantly improving computational efficiency while maintaining model performance, making it particularly suitable for large-scale models such as the Transformer.

[0088] The method is described in detail below through some embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Figure 2 As shown, the prediction method may include steps 201 to 203, as detailed below:

[0089] Step 201: Determine the current scene and obtain the target scene dictionary corresponding to the current scene.

[0090] The scenarios described here can be determined automatically by electronic devices or input manually, as detailed below.

[0091] The current scenario described here can be determined automatically by the electronic device or determined by the electronic device based on user input, as detailed below.

[0092] In cases of automatic determination, after invoking the prediction model, the electronic device can obtain target information and determine the current scene based on the target information. The target information mentioned here may include at least one of the following: the name of the foreground application that invoked the prediction model, the input context information, etc.

[0093] For example, if the predictive model is deployed within an input method application, and a user launches a work-related email application on their computer and edits the email content, the email application will then invoke the input method application, which in turn will invoke the predictive model. The input method application can then determine whether the current scenario is a work email scenario or a combination of both, based on the currently launched foreground application name (i.e., the email application name) and / or the already entered email content. The scenario division can be configured according to actual needs. In this example, the input method application is a system service application, not a foreground application.

[0094] For example, if the predictive model is deployed in a chat assistant application, when a user launches a chat application on their mobile device, the chat application can call the chat assistant application, which in turn calls the predictive model. As the user inputs chat content through the chat application, the chat assistant application can determine that the current scenario is a chat scenario based on the currently launched foreground application name (i.e., the chat assistant name) and / or the already entered chat content. In this example, the chat assistant application is a system service application, not a foreground application.

[0095] In this embodiment of the application, the current scene is automatically determined by calling the target information such as the front-end application name and the input context information of the prediction model, which can improve the intelligence of the prediction method.

[0096] For human input scenarios, multiple scene options can be set, and the current scene is determined based on the user's selection of a scene option. This can be implemented by: receiving the user's selection of a target scene option; and setting the scene corresponding to the target scene option as the current scene. This target scene option can be one of multiple scene options.

[0097] Specifically, upon detecting a call to the predictive model, a pop-up window can be displayed prompting the user to select a scenario. If the user selects yes, multiple scenario options are displayed for the user to choose from. Allowing the user to select the scenario enhances their control, enabling them to choose according to their own needs and improving the user experience.

[0098] The scenario dictionary described in this embodiment is pre-set, and there can be multiple scenario dictionaries, each used to represent different scenarios. Each scenario dictionary includes words corresponding to the scenario it represents. For example, the scenario dictionary for a work email scenario may include words such as "Sincerely," "Respectfully," "Your Company," "Sincerely," and "Respected."

[0099] The feature dictionary E(S) for each scene S can be represented by the following equation (4):

[0100] E(S) = f({w1,w2,…,w... k}) (4)

[0101] Where, {w1,w2,...,w k} represents the feature dictionary of scenario S; f represents a mapping function that maps the feature dictionary to the corresponding feature vector.

[0102] Alternatively, the function f can be an average value function, in which case E(S) = (1 / k) × ∑ i Emb(w i ), where Emb represents the word embedding function, which maps discrete symbols (such as words and characters) into continuous vector representations to capture semantic information. k represents the number of words in the feature dictionary. w i It represents a word in the feature dictionary.

[0103] Alternatively, the function f can also be a weighted average function, in which case E(S) = ∑ i α i ×Emb(w i ), where α i It is the word w i Importance weights.

[0104] Alternatively, the function f can also be a small encoder, in which case E(S) = Encoder({w1,w2,...,w...). k}).

[0105] In this embodiment of the application, after determining the current scene, a target scene dictionary corresponding to the current scene can be loaded, and the number of target scene dictionaries can be at least one.

[0106] Step 202: Activate the target expert network in the hybrid expert model that corresponds to the target scene dictionary.

[0107] After obtaining the corresponding target scene dictionary, the gating network in the hybrid expert model can activate the target expert network in the hybrid expert model that corresponds to the target scene dictionary, based on the target scene dictionary.

[0108] The correspondence between the scene dictionary and the expert network can be obtained through model training.

[0109] The activated target expert network is the expert network with higher weights, typically the top K (K < N) expert networks ranked from highest to lowest weight, such as the top two expert networks. The weights of each expert network are pre-trained using the model, depending on the scenario.

[0110] The target expert network includes at least a personalized expert network corresponding to the current scenario, such as an expert network corresponding to a work email scenario, or an expert network corresponding to a work scenario and an expert network corresponding to an email scenario, thereby achieving scenario-related personalized prediction.

[0111] Step 203: Using a target expert network and based on the target scene dictionary, predict candidate information that matches the model input information.

[0112] In this embodiment, the target expert network can predict candidate information matching the current input information based on the model input information. For example, if the model input information includes a pinyin string, it predicts candidate words corresponding to the pinyin string; if the model input information includes words or phrases, it predicts candidate information for the next word or phrase. During the prediction process, the target expert network can refer to a target scene dictionary to improve prediction accuracy.

[0113] Optionally, the model input information may also include contextual information, and combining contextual information for prediction can further improve prediction accuracy.

[0114] In this embodiment, a scene-aware expert selection mechanism enables deeply personalized text generation, providing an accurate, efficient, and scene-adaptive solution for applications such as input methods and chat assistants. Furthermore, this eliminates the need for model prediction followed by filtering scene-specific candidate information from the user's personalized dictionary based on personalized filtering rules, thus overcoming the difficulty of maintaining filtering rules.

[0115] In some alternative embodiments, when the candidate information is a statement and there are multiple statements, after predicting the candidate information that matches the model input information, the method may further include steps A1 and A2, as described below.

[0116] Step A1: Analyze the semantic accuracy of each candidate information separately.

[0117] Step A2: Use the candidate information with a semantic accuracy greater than or equal to the preset value as the final candidate information.

[0118] In the embodiments of this application, when the candidate information is a sentence, the prediction model can also understand the complete semantics of the whole sentence, analyze the semantic accuracy of each candidate information, and screen out the candidate information with higher semantic accuracy as the final candidate information, which can improve the accuracy of the candidate information. For example, for an input method application, when the user inputs the拼音 string "nihaotiankong", the sentence candidate information predicted by the model includes: "你好天空" (Hello sky), "你好填空" (Hello fill in the blank). Through semantic analysis, it is found that "你好天空" has smooth semantics, that is, high accuracy; while "你好填空" has unsmooth semantics, that is, low accuracy. Therefore, "你好天空" can be used as the final candidate information.

[0119] Optionally, when judging the semantic accuracy, the context information can also be combined for judgment. The sentence with higher accuracy determined in this way can fit better with the current context. For example, for a chat assistant application, when the user is currently inputting "我想出去旅游" (I want to travel), the candidate information predicted by the model can include "去A城市怎么样" (How about going to City A) and "能陪我一起去吗" (Can you accompany me there). Based on the context information, it is determined that before this, the user described relevant descriptions about how City A is suitable for tourism, then "去A城市怎么样" can be used as the final candidate information; if based on the context information, it is determined that before this, the user was asking a friend if they have time in July, then "能陪我一起去吗" can be used as the final candidate information.

[0120] Among them, for the existing input method applications, the candidate information is generally a partial understanding of 2 - 3 characters, and the context information is not considered. For example, for the拼音 string "nihaotiankong" input by the user through the input method, the traditional method only looks at local statistics. For example, nihao→"你好" (Hello), with the highest corresponding probability of 95%; tiankong→"填空" (Fill in the blank), with the highest corresponding probability of 90%. Then the prediction result is: "你好填空", which has the problem of unsmooth semantics. Therefore, the traditional method generally only implements word - class prediction and is difficult to implement sentence - class prediction. Through the method provided by this embodiment, the above problems existing in the existing input method applications can be overcome, sentence - class prediction can be achieved, and candidate information with higher overall accuracy can be provided, thereby improving the user's input efficiency.

[0121] In some optional embodiments, the target scenario dictionary includes the user's personalized vocabulary in the current scenario.

[0122] For pre-trained but unused prediction models, initial scene dictionaries can be configured for different scenarios. During model usage, user-specific vocabulary can be continuously accumulated and updated to the corresponding scene dictionaries. For example, in a physics learning scenario dictionary, users frequently use terms like "magnetic flux" and "solenoid valve," which are not recorded in the dictionary. These terms can be added and marked as user-specific vocabulary. When predicting candidate information, a target expert network can predict candidate information matching the model input information, based at least on user-specific vocabulary in the target scene dictionary. This enables personalized predictions for different users within the same scenario, meeting individual user needs.

[0123] Optionally, in addition to recording and marking user-personalized words, the scene dictionary can also be used to assign weights to different user-personalized words based on their frequency of use. This allows for a focus on predicting candidate information based on words with higher weights, improving prediction accuracy. For example, in a user chat scenario, the frequency of the word "nihao" is lower than that of "nihao," so the weight of "nihao" can be set higher than that of "nihao." When the pinyin string "nihao" is detected, candidate information can be prioritized based on "nihao," such as outputting "nihao" as the primary candidate information.

[0124] In some alternative embodiments, the target expert network may also include a general expert network.

[0125] This general expert network can be used to predict text candidate information in general scenarios, that is, to handle general tasks in the input data that are not covered by experts in a specific domain, ensuring the model's adaptability and robustness to diverse inputs.

[0126] In some alternative embodiments, when the current scenario cannot be determined, the method may further include steps B1 to B3, as described below:

[0127] Step B1: Obtain the general dictionary for the scene.

[0128] Step B2: Activate the general expert network corresponding to the scene general dictionary through the gating network in the hybrid expert network model.

[0129] Step B3: Using a general expert network and a general scene dictionary, predict candidate information that matches the current input information.

[0130] In cases where the current scene cannot be determined, a general expert network can be activated to ensure the model outputs a prediction result. The general expert network predicts candidate information that matches the current input information based on a general scene dictionary, thereby improving the reliability of the model.

[0131] In some alternative embodiments, the prediction model can be trained through the following process.

[0132] 1. Training data preparation.

[0133] Text corpora are collected according to scenario categories as training samples. Specifically, dedicated text corpora are collected for each scenario (such as daily chat scenarios, work email scenarios, etc.) as training samples. Each training sample is labeled with its respective scenario S.

[0134] To construct a scene dictionary for different scenarios, specifically: extract feature words for each scenario and generate a scene embedding representation E(S).

[0135] 2. Training process.

[0136] The training can be divided into two phases.

[0137] Phase 1 is pre-training, which initializes the model parameters of the basic prediction model and trains the hybrid expert model in the basic prediction model using a general corpus, so that the experts are initially differentiated and an intermediate prediction model is obtained.

[0138] Phase 2 is scene-specific expert training, which uses scene-labeled corpora to train intermediate prediction models. The loss function in this phase may include the language model loss function (see Equation (5) below), the expert balance loss function (see Equation (6) below), and the total loss function (see Equation (7) below).

[0139] Language model loss function: L_LM = -log P(w t |w1,w2,...,w t-1 ,S) (5)

[0140] The significance of the language model loss function lies in enabling the model to "speak human language" in order to accurately predict text. In this function, wt represents the content to be predicted, and S represents scene information; log P(w t |w1, w2, ..., w t-1 S) represents the probability distribution of the model's prediction of wt given a historical word sequence and scene information.

[0141] Expert-balanced loss function: L_Balance = CV(∑ i G(x,S) i (6)

[0142] Where C and V represent the coefficients of variation, C represents the balance coefficient, which is used to control the intensity of balance loss; V represents the difference measurement function, which is used to quantify the degree of imbalance of expert load.

[0143] Total loss function: L=L_LM+λ×L_Balance (7)

[0144] The total loss function combining the language model loss function and the expert balance loss function can not only ensure the performance of language tasks, but also maintain the efficiency and robustness of the model structure.

[0145] 3. Training Algorithm

[0146] For each training batch, a batch of text sequences and their scene labels can be sampled, and the following processing can be performed for each position t in the sequence:

[0147] a. Calculate the attention output h_t;

[0148] b. Calculate the gating network output G(h_t, S);

[0149] c. Calculate the output FFN of each expert i (h_t);

[0150] d. Combine the expert outputs MoE-FFN(h_t) = ∑ i G(h_t,S) i ×FFN i (h_t);

[0151] Then calculate the total loss L, and perform backpropagation according to the total loss L to update the parameters.

[0152] After completing the model training, in actual use, the system can perform inference according to the following process:

[0153] 1. Determine the current scene S;

[0154] 2. The system loads the dictionary representation E(S) of the scene S;

[0155] 3. For each token to be generated:

[0156] a. Calculate the multi-head attention output h_t;

[0157] b. Calculate the gating network output G(h_t, S);

[0158] c. Activate the K experts with the highest weights (usually K << N, such as Top-2 sparse activation);

[0159] d. Calculate the weighted output of the activated experts: MoE-FFN(h_t) = ∑ i ∈Top-KG(h_t,S) i ×FFN i (h_t);

[0160] e. Perform subsequent processing and generate the token probability distribution.

[0161] The following example, using an input method scenario, illustrates the actual application process of the system.

[0162] The scenario is a user composing a work email.

[0163] Step 1: Scene recognition.

[0164] Based on the current front-end application name, the current scenario is determined to be a work email scenario, and the system loads the corresponding scenario embedding representation E("work email").

[0165] Step 2: User input processing.

[0166] User input: "Thank you for your letter."

[0167] The system processes information based on the current input and the scene.

[0168] Step 3: MoE Expert Activation

[0169] The gating network determines the target expert to be activated based on the input and scene information.

[0170] In the context of work emails, Expert 2 (the expert in work email scenarios) has the highest weight at 0.75, followed by Expert 5 (the expert in general text) at 0.20; the total weight of the other experts is 0.05. Therefore, Expert 2 and Expert 5 are activated.

[0171] Step 4: Personalized recommendation generation.

[0172] The system primarily generates recommendations based on the outputs of Expert 2 and Expert 5.

[0173] The generated personalized completion result can be as follows:

[0174] We will process your question as soon as possible;

[0175] We have received your proposal.

[0176] Please provide more relevant information;

[0177] These recommendations clearly adopt the formal tone and expression of work emails, which aligns with the needs of the current scenario.

[0178] Alternatively, in actual implementation, the following parameter configurations can be used:

[0179] The basic prediction model is GPT-2Medium;

[0180] The expert network consists of 8 expert FFNs;

[0181] Each expert network FFN is the same size as the original GPT-2Medium FFN;

[0182] Top-2 sparse activation means that only the two experts with the highest weight are activated each time.

[0183] It supports a variety of predefined scenarios, including "daily chat", "work emails", "academic writing", "creative writing", and "technical documents".

[0184] The gated network is a single-layer feedforward network with input dimensions of hidden state and scene embedding, and output dimension of the number of activated expert networks.

[0185] The above is a description of the prediction method provided in the embodiments of this application.

[0186] In summary, the embodiments of this application, through a scene-aware expert selection mechanism, can achieve deeply personalized text generation, providing an accurate, efficient, and scene-adaptive solution for applications such as input methods and chat assistants. Furthermore, this eliminates the step of filtering scene-specific candidate information from the user's personalized dictionary based on personalized filtering rules after model prediction, thus overcoming the difficulty of maintaining filtering rules. In addition, the embodiments of this application overcome the problem of existing input method applications only being able to perform word-level prediction and struggling to achieve sentence prediction, providing candidate information with higher overall accuracy, thereby improving user input efficiency.

[0187] Exemplary device

[0188] Accordingly, this application also provides a prediction device that can be applied to electronic devices with text prediction capabilities. This electronic device can be a terminal device, such as a mobile phone, tablet computer, desktop computer, personal digital assistant device, etc., or it can be a server.

[0189] Electronic devices can perform text prediction using a predictive model installed within them. This predictive model can be installed along with a corresponding application, such as an input method application or a chat application, or it can be installed through other means.

[0190] like Figure 1 As shown, the model structure of this prediction model may include: a multi-head attention layer, a hybrid expert model, and a normalization layer.

[0191] The multi-head attention layer is the core component of this prediction model. By computing multiple independent attention mechanisms in parallel, the model can capture diverse semantic associations of the input from different perspectives. This mechanism segments the input features into multiple 'heads,' each of which independently learns attention weights. Finally, the information is integrated through concatenation and linear transformation, which improves sequence modeling capabilities.

[0192] Hybrid expert models are a type of machine learning approach that improves computational efficiency and inference performance by dividing the model into multiple sub-networks (i.e., expert networks), each specializing in a specific subset of the input data.

[0193] The normalization layer is used to scale and normalize the input vector, thereby improving computational efficiency while maintaining model performance.

[0194] like Figure 3 As shown, the device may include:

[0195] The scene processing module 301 is used to determine the current scene and obtain the target scene dictionary corresponding to the current scene.

[0196] The expert activation module 302 is used to activate the target expert network in the hybrid expert network model that corresponds to the target scene dictionary.

[0197] The target expert network includes a personalized expert network corresponding to the current scenario.

[0198] The prediction module 303 is used to predict candidate information that matches the current input information based on the target scene dictionary through the target expert network.

[0199] In some alternative embodiments, when the candidate information consists of statements and there are multiple statements, the apparatus may further include:

[0200] The speech analysis module is used to analyze the semantic accuracy of each candidate piece of information.

[0201] The determination module is used to select candidate information with semantic accuracy greater than or equal to a preset value as the final candidate information.

[0202] In some alternative embodiments, the target scenario dictionary includes user-personified terms for the current scenario.

[0203] The prediction module 303 can be specifically used for:

[0204] The target expert network predicts candidate information that matches the model input information, based at least on user-personalized words in the target scene dictionary.

[0205] In some alternative embodiments, the target expert network further includes a general expert network for predicting candidate information for the general scenario portion.

[0206] In some optional embodiments, when the current scene cannot be determined, the scene processing module 301 can also be used to obtain a general scene dictionary; the expert activation module 302 can also be used to activate the general expert network corresponding to the general scene dictionary through the gating network in the hybrid expert network model; the prediction module 303 can also be used to predict candidate information matching the current input information based on the general scene dictionary through the general expert network.

[0207] In some optional embodiments, the scene processing module 301 may be specifically used for:

[0208] Obtain target information; determine the current scene based on the target information. The target information includes at least one of the following: the name of the foreground application that invokes the prediction model, and the input context information.

[0209] The prediction device provided in this application embodiment can achieve deep personalized text generation through a scene-aware expert selection mechanism, providing an accurate, efficient, and scene-adaptive solution for applications such as input methods and chat assistants. This eliminates the need for the step of filtering scene-specific candidate information from the user's personalized dictionary based on personalized filtering rules after model prediction, thus overcoming the problem of difficult filtering rule maintenance. Furthermore, this application embodiment overcomes the problem of existing input method applications only being able to perform word-level prediction and struggling to achieve sentence prediction, providing candidate information with higher overall accuracy, thereby improving user input efficiency.

[0210] The prediction device provided in this embodiment belongs to the same concept as the prediction method provided in the above embodiments of this application. It can execute the prediction method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the prediction method provided in the above embodiments of this application, and will not be repeated here.

[0211] It should be understood that the modules in the above prediction device can be implemented in the form of processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships of the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units of the above device can be implemented entirely in the form of processor calling software, entirely in the form of hardware circuits, or partially in the form of processor calling software with the remaining parts implemented in the form of hardware circuits.

[0212] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0213] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0214] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0215] Exemplary electronic devices

[0216] This application also provides an electronic device, such as... Figure 4 As shown, the electronic device includes a memory 400 and a processor 410.

[0217] The memory 400 is connected to the processor 410 and is used to store programs.

[0218] The processor 410 is used to implement the prediction method in the above embodiments by running the program stored in the memory 400.

[0219] Specifically, the aforementioned electronic device may also include: a communication interface 420, an input device 430, an output device 440, and a bus 450.

[0220] The processor 410, memory 400, communication interface 420, input device 430, and output device 440 are interconnected via a bus. Among them:

[0221] Bus 450 may include a pathway for transmitting information between various components of a computer system.

[0222] The processor 410 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0223] Processor 410 may include a main processor, as well as a baseband chip, modem, etc.

[0224] The memory 400 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 400 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0225] Input device 430 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0226] Output device 440 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0227] The communication interface 420 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0228] The processor 410 executes the program stored in the memory 400 and calls other devices, which can be used to implement the various steps of the prediction method provided in the above embodiments of this application.

[0229] Exemplary computer program products and storage media

[0230] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the prediction method described in the embodiments of this application.

[0231] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0232] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0233] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor of the steps in the prediction method described in the embodiments of this application.

[0234] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0235] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0236] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0237] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0238] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0239] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0240] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0241] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0242] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0243] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element. The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A prediction method, characterized in that, The method, applied to a prediction model equipped with a hybrid expert model, includes: Determine the current scene and obtain the target scene dictionary corresponding to the current scene; Activate the target expert network in the hybrid expert model that corresponds to the target scene dictionary; wherein, the target expert network includes a personalized expert network corresponding to the current scene; The target expert network predicts candidate information that matches the model input information based on the target scene dictionary.

2. The prediction method according to claim 1, characterized in that, When the candidate information consists of multiple statements, after predicting candidate information that matches the model input information, the method further includes: Analyze the semantic accuracy of each candidate piece of information separately; Candidate information with semantic accuracy greater than or equal to a preset value will be used as the final candidate information.

3. The prediction method according to claim 1, characterized in that, The target scenario dictionary includes user-personalized terms for the current scenario; The step of predicting candidate information that matches the model input information based on the target scene dictionary through the target expert network includes: The target expert network predicts candidate information that matches the model input information, based at least on user-personalized words in the target scene dictionary.

4. The prediction method according to claim 1, characterized in that, The target expert network also includes a general expert network for predicting candidate information for general scenarios.

5. The prediction method according to claim 1, characterized in that, In cases where the current scenario cannot be determined, the method further includes: Obtain a general dictionary for specific scenarios; The gating network in the hybrid expert network model is used to activate the general expert network corresponding to the general dictionary of the scene. The general expert network, based on the general scene dictionary, predicts candidate information that matches the current input information.

6. The prediction method according to claim 1, characterized in that, Determining the current scenario includes: Obtain target information; wherein, the target information includes at least one of the following: the name of the foreground application that calls the prediction model, and the input context information; Based on the target information, the current scene is determined.

7. A prediction device, characterized in that, The apparatus, applied to a prediction model equipped with a hybrid expert network model, comprises: The scene processing module is used to determine the current scene and obtain the target scene dictionary corresponding to the current scene; An expert activation module is used to activate the target expert network in the hybrid expert network model that corresponds to the target scene dictionary; wherein, the target expert network includes a personalized expert network corresponding to the current scene; The prediction module is used to predict candidate information that matches the current input information based on the target scene dictionary through the target expert network.

8. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the prediction method as described in any one of claims 1 to 6 by running a program in the memory.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the prediction method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the prediction method as described in any one of claims 1 to 6.

Images