Language model training method, prediction method, apparatus, device, medium and product
By introducing context change sequence training and modal co-evolution modules into the large language model, the problem of insufficient context change capture in existing technologies is solved, and the model's ability to generate and understand in dynamic contexts is improved, making it suitable for multimodal interaction scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE JIUTIAN ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
- Filing Date
- 2026-02-11
- Publication Date
- 2026-05-29
AI Technical Summary
Existing large language models struggle to effectively capture changes in context over time, user/environment behavior, and modal state in multi-turn dialogues involving real-time interaction, continuous user feedback, and changing environments. This results in decreased generation or comprehension performance and reduced prediction accuracy.
The language model is trained by inputting a sequence of contextual changes. The current contextual state is identified by the contextual event detection sub-model. The model is then combined with the modal co-evolution module and the language prediction module to improve the coherence of cross-modal understanding and generation. The evolutionary chain between contextual state, current modality and historical modality is established through temporal encoding, modal attention and cross-modal recoding to train the model and improve prediction accuracy.
It enables continuous optimization of language models in dynamic contexts, improving generation or understanding capabilities and prediction accuracy. It is suitable for interactive and real-time multimodal scenarios, such as AR glasses assistants and intelligent robots.
Smart Images
Figure CN122116881A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a language model training method, prediction method, device, equipment, medium and product. Background Technology
[0002] Currently, in the field of artificial intelligence, especially the technical solutions represented by Large Language Models (LLMs) and their extensions such as multimodal models and foundation models have become mainstream. Foundation models are usually pre-trained on massive amounts of unlabeled data using self-supervised or weakly supervised learning to achieve good general representation capabilities, and then adapted to downstream tasks through fine-tuning or prompting.
[0003] However, current large language models are typically trained on static, large-scale corpora, and their contextual assumptions are relatively fixed (e.g., a language model is based on a large-scale static text corpus, or a visual-language model is based on static image-text pairs). When the system needs to perform reasoning or generation in real-time interaction, continuous user feedback, environmental changes, or multi-turn dialogues, the model often fails to capture three types of factors well: changes in context over time, changes in user / environment behavior, and changes in modal state. As a result, the model's generation or understanding performance deteriorates, reducing the accuracy of predictions. Summary of the Invention
[0004] This application provides a language model training method, prediction method, apparatus, device, medium, and product to address the shortcomings of existing technologies where, when a system needs to perform reasoning or generation in real-time interaction, continuous user feedback, environmental changes, or multi-turn dialogues, the model often fails to capture three types of factors: context changes over time, user / environment behavior changes, and modal state changes. This results in a decline in the model's generation or understanding performance and reduced prediction accuracy. The application achieves this by training the language model using context change sequences, enabling the language model to continuously optimize in dynamic contexts such as interaction, user behavior, and environmental changes, thereby improving the target language model's generation or understanding capabilities and prediction accuracy.
[0005] In a first aspect, embodiments of this application provide a language model training method, comprising the following steps: The context change sequence is input into the context event detection sub-model in the initial language model to obtain the current context state output by the context event detection sub-model. The context change sequence is used to represent the changes in the user's scene. The current context state, the current modality and the historical modality corresponding to the context change sequence are input into the prediction sub-model in the initial language model to obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model. Based on the current context state and the first prediction result, the initial language model is trained to obtain the target language model.
[0006] In one embodiment, the prediction sub-model includes a modality co-evolution module and a language prediction module. The step of inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the prediction sub-model in the initial language model to obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model includes: inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the modality co-evolution module to obtain the target fusion feature of the context change sequence; and inputting the target fusion feature into the language prediction module to obtain the first prediction result corresponding to the context change sequence output by the language prediction module.
[0007] In one embodiment, the modal co-evolution module includes temporal encoding, modal attention, and cross-modal recoding. The step of inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the modal co-evolution module to obtain the target fusion feature of the context change sequence includes: inputting the historical modality into the temporal encoding to obtain temporally-based historical features corresponding to the historical modality; inputting the temporally-based historical features, the current context state, and the current modality into the modal attention to obtain initial fusion features; and inputting the initial fusion features into the cross-modal recoding to obtain the target fusion feature.
[0008] In one embodiment, training the initial language model based on the current context state and the first prediction result to obtain the target language model includes: determining a context recognition loss value based on the current context state and the context state label corresponding to the context change sequence; and determining a prediction loss value based on the first prediction result and the result label corresponding to the context change sequence. The initial language model is trained based on the context recognition loss value and the prediction loss value to obtain the target language model.
[0009] In one embodiment, the language prediction module is a large language sub-model, which is trained through the following steps: acquiring multimodal data; the multimodal data being used to characterize the data format; inputting the multimodal data into an initial large language sub-model to obtain a second prediction result corresponding to the multimodal data; and training the model parameters of the initial large language sub-model based on the second prediction result and the result label corresponding to the multimodal data to obtain the large language sub-model.
[0010] Secondly, embodiments of this application also provide a prediction method, comprising the following steps: Obtain the sequence of changes in the user-input context to be predicted; The predicted context change sequence is input into the target language model to obtain the third prediction result corresponding to the predicted context change sequence output by the target language model; wherein, the target language model is trained by the language model training method described in the first aspect above.
[0011] In one embodiment, after obtaining the third prediction result corresponding to the predicted context change sequence output by the target language model, the method further includes: obtaining user feedback on the third prediction result; and fine-tuning the target language model based on the feedback result to obtain an updated target language model.
[0012] Thirdly, embodiments of this application also provide a language model training device, comprising the following modules: The recognition module is used to input the context change sequence into the context event detection sub-model in the initial language model to obtain the current context state output by the context event detection sub-model. The context change sequence is used to characterize the changes in the user's scene. The first prediction module is used to input the current context state, the current modality and the historical modality corresponding to the context change sequence into the prediction sub-model in the initial language model, and obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model; The training module is used to train the initial language model based on the current context state and the first prediction result to obtain the target language model.
[0013] Fourthly, embodiments of this application also provide a prediction device, including the following modules: The acquisition module is used to acquire the sequence of changes in the context to be predicted input by the user; The second prediction module is used to input the context change sequence to be predicted into the target language model to obtain the third prediction result corresponding to the context change sequence to be predicted output by the target language model; wherein, the target language model is trained by the language model training method described in the first aspect above.
[0014] Fifthly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the language model training method as described in the first aspect or the prediction method as described in the second aspect.
[0015] Fifthly, embodiments of this application also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the language model training method as described in the first aspect or the prediction method as described in the second aspect.
[0016] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the language model training method as described in the first aspect or the prediction method as described in the second aspect.
[0017] This application provides a language model training method, prediction method, apparatus, device, medium, and product. The method involves inputting a context change sequence into a context event detection sub-model within an initial language model to obtain the current context state output by the context event detection sub-model. The context change sequence represents changes in the user's environment. The current context state, the current modality corresponding to the context change sequence, and the historical modality are then input into a prediction sub-model within the initial language model to obtain a first prediction result corresponding to the context change sequence output by the prediction sub-model. Based on the current context state and the first prediction result, the initial language model is trained to obtain a target language model. Thus, by training the language model using a context change sequence, the language model gains the ability to continuously optimize in dynamic contexts such as interactions, user behavior, and environmental changes, thereby improving the target language model's generation or understanding capabilities and prediction accuracy. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating the language model training method provided in this application.
[0020] Figure 2 This is a schematic diagram of the structure of the language model training system provided in this application.
[0021] Figure 3 This is a flowchart illustrating the prediction method provided in this application.
[0022] Figure 4 This is a schematic diagram of the structure of the language model training device provided in this application.
[0023] Figure 5 This is a schematic diagram of the predictive device provided in this application.
[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] For Natural Language Processing (NLP) tasks, models such as Generative Pre-trained Transformer 3 (GPT3) and eXtreme Language Understanding Network (XLNet) are trained on large-scale corpora and then fine-tuned for tasks such as classification, generation, and question answering. Rapid advancements are also occurring in multimodal domains (such as vision-language). These models can simultaneously process multiple modalities such as images, text, and video inputs, thereby improving understanding and generation capabilities. The model construction process is as follows: (1) Data collection and preprocessing: Collect large-scale corpora (text, images, videos, audio, etc.), and form a training set after deduplication, cleaning, and labeling (optional).
[0027] (2) Model structure design: Transformer architecture is usually adopted, such as encoder-only architecture, decoder-only architecture or encoder-decoder form, setting parameter scale, context window, attention mechanism, etc.
[0028] (3) Pre-training task design: such as language model (LM) task, masked language model (MLM), contrastive learning, cross-modal alignment task, etc.
[0029] (4) Training strategy: Use distributed large-scale computing resources and apply large-batch, long-sequence training, as well as optimization strategies such as learning rate scheduling, mixed precision, and gradient accumulation.
[0030] (5) Downstream adaptation or fine-tuning: After pre-training is completed, the specific task is adapted through fine-tuning or prompting (PromptEngineering).
[0031] While the modeling techniques described above have made significant progress in terms of generalization, modality fusion, and task adaptation, current large language models are typically trained on static, large-scale corpora, and their contextual assumptions are relatively fixed (e.g., a language model is based on a large-scale static text corpus, or a visual-language model is based on static image-text pairs). Current large language models are weak in adapting to multimodal dynamic contexts. When the system needs to reason or generate in real-time interaction, continuous user feedback, environmental changes, or multi-turn dialogues, the model often fails to capture the three types of factors: context changes over time, changes in user / environment behavior, and changes in modal state. This leads to a decline in modality generation or understanding performance and reduces the accuracy of predictions.
[0032] On the other hand, existing multimodal pre-training also has the following drawbacks: different modalities (text, images, videos, and sounds) are usually treated as parallel inputs or simply fused. In interactive or real-time systems (such as augmented reality, intelligent assistants, and robots), the lack of this context-modal co-evolution capability severely limits the applicability of the model; it requires huge computational resources (billions of parameters, large-scale tokens, and weeks of training time) and a large amount of static data, and has poor ability to adapt to specific changing contexts or real-time inputs.
[0033] To address the aforementioned problems, this application proposes a language model training method. By training the language model using contextual change sequences, the language model gains the ability to continuously optimize in dynamic contexts such as interactions, user behavior, and environmental changes, thereby improving the target language model's generation or understanding capabilities and prediction accuracy.
[0034] The following is combined with Figures 1 to 3 The language model training method described in this application is applicable not only to text generation but also to multimodal (text, image, video, speech) tasks, and is particularly suitable for interactive, real-time scenarios with frequent modal switching (such as AR glasses assistants, intelligent robots, multimodal customer service, etc.). The execution subject of this method can be an electronic device or a language model training method set in the electronic device. The language model training device can be implemented by software, hardware, or a combination of both.
[0035] Figure 1 This is a flowchart illustrating the language model training method provided in this application, as shown below. Figure 1 As shown, the method includes the following: Step 101: Input the context change sequence into the context event detection sub-model in the initial language model to obtain the current context state output by the context event detection sub-model.
[0036] The context change sequence is used to characterize changes in the user's environment.
[0037] Here, the contextual transformation sequence can include modality switching, user goal change, environmental scene change, etc. For example, the change from indoor to outdoor to car is a scene change, and the change from text to image to speech is a modality change.
[0038] Here, the context change sequence may include one or more context change sub-sequences, wherein the context change sequence may be obtained by manually labeling or simulating the collected multimodal data, and a context change sub-sequence is equivalent to a context change event (CSE).
[0039] Here, the contextual event detection sub-model can be a Transformer architecture network.
[0040] It should be noted that the mechanism involves inputting a sequence of contextual changes into the Context State Module (CSM) to identify contextual change events (such as modality switching, environmental changes, and changes in user goals) in the input and updating the contextual state accordingly.
[0041] Step 102: Input the current context state, the current modality and the historical modality corresponding to the context change sequence into the prediction sub-model in the initial language model to obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model.
[0042] Here, modality refers to different forms of input data, such as text, language, images, and videos. The current modality may contain only one modality or multiple modalities.
[0043] It should be noted that changes in context may also be accompanied by changes in modality.
[0044] Here, the first prediction result can be the query result, generated result, etc., output for the context change sequence.
[0045] Here, the prediction sub-model can be a Transformer architecture network.
[0046] Optionally, the prediction sub-model can be a single module or a combination of two modules.
[0047] Furthermore, the prediction sub-model includes a modality co-evolution module and a language prediction module. The step of inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the prediction sub-model in the initial language model to obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model includes: inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the modality co-evolution module to obtain the target fusion feature of the context change sequence; and inputting the target fusion feature into the language prediction module to obtain the first prediction result corresponding to the context change sequence output by the language prediction module.
[0048] Here, the Modality Evolution Network (MEN) is used to fuse the current context state, the current modality, and the historical modality to obtain the target fused features. Then, the target fused features are input into the language prediction module to obtain the first prediction result corresponding to the context change sequence.
[0049] Here, the modal co-evolution module and the language prediction module can be Transformer architecture networks.
[0050] In the embodiments of this application, compared with simple modal parallel input or light fusion, the modal co-evolution module establishes an evolutionary chain between contextual state, current modality, and historical modality, thereby improving the coherence and accuracy of cross-modal understanding and generation.
[0051] Here, the language prediction model can be a pre-trained large language model or a model downloaded from an open-source database.
[0052] For example, the language prediction module is a large language sub-model, which is trained through the following steps: acquiring multimodal data; the multimodal data is used to represent the format of the data; inputting the multimodal data into an initial large language sub-model to obtain a second prediction result corresponding to the multimodal data; and training the model parameters of the initial large language sub-model based on the second prediction result and the result label corresponding to the multimodal data to obtain the large language sub-model.
[0053] Here, multimodal data can include images, text, video, audio, etc., and can be data that is directly collected or downloaded, or data obtained by requesting data from collected or downloaded data.
[0054] The loss value of the model is determined based on the second prediction result and the corresponding label of the multimodal data. The model parameters of the initial large language sub-model are continuously updated based on the loss value until the loss value converges, thus obtaining the large language sub-model.
[0055] In this embodiment, a large language sub-model is statically pre-trained using multimodal data, enabling it to possess basic question-answering capabilities and providing a foundation for subsequent model training. Furthermore, the pre-training is divided into two stages: training the large language sub-model and training the target language model, thus reducing training costs.
[0056] Furthermore, the modal co-evolution module includes temporal encoding, modal attention, and cross-modal recoding. The step of inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the modal co-evolution module to obtain the target fusion feature of the context change sequence includes: inputting the historical modality into the temporal encoding to obtain the temporalized historical feature corresponding to the historical modality; inputting the temporalized historical feature, the current context state, and the current modality into the modal attention to obtain the initial fusion feature; and inputting the initial fusion feature into the cross-modal recoding to obtain the target fusion feature.
[0057] Here, temporal coding is the process of converting the temporal dimension information of temporal data corresponding to a modality into a feature vector, extracting the dependencies of historical modality data, and obtaining temporally sequenced historical features.
[0058] Modal attention refers to mapping temporal historical features, current context state, and current modality to a unified feature space, generating query (Q), key (K), and value (V) vectors for each modality, calculating the attention score between the current modality and the historical modality, and then weighting and summing the attention scores for each modality to obtain the initial fused features.
[0059] Cross-modal recoding can be understood as performing secondary encoding on the initial fused features, extracting secondary features from the initial fused features, aligning the semantic spaces of different modalities, and eliminating interference caused by modal heterogeneity.
[0060] In the embodiments of this application, an evolutionary chain is established between the context state, the current modality, and the possible historical modalities through three sub-modules: temporal encoding, modal attention, and cross-modal recoding. This improves the coherence and accuracy of cross-modal understanding and generation, thereby enhancing the accuracy of the target language model's reasoning.
[0061] Step 103: Based on the current context state and the first prediction result, train the initial language model to obtain the target language model.
[0062] It should be noted that the training process involves determining the loss value based on the current context state and the first prediction result. The model updates its parameters using gradient descent based on the loss value until the loss value converges, thus obtaining the target language model. Generally, the smaller the loss value, the better the performance of the target language model; conversely, the larger the loss value, the worse the performance of the target language model.
[0063] In the embodiments of this application, the objectives of training the target language model may include language modeling, cross-modal prediction, contextual state prediction, and modality switching prediction.
[0064] Further, the step of training the initial language model based on the current context state and the first prediction result to obtain the target language model includes: determining a context recognition loss value based on the current context state and the context state labels corresponding to the context change sequence; determining a prediction loss value based on the first prediction result and the result labels corresponding to the context change sequence; and training the initial language model based on the context recognition loss value and the prediction loss value to obtain the target language model.
[0065] Here, the context state label is the real context state corresponding to the context change sequence, and the current context state is the context state identified by the context recognition model.
[0066] Here, the context recognition loss value refers to the difference between the current context state and the context state label. The larger the difference value, the larger the loss value and the worse the recognition performance of the context event detection sub-model; the smaller the difference, the smaller the loss value and the better the performance of the context event detection sub-model.
[0067] Here, the result label corresponding to the context change sequence is actually the true result corresponding to the context change sequence. The prediction loss value refers to the difference between the predicted result and the result label. The larger the difference value, the larger the loss value and the worse the prediction performance of the prediction sub-model; the smaller the difference, the smaller the loss value and the better the prediction performance of the prediction sub-model.
[0068] Optionally, the methods for determining the context loss value and the predicted loss value can use loss functions, including but not limited to cross-entropy loss function, mean absolute error loss function, mean squared error loss function, etc.
[0069] Optionally, the contextual event detection sub-model and the prediction sub-model in the initial language model can be trained using the contextual recognition loss value and the prediction loss value respectively, or the parameters in the initial language model can be trained by combining the sum of the contextual recognition loss value and the prediction loss value.
[0070] In this embodiment of the invention, the target language model is trained by combining context recognition loss and prediction loss, enabling the model to recognize and adapt to context changes, thereby performing better in complex interactive scenarios. The pre-training is divided into two stages: training the large language sub-model and the target language model, which reduces the training cost.
[0071] Figure 2 This is a schematic diagram of the language model training system provided in this application, used to train the language model, such as... Figure 2 As shown, the system includes a data lake, a data processing module, a contextual event detection sub-model, a modal co-evolution module, a main pre-trained model (i.e., the aforementioned language prediction module), an online update module, and a downstream task adaptation module. The data processing module is responsible for multimodal data cleaning and contextual event annotation or simulation. The contextual event detection sub-model identifies contextual change events in the input (such as modal switching, user behavior changes, environmental changes, etc.) and labels them as CSEs. The modal co-evolution module fuses the current modal representation, historical modal representation, and contextual state representation into a target fusion feature. The main pre-trained model receives the target fusion feature (fused_rep) based on the Transformer architecture and generates the output. The online update module monitors runtime input / feedback and triggers incremental learning, i.e., fine-tuning the model. The downstream adaptation module applies the trained model to specific tasks (such as dialogue generation, scene understanding, multimodal control, etc.).
[0072] Specifically, the data processing module acquires multimodal data from the data lake, cleans and labels the multimodal data with context change sequences, inputs the context change sequences into the context event detection sub-model, identifies the context change sequences, and outputs the current context state. The current context state, the current modality corresponding to the context change sequence, and the historical modality are input into the modality co-evolution module. The modality co-evolution module outputs target fusion features, inputs the target fusion features into the main pre-trained model, obtains the prediction results corresponding to the context change sequences, updates the model parameters of the language model based on the prediction results and the true result labels, obtains the language model, and uses the downstream task adaptation module to deploy the language model on the terminal, monitors user feedback results, inputs the feedback results into the online update module, and fine-tunes the target language model.
[0073] Figure 3 This is a flowchart illustrating the prediction method provided in this application, as shown below. Figure 3 As shown, the method includes the following: Step 301: Obtain the user-inputted sequence of context changes to be predicted.
[0074] Optionally, users can input information through an AR assistant device or cloud service, and then the target language model can determine the sequence of context changes to be predicted based on the input information; users can also directly input the sequence of context changes to be predicted.
[0075] Step 302: Input the predicted context change sequence into the target language model to obtain the third prediction result corresponding to the predicted context change sequence output by the target language model.
[0076] The target language model is trained using any of the language model training methods described above.
[0077] It should be noted that after the target language model is trained, it can be deployed on AR assistant devices or cloud services. The target language model receives the sequence of contextual changes to be predicted corresponding to the user's voice input or text input information, and directly generates a third prediction result.
[0078] In the embodiments of this application, by using the target language model to understand contextual changes and quickly respond to modality switching, a more natural, efficient, and intelligent service can be provided. For example, it can continue to understand and respond when the user switches from voice to image input or when the environment changes, thereby improving the smoothness of interaction.
[0079] Furthermore, after obtaining the third prediction result corresponding to the predicted context change sequence output by the target language model, the method further includes: obtaining user feedback on the third prediction result; and fine-tuning the target language model based on the feedback result to obtain an updated target language model.
[0080] It should be noted that the target language model needs to be deployed in an actual system to monitor context change events in the input. The system records CSE and receives user feedback on the prediction results of the context change sequence. The context change sequence is used to perform a small amount of real-time fine-tuning or incremental training on the context event detection sub-model and modality co-evolution, so that the model can continuously adapt to the new context-modality input pairs corresponding to the context change as online incremental data.
[0081] In the embodiments of this application, during the application phase, a small amount of real-time fine-tuning or incremental training is performed based on contextual event records and user feedback data, so that the model can continuously adapt to contextual changes, avoiding the problems of high retraining cost and slow adaptation of traditional models.
[0082] The following are specific application scenarios of the target language model in this application, including: Initialization phase: During system deployment, the pre-trained model (including context state module, modal evolution network, and main model) is loaded into the AR assistant device or cloud service.
[0083] A user enters a home kitchen: User: "Could you check the nutritional information of this vegetable?" The user inputs via voice and takes an image of the vegetable using the AR glasses' camera. The input modality is voice + image. The system detects the "voice → image" modal switch, triggering a CSE event. The Contextual State Module (CSM) updates the state from "casual conversation / idle" to "kitchen cooking scenario." The Modal Evolution Network (MEN) integrates voice representation, image representation, and contextual state. The main model M outputs: "This is broccoli, rich in Vitamin C and dietary fiber..."
[0084] The user suddenly decides to go out: After getting ready, the user leaves the kitchen and enters the in-car commuting environment. The system detects changes in camera / geometry / in-car Bluetooth input, recognizing the "environment change: home → in-car" triggering a CSE event. The context status is updated to "in-car commuting scenario." The user says via voice: "Help me find dinner ordering options after get off work and plan my route." The current input modality is voice + map view. MEN is fused, and M generates reasonable planning suggestions. The system also logs this context change to the online update module.
[0085] Online incremental adaptation: During the user's commuting phase described above, the system records a small amount of user feedback (user's choice to accept / reject suggestions, and subsequent actions) to constitute online incremental data. The online update module periodically triggers fine-tuning, updating the MEN and CSM parameters to better adapt the model to the user's preferences in the "in-car commuting" context, making suggestions more accurate in similar contexts in the future.
[0086] Scene transition to outdoor walking: The user gets out of the car and begins walking to the meeting room. The system recognizes the "car → walking environment" modality and the environmental change, triggering CSE. The context state is updated to "outdoor walking scene," and the input modality may be voice + camera + walking sensor. MEN fuses the multimodal input, and the main model M generates based on the new context: "It is recommended that you walk along Red Street for 6 minutes, and pay attention to the sculpture on your right along the way..."
[0087] Downstream Task Adaptation: The above model can also be used in dialogue systems, robot navigation, multimodal customer service, smart home control, etc. Taking a customer service system as an example: A user first enters text into a customer service chat, then switches to uploading a photo of a damaged device. The system detects the text-to-image modal switch, triggering CSE (Contextual Context Analysis), and the context state changes from "text consultation" to "image fault diagnosis." Model M uses MEN (Model Contextual Context) to fuse the two modal information, generating a preliminary fault diagnosis and suggesting a repair solution. This is updated online based on customer feedback.
[0088] The language model training device and prediction device provided in this application are described below. The language model training device and prediction device described below can be referred to in correspondence with the language model training method and prediction device described above.
[0089] Figure 4 This is a schematic diagram of the structure of the language model training device provided in this application, such as... Figure 4 As shown, the language model training device 400 includes the following: The recognition module 410 is used to input the context change sequence into the context event detection sub-model in the initial language model to obtain the current context state output by the context event detection sub-model. The context change sequence is used to characterize the changes in the user's scene. The first prediction module 420 is used to input the current context state, the current mode and the historical mode corresponding to the context change sequence into the prediction sub-model in the initial language model, and obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model; Training module 430 is used to train the initial language model based on the current context state and the first prediction result to obtain the target language model.
[0090] In this embodiment of the application, the prediction sub-model includes a modality co-evolution module and a language prediction module. The first prediction module 420 is specifically used to: input the current context state, the current modality and the historical modality corresponding to the context change sequence into the modality co-evolution module to obtain the target fusion feature of the context change sequence; input the target fusion feature into the language prediction module to obtain the first prediction result corresponding to the context change sequence output by the language prediction module.
[0091] In this embodiment, the modal co-evolution module includes temporal encoding, modal attention, and cross-modal recoding. The first prediction module 420 is further specifically used for: inputting the historical modality into the temporal encoding to obtain temporalized historical features corresponding to the historical modality; inputting the temporalized historical features, the current context state, and the current modality into the modal attention to obtain initial fusion features; and inputting the initial fusion features into the cross-modal recoding to obtain the target fusion features.
[0092] In this embodiment of the application, the first prediction module 420 is specifically used to: determine a context recognition loss value based on the current context state and the context state label corresponding to the context change sequence; determine a prediction loss value based on the first prediction result and the result label corresponding to the context change sequence; and train the initial language model based on the context recognition loss value and the prediction loss value to obtain the target language model.
[0093] In this embodiment, the language prediction module is a large language sub-model, and the language model training device 400 further includes a training module, specifically used for: acquiring multimodal data; the multimodal data being used to characterize the data format; inputting the multimodal data into an initial large language sub-model to obtain a second prediction result corresponding to the multimodal data; and training the model parameters of the initial large language sub-model based on the second prediction result and the result label corresponding to the multimodal data to obtain the large language sub-model.
[0094] Figure 5 This is a schematic diagram of the predictive device provided in this application, as shown below. Figure 5 As shown, the prediction device 500 includes the following: The acquisition module 510 is used to acquire the sequence of changes in the context to be predicted input by the user; The second prediction module 520 is used to input the context change sequence to be predicted into the target language model to obtain the third prediction result corresponding to the context change sequence to be predicted output by the target language model; wherein, the target language model is trained by the language model training method described above.
[0095] In this embodiment of the application, after obtaining the third prediction result corresponding to the predicted context change sequence output by the target language model, the prediction device 500 further includes an update module, specifically used for: obtaining user feedback on the third prediction result; and fine-tuning the target language model based on the feedback result to obtain an updated target language model.
[0096] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute steps of a language model training method and / or a prediction method. The language model training method includes: inputting a context change sequence into a context event detection sub-model in an initial language model to obtain the current context state output by the context event detection sub-model, wherein the context change sequence is used to characterize changes in the user's scene; inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into a prediction sub-model in the initial language model to obtain a first prediction result corresponding to the context change sequence output by the prediction sub-model; and training the initial language model based on the current context state and the first prediction result to obtain a target language model.
[0097] And / or, The prediction method includes: acquiring a user-inputted sequence of context changes to be predicted; inputting the sequence of context changes to be predicted into a target language model to obtain a third prediction result corresponding to the sequence of context changes to be predicted output by the target language model; wherein the target language model is trained by the language model training method described above.
[0098] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0099] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the language model training method and / or prediction method provided by the above methods. The language model training method includes: inputting a context change sequence into a context event detection sub-model in an initial language model to obtain the current context state output by the context event detection sub-model, wherein the context change sequence is used to characterize the changes in the user's scene; inputting the current context state, the current modality and the historical modality corresponding to the context change sequence into a prediction sub-model in the initial language model to obtain a first prediction result corresponding to the context change sequence output by the prediction sub-model; and training the initial language model based on the current context state and the first prediction result to obtain a target language model.
[0100] And / or, The prediction method includes: acquiring a user-inputted sequence of context changes to be predicted; inputting the sequence of context changes to be predicted into a target language model to obtain a third prediction result corresponding to the sequence of context changes to be predicted output by the target language model; wherein the target language model is trained by the language model training method described above.
[0101] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements steps for performing the language model training method and / or prediction method provided by the above methods. The language model training method includes: inputting a context change sequence into a context event detection sub-model in an initial language model to obtain the current context state output by the context event detection sub-model, wherein the context change sequence is used to characterize changes in the user's scenario; inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into a prediction sub-model in the initial language model to obtain a first prediction result output by the prediction sub-model corresponding to the context change sequence; and training the initial language model based on the current context state and the first prediction result to obtain a target language model.
[0102] And / or, The prediction method includes: acquiring a user-inputted sequence of context changes to be predicted; inputting the sequence of context changes to be predicted into a target language model to obtain a third prediction result corresponding to the sequence of context changes to be predicted output by the target language model; wherein the target language model is trained by the language model training method described above.
[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A language model training method, characterized in that, include: The context change sequence is input into the context event detection sub-model in the initial language model to obtain the current context state output by the context event detection sub-model. The context change sequence is used to represent the changes in the user's scene. The current context state, the current modality and the historical modality corresponding to the context change sequence are input into the prediction sub-model in the initial language model to obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model. Based on the current context state and the first prediction result, the initial language model is trained to obtain the target language model.
2. The language model training method according to claim 1, characterized in that, The prediction sub-model includes a modality co-evolution module and a language prediction module. The process involves inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the prediction sub-model within the initial language model to obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model, including: The current context state, the current mode and the historical mode corresponding to the context change sequence are input into the mode co-evolution module to obtain the target fusion feature of the context change sequence; The target fusion features are input into the language prediction module to obtain the first prediction result corresponding to the context change sequence output by the language prediction module.
3. The language model training method according to claim 2, characterized in that, The modal co-evolution module includes temporal encoding, modal attention, and cross-modal recoding. The step of inputting the current context state, the current modality corresponding to the context change sequence, and the historical modality into the modal co-evolution module to obtain the target fusion features of the context change sequence includes: The historical modes are input into the temporal encoding to obtain the temporalized historical features corresponding to the historical modes; The temporal historical features, the current context state, and the current modality are input into the modality attention to obtain the initial fusion features; The initial fusion features are input into the cross-modal recoding to obtain the target fusion features.
4. The language model training method according to any one of claims 1 to 3, characterized in that, The step of training the initial language model based on the current context state and the first prediction result to obtain the target language model includes: Based on the current context state and the context state labels corresponding to the context change sequence, the context recognition loss value is determined; Based on the first prediction result and the result label corresponding to the context change sequence, the prediction loss value is determined; The initial language model is trained based on the context recognition loss value and the prediction loss value to obtain the target language model.
5. The language model training method according to claim 2, characterized in that, The language prediction module is a large language sub-model, which is trained through the following steps: Acquire multimodal data; the multimodal data is used to characterize the data format; The multimodal data is input into the initial large language sub-model to obtain the second prediction result corresponding to the multimodal data; Based on the second prediction result and the result labels corresponding to the multimodal data, the model parameters of the initial large language sub-model are trained to obtain the large language sub-model.
6. A prediction method, characterized in that, include: Obtain the sequence of changes in the user-input context to be predicted; The predicted context change sequence is input into the target language model to obtain the third prediction result corresponding to the predicted context change sequence output by the target language model; wherein, the target language model is trained by the language model training method of any one of claims 1-5.
7. The prediction method according to claim 6, characterized in that, After obtaining the third prediction result corresponding to the predicted context change sequence output by the target language model, the method further includes: Obtain user feedback on the third prediction result; The target language model is fine-tuned based on the feedback results to obtain an updated target language model.
8. A language model training device, characterized in that, include: The recognition module is used to input the context change sequence into the context event detection sub-model in the initial language model to obtain the current context state output by the context event detection sub-model. The context change sequence is used to characterize the changes in the user's scene. The first prediction module is used to input the current context state, the current modality and the historical modality corresponding to the context change sequence into the prediction sub-model in the initial language model, and obtain the first prediction result corresponding to the context change sequence output by the prediction sub-model; The training module is used to train the initial language model based on the current context state and the first prediction result to obtain the target language model.
9. A prediction device, characterized in that, include: The acquisition module is used to acquire the sequence of changes in the context to be predicted input by the user; The second prediction module is used to input the context change sequence to be predicted into the target language model to obtain the third prediction result corresponding to the context change sequence to be predicted output by the target language model; wherein the target language model is trained by the language model training method of any one of claims 1-5.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the language model training method as described in any one of claims 1 to 5; or the prediction method as described in claim 6 or 7.
11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the language model training method as described in any one of claims 1 to 5; or the prediction method as described in claim 6 or 7.
12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the language model training method as described in any one of claims 1 to 5; or the prediction method as described in claim 6 or 7.