Real-time migration adaptive learning method and system for multi-modal intelligent agent

Abstract

The invention relates to a real-time migration adaptive learning method and system for a multi-modal agent. The method comprises the following steps: carrying out dynamic state encoding on a multi-dimensional time sequence data stream through a continuous time encoder; carrying out sequence splicing on the knowledge text obtained by retrieval, the received natural language instruction and the semantic time sequence characteristics converted by the time sequence state representation sequence; performing fusion reasoning on the spliced unified modal input sequence, and outputting a structured decision result; external correction feedback is captured, a multi-task loss function is constructed according to the difference between the external correction feedback and the structured decision result, and a heuristic weight factor is introduced to obtain a final optimization target; and performing gradient back propagation on the final optimization target, and selectively acting gradient signals on parameters of the continuous time encoder and the semantic mapping network to complete parameter updating. According to the method, efficient online learning of the electric power intelligent agent is realized, and the calculation cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a real-time transfer adaptive learning method and system for multimodal intelligent agents. Background Technology

[0002] With the rapid development of multimodal general-purpose agents (GMAs), their application potential in complex industrial scenarios such as power dispatching automation and operation and maintenance is becoming increasingly prominent. To enable agents to understand and respond to dynamic digital information from the power grid, existing technologies widely employ neural network models based on discrete time steps, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and the Transformer architecture. These models have achieved great success in processing standard serialized data; however, their underlying mechanism relies on dividing time into fixed, discrete step sizes, which fundamentally contradicts the inherent continuous, asynchronous, and non-uniform sampling characteristics of power system sensor data streams. This discretization approximation often loses crucial dynamic information when facing sudden faults or requiring refined trend prediction, limiting the agent's ability to accurately model the dynamics of complex power systems causally.

[0003] To overcome the limitations of discrete models, existing technologies employ continuous-time models, with representative techniques including Neural Ordinary Differential Equations (Neural ODEs) and their derivative, Liquid Neural Networks (LNNs). These models describe the evolution of neural networks as continuous differential equations, capable of capturing the dynamic laws of the physical world with higher fidelity, and are theoretically ideal for processing complex dynamic time-series data such as power systems. By treating the system state as a continuous trajectory, they can naturally handle irregularly sampled data and provide stronger robustness and interpretability.

[0004] However, despite the significant theoretical advantages of continuous-time models, they face severe computational performance bottlenecks in practical applications, especially in power operation and maintenance scenarios requiring real-time responses. These ODE-based models heavily rely on iterative, computationally expensive numerical differential equation (DE) solvers for forward inference and backward gradient calculation. When processing long sequences of data or demanding high precision, the computational overhead of the solver grows exponentially. This limitation prevents real-time online learning loops, making it difficult for the agent to quickly adjust the model based on new operation and maintenance instructions or fault samples after deployment, resulting in a static knowledge dilemma. Consequently, existing technologies are inefficient and costly. Furthermore, the exponentially increasing overhead in current learning methods can lead to storage space explosion, learning failures, low stability, and long processing times. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problems of easy learning failure, resulting in poor stability and low efficiency in the prior art, and thus provide a real-time transfer adaptive learning method and system for multimodal intelligent agents with high stability and learning efficiency.

[0006] To address the aforementioned technical problems, this invention provides a real-time transfer adaptive learning method for multimodal intelligent agents, comprising:

[0007] Step S1: Dynamically encode the multidimensional time-series data stream using a continuous-time encoder to obtain a time-series state representation sequence;

[0008] Step S2: Based on the semantic mapping network, the retrieved knowledge text, the received natural language instructions, and the semantic temporal features based on the temporal state representation sequence transformation are concatenated to obtain a unified modality input sequence.

[0009] Step S3: Input the unified modal input sequence into the language reasoning model, perform cross-modal information fusion reasoning through the self-attention mechanism, and output structured decision results;

[0010] Step S4: Capture external correction feedback for the structured decision result, construct a multi-task loss function based on the difference between the external correction feedback and the structured decision result, and weight the multi-task loss function using heuristic weighting factors to obtain the final optimization objective;

[0011] Step S5: Perform gradient backpropagation on the final optimization objective, and selectively apply the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network to complete the parameter update; wherein, the parameters of the language inference model remain frozen during the parameter update process.

[0012] In one embodiment of the present invention, the method for obtaining the time-series state characterization sequence in step S1 includes: step S11: obtaining a multi-dimensional time-series data stream within a time window, wherein the multi-dimensional time-series data stream contains multiple types of measurement indicators from multiple devices;

[0013] Step S12: Dynamic state encoding of the multidimensional time-series data stream through a continuous-time encoder includes: processing the multidimensional time-series data stream step by step through a closed continuous-time neural network, calculating the neuron state update based on the closed-form solution of the liquid time constant differential equation, and obtaining the time-series state representation sequence, wherein each time-series state representation corresponds to a dynamic state encoding of one time step.

[0014] In one embodiment of the present invention, in step S12, the expression for the state update of the closed continuous-time neural network is: in, For time step index, For the first Neuron states at each time step For the first Input data at each time step, This is the set of trainable parameters for a closed continuous-time neural network. This is the state transition function based on the closed solution of the differential equation with liquid time constant.

[0015] In one embodiment of the present invention, the method for obtaining the unified modal input sequence in step S2 is as follows:

[0016] S21: Input the sequence of temporal state representations into a semantic mapping network, and map each temporal state representation to a semantic vector space compatible with the language reasoning model through linear projection transformation to obtain the semantic temporal features;

[0017] S22: Using natural language commands as query vectors, retrieve multiple knowledge text fragments from the knowledge base based on the similarity of the retrieved texts.

[0018] S23: The natural language instructions, multiple knowledge text fragments, and the semantic temporal features are concatenated to form a unified modal input sequence.

[0019] In one embodiment of the present invention, when the temporal state representation sequence is input into the semantic mapping network, the network structure is a single fully connected linear layer containing a weight matrix and a bias vector. Then, the dimension of the temporal state representation is recorded as the original dimension, and the hidden layer dimension of the language inference model is recorded as the target dimension. The purpose of the semantic mapping network is to project the state representation from the original dimension space to the target dimension space.

[0020] In one embodiment of the present invention, step S4, the method for obtaining the final optimization objective, includes:

[0021] Step S41: Receive correction feedback data for the structured decision results;

[0022] Step S42: For the diagnostic task corresponding to the diagnostic conclusion of the structured decision result, calculate the distance metric between the diagnostic conclusion and the corrected diagnostic conclusion in the semantic space using a contrastive learning loss strategy to obtain a diagnostic loss term; for the risk assessment task corresponding to the risk score of the structured decision result, calculate the mean square error between the risk score and the corrected risk score to obtain a risk loss term; for the scheme generation task corresponding to the processing scheme of the structured decision result, calculate the cosine similarity loss between the sentence vector representations of the processing scheme and the corrected processing scheme to obtain a scheme loss term.

[0023] Step S43: Perform a weighted summation of the diagnostic loss term, the risk loss term, and the solution loss term to obtain the multi-task loss function;

[0024] Step S44: Calculate the heuristic weighting factor based on the correction magnitude of the correction feedback data;

[0025] Step S45: Multiply the heuristic weighting factor by the multi-task loss function to obtain the final optimization objective.

[0026] In one embodiment of the present invention, the expression for the heuristic weighting factor is: in, As a heuristic weighting factor, For indicator functions, This is the revised diagnostic conclusion. It is an empty set. The risk score output by the model, For the revised risk score, The preset first weighting coefficient, This is the preset second weighting coefficient.

[0027] In one embodiment of the present invention, the expression for the final optimization objective is: in, For the ultimate optimization goal, For multi-task loss function, To diagnose the loss, The weighting coefficients corresponding to the diagnostic loss. For risk loss, The weighting coefficients corresponding to risk losses. For the loss of the plan, These are the weighting coefficients corresponding to the loss of the proposed solution.

[0028] In one embodiment of the present invention, in step S5, the method of selectively applying the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network is as follows: the gradient signal starts from the final optimization target, passes through the output layer of the language inference model, and reaches the unified modality input sequence. At the position of the unified modality input sequence, the gradient splits into two paths: the gradient at the text word embedding position terminates its propagation due to the freezing of the language inference model, while the gradient at the semantic temporal feature position continues to propagate forward into the semantic mapping network; subsequently, the gradient calculates the partial derivatives of the weight matrix and bias vector in the semantic mapping network to obtain the gradient value of the semantic mapping network parameters and stores it; the gradient continues to propagate from the semantic mapping network to the temporal state representation sequence, and then enters the continuous-time encoder, where the partial derivatives of the weight matrix involved in the state transition function are calculated to obtain the gradient value of the continuous-time encoder parameters and stores it.

[0029] The present invention also provides a real-time transfer adaptive learning system for multimodal agents, comprising:

[0030] The encoding module is used to dynamically encode the state of a multidimensional time-series data stream using a continuous-time encoder to obtain a time-series state representation sequence.

[0031] The splicing module is configured as a semantic mapping network to splice the retrieved knowledge text, the received natural language instructions, and the semantic temporal features based on the temporal state representation sequence transformation to obtain a unified modality input sequence.

[0032] The reasoning module is configured as a language reasoning model to receive the unified modal input sequence, perform fusion reasoning on cross-modal information through a self-attention mechanism, and output structured decision results.

[0033] A construction module is used to capture external correction feedback for the structured decision result, construct a multi-task loss function based on the difference between the external correction feedback and the structured decision result, and weight the multi-task loss function through heuristic weighting factors to obtain the final optimization objective;

[0034] The training module is used to perform gradient backpropagation on the final optimization objective, selectively applying the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network to complete the parameter update; wherein, the parameters of the language inference model remain frozen during the parameter update process.

[0035] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0036] The real-time transfer adaptive learning method and system for multimodal intelligent agents described in this invention uses a continuous-time encoder to dynamically encode the state of multidimensional time-series data streams and generate a time-series state representation sequence. This can accurately characterize the evolution trajectory of equipment states in the power system in the form of continuous differential equations. It effectively solves the information loss problem in traditional discrete-time models when dealing with non-uniformly sampled and asynchronously arriving sensor data, making the capture of dynamic features of sudden faults more accurate and providing a high-fidelity time-series feature foundation for subsequent causal inference.

[0037] Secondly, this invention transforms the temporal state representation sequence into semantic temporal features through a semantic mapping network and concatenates them with natural language instructions and knowledge text to form a unified modal input sequence. This achieves deep fusion of heterogeneous data at the input level, enabling the self-attention mechanism of the language reasoning model to establish the connection between temporal dynamic information and textual semantic information at the lowest level. This avoids the modal fragmentation phenomenon caused by simple back-end concatenation in traditional multimodal fusion methods and significantly enhances the model's ability to understand cross-modal causal relationships in complex industrial scenarios.

[0038] Furthermore, this invention captures external correction feedback and constructs a multi-task loss function, combining heuristic weighting factors to differentiate feedback of varying importance. This enables the model to autonomously identify key learning samples and assign them higher learning priority, effectively improving the efficiency and relevance of online learning and avoiding the waste of learning resources caused by treating all feedback equally in traditional methods. In addition, this invention uses a gradient-selective propagation mechanism to update only the parameters of the continuous-time encoder and semantic mapping network while keeping the language inference model parameters frozen, reducing the parameter space of online learning and shortening the computation time for model updates, thus achieving real-time response. At the same time, by retaining the vast knowledge base of the pre-trained language model, it effectively prevents catastrophic forgetting during continuous learning, ensuring that the model does not lose its original general reasoning ability while continuously learning new knowledge.

[0039] In addition, this invention further stabilizes the online learning process by smoothing the parameter update amount through a moving average strategy and establishing a replay mechanism for high-value experience samples, reducing the risk of model fluctuations caused by single noise feedback, and consolidating core knowledge by periodically reviewing key cases, enabling the agent to maintain a stable performance improvement trajectory in a complex and ever-changing operation and maintenance environment, and ultimately realizing a fundamental transformation from a static knowledge base to a continuously evolving agent. Attached Figure Description

[0040] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0041] Figure 1 This is a schematic diagram of the real-time transfer adaptive learning method for multimodal intelligent agents as described in Embodiment 1 of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of the real-time transfer adaptive learning system for multimodal intelligent agents as described in Embodiment 2 of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0044] Example 1:

[0045] Reference Figure 1 As shown, this invention provides a real-time transfer adaptive learning method for multimodal intelligent agents, comprising:

[0046] Step S1: Dynamically encode the multidimensional time-series data stream using a continuous-time encoder to obtain a time-series state representation sequence.

[0047] In step S1, the method for obtaining the temporal state representation sequence includes:

[0048] Step S11: Obtain a multi-dimensional time-series data stream within the time window, wherein the multi-dimensional time-series data stream contains multiple types of measurement indicators from multiple devices.

[0049] In a specific embodiment, a power system intelligent agent is taken as an example. In step S11, the present invention obtains a multi-dimensional time-series data stream within a time window from the data acquisition layer of the power system. The data stream originates from sensors distributed within the substation, including voltage and current measurements of circuit breakers, oil temperature, winding temperature, load rate of transformers, and status signals of switching equipment. However, the obtained data has heterogeneous characteristics, manifested in inconsistent sampling frequencies (current data is sampled every second while temperature data is sampled every 5 seconds); asynchronous data arrival times, with network transmission delays for data packets from different devices; and significant differences in data dimensions (circuit breakers contain six channels for three-phase voltage and current, while transformers contain more than ten monitoring indicators such as temperature, pressure, and oil level). After data acquisition, the present invention organizes the raw data into a structured time-series data stream. Each data point includes a device identifier, indicator type, value, and timestamp. Within the time window, the present invention sorts all indicators of all devices according to the timestamp, forming an ordered sequence containing multiple data points. For indicators that have not been sampled at a certain moment, a forward filling strategy is adopted, using the most recent valid observation value of the indicator to fill the gap.

[0050] Step S12: Dynamic state encoding of the multidimensional time-series data stream through a continuous-time encoder includes: processing the multidimensional time-series data stream step by step through a closed continuous-time neural network, calculating the neuron state update based on the closed-form solution of the liquid time constant differential equation, and obtaining the time-series state representation sequence, wherein each time-series state representation corresponds to a dynamic state encoding of one time step.

[0051] In step S12, the present invention inputs the acquired multidimensional time-series data stream into a closed continuous-time neural network step by step. The closed continuous-time neural network performs mathematical transformation on the liquid time constant differential equation to obtain a closed-form solution. The differential equation describes the continuous evolution of the neuron state over time. The right side of the equation contains a decay term controlled by the time constant and an activation term composed of a weight matrix and a nonlinear activation function.

[0052] Traditional neural networks often require iterative numerical solvers to discretize differential equations, necessitating multiple iterations at each time step. This invention approximates the analytical solution of the differential equation as an explicit algebraic expression, namely a state transition function. This function receives the neuron's state from the previous time step and the current input data, directly calculating and outputting the current neuron state, requiring only one forward propagation without iteration.

[0053] Specifically, the state transition function first multiplies the input with a weight matrix to obtain the input contribution, then multiplies the previous state with another weight matrix to obtain the state contribution. A nonlinear activation function is applied to the sum of these two contributions to obtain the activation value. Finally, the activation value is weighted by the weight matrix and fused with the decayed previous state. This invention repeatedly performs state update calculations at each time step within the time window, progressively calculating a temporal state representation sequence starting from the initial state. Each state representation in the sequence is a fixed-dimensional real-number vector, encoding the dynamic evolution information of all historical input data from the starting point to the current time.

[0054] Based on the CfC network provided by the present invention described above, the core is to approximately solve the closed-form solution of the following differential equation of the liquid time constant (LTC), expressed as:

[0055] in, It is a neuronal state. It is the input of the CfC network. Given multiple network parameters, the CfC network passes through a closed-form solution. By directly calculating the state update, we can obtain the following expression for the state update of a closed continuous-time neural network.

[0056] In step S12, the expression for the state update of the closed continuous-time neural network is:

[0057] in, For time step index, For the first Neuron states at each time step For the first Input data at each time step, This is the set of trainable parameters for a closed continuous-time neural network. This is the state transition function based on the closed solution of the differential equation with liquid time constant.

[0058] Step S2: Based on the semantic mapping network, the retrieved knowledge text, the received natural language instructions, and the semantic temporal features based on the temporal state representation sequence transformation are concatenated to obtain a unified modality input sequence.

[0059] The method for obtaining the unified modal input sequence in step S2 is as follows:

[0060] Step S21: Input the temporal state representation sequence into the semantic mapping network, and map each temporal state representation to a semantic vector space compatible with the language reasoning model through linear projection transformation to obtain the semantic temporal features.

[0061] In step S21, the generated temporal state representation sequence is input into a semantic mapping network. This network is a single-layer fully connected linear layer containing a weight matrix and a bias vector. The dimension of the temporal state representation is then denoted as the original dimension, and the hidden layer dimension of the language inference model is denoted as the target dimension. The purpose of the semantic mapping network is to project the state representation from the original dimension space to the target dimension space. Specifically, the projection transformation is completed through matrix multiplication and bias addition. The present invention performs projection transformation independently on each temporal state representation in the sequence, and sequentially calculates to form a semantic temporal feature sequence. The numerical range and distribution characteristics of each dimension value of the projected vector are consistent with the text word vectors processed by the language inference model. This is because the parameters of the semantic mapping network are optimized through a multimodal alignment loss function during the pre-training stage. The alignment loss requires that the projected vector of the temporal event be as close as possible to the semantic vector of the text sentence describing the event under the cosine similarity metric. Finally, through alignment training, the dynamic pattern information in the temporal state representation is mapped to the corresponding position in the text semantic space.

[0062] Step S22: Using natural language commands as query vectors, retrieve multiple knowledge text fragments from the knowledge base based on the similarity of the retrieved data.

[0063] Further, in step S22, the present invention first receives a natural language instruction describing the query request input by the maintenance personnel. The present invention calls a pre-trained text encoding model to encode the instruction. The encoding model adopts a Transformer encoder structure to convert the text string into a fixed-dimensional sentence vector representation. In the knowledge base, a large number of fault case texts, equipment manual entries, maintenance procedure fragments, and other knowledge texts are pre-stored. Each knowledge text is converted into a sentence vector and a vector index is established offline using the same text encoding model. The present invention uses the query sentence vector to perform similarity retrieval in the knowledge base vector index. The similarity calculation uses cosine similarity; a similarity score is obtained by calculating the cosine similarity between the sentence vector of each knowledge text in the knowledge base and the query sentence vector. After the query, the present invention sorts all knowledge texts by similarity score, selects the multiple knowledge texts with the highest scores, and arranges the retrieved knowledge text fragments in descending order of similarity.

[0064] Step S23: The natural language instructions, multiple knowledge text fragments, and the semantic temporal features are concatenated to form a unified modal input sequence.

[0065] In step S23, the present invention assembles natural language instructions, multiple retrieved knowledge text fragments, and the generated semantic temporal feature sequence according to a predefined format. The assembly format adopts the standard input format of the language inference model, and special delimiters are used to mark the boundaries of different types of information. The assembly order is as follows: first, the starting delimiter is placed; then, the natural language instruction text content is placed; next, the first delimiter is placed; then, multiple knowledge text fragments are placed sequentially; after all knowledge texts are placed, delimiters are placed again; finally, the vectors in the semantic temporal feature sequence are placed sequentially. Before assembly, the text content is segmented by the word segmenter of the language inference model. The word segmenter divides the text string into a sequence of sub-words, and each sub-word is converted into a corresponding word embedding vector by looking up a table. The dimension of the word embedding vector is the same as the dimension of the semantic temporal feature vector. The assembled unified modal input sequence is a vector sequence. Each position in the sequence corresponds to a fixed-dimensional vector. The positions from the text sub-words store the word embedding vectors, and the positions from the temporal data store the semantic temporal feature vectors. The special delimiters correspond to their respective special embedding vectors.

[0066] In step S2, the unified modal input sequence is finally obtained through integration. Its structure is as follows:

[0067] in, For time window, The number of knowledge text fragments. For received natural language instructions, This refers to a collection of multiple retrieved knowledge text fragments, including all retrieved knowledge text fragments. Semantic temporal feature sequences generated for dynamic state representation transformation. and These are different delimiters used to mark the boundaries of different types of information.

[0068] Step S3: Input the unified modal input sequence into the language reasoning model, perform cross-modal information fusion reasoning through the self-attention mechanism, and output structured decision results.

[0069] The structured decision results in step S3 include: diagnostic conclusions, risk scores, and treatment plans.

[0070] In step S3, the present invention inputs the unified modal input sequence into the language inference model GLM-4.5 for deep processing. The language inference model adopts the Transformer architecture, and the core computing unit is a multi-layered stacked self-attention module. The unified modal input sequence of the present invention includes text word embedding vectors and semantic temporal feature vectors. The self-attention mechanism processes these two types of vectors. Attention weights are calculated between the query vector at the text position and the key vector at the temporal position, and attention weights are also calculated between the query vector at the temporal position and the key vector at the text position. Cross-modal information fusion is achieved through this bidirectional attention calculation. The subsequent model passes the calculation layer by layer, and the output of each layer serves as the input of the next layer. The vector sequence output by the final layer contains fully fused cross-modal semantic information. Finally, the present invention extracts vectors at specific positions from the final layer output and generates structured decision results through the output projection layer. The output projection layer contains three independent decoding heads: the diagnostic decoding head maps vectors to text sequences to generate diagnostic conclusions, the risk scoring decoding head maps vectors to scalar values ​​between 0 and 1, and the treatment plan decoding head maps vectors to text sequences to generate treatment plans. The outputs of the three decoding heads are finally assembled into a structured decision result in JSON format.

[0071] In addition, after step S3, a confidence score is calculated based on the probability distribution entropy of the token output by the language inference model. The specific scoring expression is as follows:

[0072] in, To output the Shannon entropy of the probability distribution, Given the vocabulary size, for the calculated confidence scores, low entropy corresponds to high confidence, meaning a sharp distribution implies high confidence.

[0073] Step S4: Capture external correction feedback for the structured decision result, construct a multi-task loss function based on the difference between the external correction feedback and the structured decision result, and weight the multi-task loss function using heuristic weighting factors to obtain the final optimization objective.

[0074] Step S4 further includes:

[0075] Step S41: Receive the correction feedback data for the structured decision result; S42: For the diagnostic task corresponding to the diagnostic conclusion, calculate the distance metric between the diagnostic conclusion and the corrected diagnostic conclusion in the semantic space using a contrastive learning loss strategy to obtain a diagnostic loss term; For the risk assessment task corresponding to the risk score, calculate the mean square error between the risk score and the corrected risk score to obtain a risk loss term; For the scheme generation task corresponding to the processing scheme, calculate the cosine similarity loss between the sentence vector representations of the processing scheme and the corrected processing scheme to obtain a scheme loss term.

[0076] In steps S41 and S42, the present invention receives correction feedback data submitted by personnel in the relevant field. This correction feedback data corrects the structured decision results output by the language reasoning model and includes three fields: the corrected diagnostic conclusion, the corrected risk score, and the corrected treatment plan. The present invention pairs and stores the received correction feedback data with the structured decision results output by the model, forming training sample pairs. Each sample pair contains the model's predicted value and the expert-annotated true value.

[0077] Subsequently, this invention calculates a diagnostic loss term for the diagnostic task. Specifically, this invention uses a contrastive learning loss strategy to input the model's output diagnostic conclusion and the corrected diagnostic conclusion into a text encoder, converting them into sentence vector representations. The sentence vectors are fixed-dimensional real-number vectors. This invention uses the sentence vectors of the corrected diagnostic conclusions as positive samples and randomly extracts diagnostic texts from other fault cases in the knowledge base as negative samples, converting them into sentence vectors. The contrastive learning loss requires that the cosine similarity between the model's output diagnostic conclusion sentence vector and the positive sample sentence vector be as large as possible, and the cosine similarity with the negative sample sentence vector be as small as possible. The contrastive loss value is constructed by calculating the similarity difference between the positive and negative samples.

[0078] In addition, for risk assessment tasks, this invention directly calculates the difference between the risk score output by the model and the corrected risk score, and squares the difference to obtain the mean square error as the risk loss term.

[0079] For the scheme generation task, this invention inputs the processing scheme output by the model and the corrected processing scheme into the text encoder to convert them into sentence vector representations, calculates the cosine similarity between the two sentence vectors, and uses the cosine distance obtained by subtracting the cosine similarity from 1 as the scheme loss term. The larger the cosine distance, the greater the semantic difference between the two schemes.

[0080] Step S43: Perform a weighted summation of the diagnostic loss term, the risk loss term, and the solution loss term to obtain the multi-task loss function.

[0081] In step S43, this invention aims to construct a multi-task loss function by weighted summation of the loss terms of the three tasks. The diagnosis loss term, risk loss term, and solution loss term correspond to three independent learning objectives. This invention assigns a weight coefficient to each loss term. The diagnosis loss term is multiplied by the diagnosis weight coefficient, the risk loss term by the risk weight coefficient, and the solution loss term by the solution weight coefficient. The three weighted loss values ​​are then summed to obtain the multi-task loss function. The values ​​of the weight coefficients are determined based on the importance of the tasks; the diagnosis task, as the core task, is assigned a larger weight coefficient, while the risk assessment and solution generation tasks are assigned relatively smaller weight coefficients.

[0082] Step S44: Calculate the heuristic weighting factor based on the correction magnitude of the correction feedback data.

[0083] Further, in step S44, the present invention calculates a heuristic weighting factor based on the correction magnitude of the correction feedback data. This factor consists of a base value and two adjustment terms. The base value is set to 1. The first adjustment term uses an indicator function to determine whether the corrected diagnostic conclusion is empty. If the corrected diagnostic conclusion is not empty, the indicator function takes the value of 1; otherwise, it takes the value of 0. The value of the indicator function is multiplied by the first weighting coefficient to obtain the first adjustment term. The second adjustment term calculates the absolute value of the difference between the model output risk score and the corrected risk score. The absolute value of the difference is multiplied by the second weighting coefficient to obtain the second adjustment term.

[0084] This invention adds the base value to two adjustment terms to obtain a heuristic weighting factor, as shown in the following expression. When the diagnostic conclusion is revised, the first adjustment term is non-zero; when the risk score is adjusted significantly, the second adjustment term is large. Both situations lead to a significant increase in the heuristic weighting factor, indicating that the training sample has high learning value. The first and second weighting coefficients are preset hyperparameters that control the degree of influence of the two types of revisions on the weighting factor.

[0085] The expression for the heuristic weighting factor in step S44 is as follows:

[0086] in, As a heuristic weighting factor, For indicator functions, This is the revised diagnostic conclusion. It is an empty set. The risk score output by the model, For the revised risk score, The preset first weighting coefficient, This is the preset second weighting coefficient.

[0087] Step S45: Multiply the heuristic weighting factor by the multi-task loss function to obtain the final optimization objective.

[0088] In step S45, the present invention multiplies the heuristic weighting factor by the multi-task loss function to obtain the final optimization objective. The value of the multi-task loss function represents the magnitude of the model prediction error, and the value of the heuristic weighting factor represents the importance weight of the current training sample. The final optimization objective serves as the objective function for backpropagation. Gradient calculation is based on the partial derivative of this objective function with respect to the model parameters. The magnitude of the gradient is modulated by the heuristic weighting factor; samples with larger weighting factors produce stronger gradient signals, resulting in a stronger fit to these samples when updating the model parameters.

[0089] The expression for the final optimization objective in step S45 is:

[0090] in, For the ultimate optimization goal, For multi-task loss function, To diagnose the loss, The weighting coefficients corresponding to the diagnostic loss. For risk loss, The weighting coefficients corresponding to risk losses. For the loss of the plan, These are the weighting coefficients corresponding to the loss of the proposed solution.

[0091] Step S5: Perform gradient backpropagation on the final optimization objective, and selectively apply the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network to complete the parameter update; wherein, the parameters of the language inference model remain frozen during the parameter update process.

[0092] Furthermore, in step S5, the present invention performs gradient backpropagation calculation on the final optimization objective. Backpropagation starts from the value of the final optimization objective and calculates the partial derivative of each parameter with respect to the objective function, i.e., the gradient, layer by layer along the opposite direction of the computation graph. The computation graph contains three main modules: a language inference model, a semantic mapping network, and a continuous-time encoder.

[0093] This invention sets a freeze flag on all parameters of the language inference model before backpropagation begins. The freeze flag prevents the gradient from continuing to propagate forward within the module, and the gradient values ​​of the weight matrix and bias vector of each layer inside the language inference model are forced to be zero.

[0094] The gradient signal starts from the final optimization objective, passes through the output layer of the language inference model, and reaches the unified modality input sequence. At the unified modality input sequence, the gradient splits into two paths: the gradient at the text word embedding position terminates its propagation due to the freezing of the language inference model, while the gradient at the semantic temporal feature position continues to propagate forward into the semantic mapping network. Subsequently, the gradient calculates the partial derivatives of the weight matrix and bias vector in the semantic mapping network to obtain the gradient values ​​of the semantic mapping network parameters and stores them. The gradient continues to propagate from the semantic mapping network to the temporal state representation sequence, and then enters the continuous-time encoder. In the encoder, the partial derivatives of the weight matrix involved in the state transition function are calculated to obtain the gradient values ​​of the continuous-time encoder parameters and store them.

[0095] After obtaining the gradient values, this invention uses the AdamW optimizer to process the stored gradient values. AdamW maintains the first-order moment estimate and the second-order moment estimate for each parameter, updates these two estimates based on the gradient values, and then calculates the parameter update amount. Finally, this invention applies the calculated parameter update amount to the parameters of the continuous-time encoder and the semantic mapping network, respectively, and completes the parameter update by subtracting the parameter update amount from the parameter.

[0096] Specifically, during backpropagation, the massive parameter set of GLM-4.5 is completely frozen, and gradients are only computed and propagated to the projected network parameters. CfC timing encoder parameters The expression is:

[0097] in, This is the gradient operator.

[0098] During parameter updates, the AdamW optimizer is used to adjust the calculated gradients. and Update.

[0099] in, for Updatable parameters at any given time for Updatable parameters at any given time For learning rate, This is the bias correction value for the first-order moment estimate. This is the bias correction value for the second-order moment estimation. It is a numerically stable term.

[0100] The step S5 is followed by:

[0101] Step S6: The current parameter update amount and the historical parameter update amount are weighted and fused using a moving average strategy to obtain a smoothed parameter update amount, which is then applied to the continuous-time encoder and semantic mapping network.

[0102] In step S6, this invention smooths the parameter update amount using a moving average strategy. A cumulative value of historical parameter updates is maintained using this strategy, recording the average effect of multiple past parameter updates. After calculating the current parameter update amount in step S5, this invention does not directly apply it to the parameters. Instead, it first performs a weighted fusion with the historical parameter update amount. The weighted fusion is calculated by multiplying the current parameter update amount by a smoothing coefficient, multiplying the historical parameter update amount by 1 minus the smoothing coefficient, and adding the two to obtain the smoothed parameter update amount. A typical value for the smoothing coefficient is 0.9, indicating that historical information dominates. Finally, this invention applies the smoothed parameter update amount to the parameters of the continuous-time encoder and the semantic mapping network, that is, subtracting the smoothed parameter update amount from the current parameter value to obtain the new parameter value. The smoothed parameter update amount is also stored as a new historical parameter update amount for use in the next parameter update. For the continuous-time encoder, each weight matrix and the weight matrix and bias vector of the semantic mapping network independently maintain their own historical parameter update amounts, and the smoothing process is performed independently on each parameter tensor.

[0103] Step S7: Compare the heuristic weight factor with the preset storage threshold. When the heuristic weight factor exceeds the preset storage threshold, store the current input data and external correction feedback as experience samples in the experience replay buffer. When the system is idle, randomly select samples from the experience replay buffer for repeated training to continuously optimize the parameters of the continuous time encoder and semantic mapping network.

[0104] In step S7, the present invention further compares the heuristic weight factor with a preset storage threshold to determine whether to store the current training sample. The preset storage threshold is a fixed value, indicating that only samples with significant correction are considered high-value experiences. During the comparison, the present invention performs a numerical comparison operation. If the heuristic weight factor is greater than the preset storage threshold, a storage operation is triggered. The storage operation encapsulates the current input data and external correction feedback into an experience sample. The current input data includes a multi-dimensional time-series data stream, natural language instructions, and retrieved knowledge text fragments. The external correction feedback includes the corrected diagnostic conclusion, the corrected risk score, and the corrected processing plan. The experience sample is stored in an experience replay buffer in key-value pairs. The buffer adopts a first-in, first-out queue structure; when the buffer capacity reaches its limit, the earliest stored sample is removed.

[0105] When the system is idle, i.e., during periods when there are no real-time maintenance tasks to process, this invention initiates an experience replay training process during these idle periods. In the training process, a fixed number of experience samples are first randomly selected from the experience replay buffer using a uniform distribution. Each sample is then sequentially input into the model. The input data from the experience samples is processed through steps S1 to S3 to generate structured decision results. External correction feedback from the experience samples is paired with the structured decision results to execute steps S4 to S6 for loss calculation and parameter update. Subsequently, gradient descent optimization is performed on the parameters of the continuous-time encoder and semantic mapping network. Parameter values ​​are continuously adjusted through multiple rounds of sampling and training.

[0106] The following describes a real-time transfer adaptive learning method for multimodal intelligent agents based on specific applications.

[0107] The present invention discloses a real-time migration adaptive learning method for multimodal power agents based on a closed continuous-time model. In practice, this method can be used in the automated operation and maintenance scenario of a regional power grid control center. In this scenario, a multimodal power agent is responsible for monitoring the operating status of key equipment (such as transformers and circuit breakers) 24 / 7, assisting operation and maintenance personnel in fault diagnosis and decision-making.

[0108] In the application, the agent detected that the No. 7 main circuit breaker and the No. 2 main transformer in the 110kV substation successively showed abnormal data in a short period of time. The junior maintenance personnel issued instructions to the agent in order to assess the risk.

[0109] First, the agent receives a multimodal data input containing real-time time-series data and natural language commands from maintenance personnel. At this point, the CfC time encoder and the GLM-4.5 large model base within the agent have been deployed. The CfC time encoder efficiently processes the data stream, encoding it into a series of time-series state representations. The obtained representations and operation and maintenance instructions (operator_command) are integrated into a unified input sequence and fed into the GLM-4.5 large model.

[0110] Based on the pre-trained knowledge, the GLM-4.5 model performs initial inference. Since the model has not yet learned this kind of strongly correlated failure mode between specific devices, the output tends to treat the two abnormal events as independent problems.

[0111] Specifically, the agent's initial output is:

[0112] {

[0113] "diagnosis": "CB-07 may pose an overcurrent risk, which is not directly related to the high temperature of TR-02."

[0114] "risk_score": 0.6

[0115] "action_plan": "It is recommended to send personnel to check the load status of CB-07."

[0116] "confidence": 0.75

[0117] }

[0118] The diagnostic results identified the overcurrent risk but failed to find the root cause of the problem, i.e., the intrinsic link between the two events, thus the risk assessment was underestimated.

[0119] Subsequently, the operation and maintenance experts reviewed the output of the intelligent agent and determined that it was a typical chain reaction caused by a downstream line fault. The fault caused the line current to increase, which in turn caused the upstream circuit breaker CB-07 to overcurrent. At the same time, the large current also caused the transformer TR-02 to overload and heat up. The corrective feedback obtained is as follows.

[0120] {

[0121] "feedback_diagnosis": "The overcurrent of CB-07 and the high temperature of TR-02 are strongly correlated, which is a chain reaction caused by a fault in the downstream line."

[0122] "feedback_risk_score": 0.9,

[0123] "feedback_action_plan": "Isolate the downstream faulty line immediately and check the health status of CB-07 and TR-02."

[0124] }

[0125] The system captures this feedback, triggers the online learning process, and constructs a composite loss function. Subsequently, the heuristic weight S is calculated based on the significance of the feedback. Because the diagnosis was significantly revised and the risk score was adjusted considerably (|0.6 - 0.9| = 0.3), S is calculated as a relatively high value, which is ultimately used for the loss in backpropagation. .

[0126] Subsequent training using backpropagation loss yields an updated CfC encoder that is better able to extract the specific pattern between current surges and gradual temperature increases from time-series data and map it to a region in the semantic space representing a chain reaction or strong correlation. Simultaneously, due to the high heuristic weight of this feedback, the corresponding input data and expert feedback pair are stored as a high-value experience in an experience replay buffer, allowing the agent to periodically review the experience value during subsequent idle time.

[0127] The corrected agent's updated output is:

[0128] {

[0129] "diagnosis": "Overcurrent was detected in the circuit breaker, and it was highly correlated with the high temperature of the transformer. The preliminary assessment is that this is a chain reaction triggered by a downstream line fault."

[0130] "risk_score": 0.88

[0131] "action_plan": "It is recommended to immediately investigate and isolate the relevant downstream lines, while closely monitoring the status of circuit breakers and transformers, and preparing emergency plans."

[0132] "confidence": 0.92

[0133] }

[0134] This invention aims to construct a multimodal power intelligent agent capable of real-time, adaptive learning, progressively achieving self-iteration of the model and knowledge-adaptive design. Therefore, it has the following advantages:

[0135] First, this invention constructs a highly efficient coupled multimodal intelligent agent architecture. Specifically, by designing the Closed Continuous-Time (CfC) model as a dedicated temporal encoder and deeply integrating it into a large language model (using the GLM-4.5 open-source model as a foundation), this invention obtains a unified framework capable of instantaneously processing continuous temporal data streams and seamlessly transferring dynamic information to the language model for advanced reasoning.

[0136] Secondly, this invention implements a feedback-oriented real-time transfer adaptive learning mechanism. This mechanism utilizes the above architecture to design an online learning process, enabling the agent to capture real-time instructions and feedback from maintenance personnel and transform them into direct, incremental adjustments to model parameters. The core of the process is to update only the CfC model, which acts as a time encoder, and its projection network parameters, thereby ensuring extremely high learning efficiency while avoiding catastrophic forgetting.

[0137] Example 2:

[0138] Based on the same inventive concept, this embodiment provides a real-time transfer adaptive learning system for multimodal intelligent agents. The principle of solving the problem is similar to that of the real-time transfer adaptive learning method for multimodal intelligent agents provided in Embodiment 1, and the repeated parts will not be described again.

[0139] like Figure 2 As shown, the real-time transfer adaptive learning system for multimodal intelligent agents according to the present invention includes:

[0140] Encoding module 100: used to dynamically encode the state of a multidimensional time-series data stream using a continuous-time encoder to obtain a time-series state representation sequence;

[0141] The splicing module 200 is configured as a semantic mapping network to splice the retrieved knowledge text, the received natural language instructions, and the semantic temporal features based on the temporal state representation sequence transformation to obtain a unified modality input sequence.

[0142] The reasoning module 300 is configured as a language reasoning model to receive the unified modal input sequence, perform fusion reasoning on cross-modal information through a self-attention mechanism, and output structured decision results.

[0143] Construction module 400: used to capture external correction feedback for the structured decision result, construct a multi-task loss function based on the difference between the external correction feedback and the structured decision result, and weight the multi-task loss function through heuristic weighting factors to obtain the final optimization target;

[0144] Training module 500: used to perform gradient backpropagation on the final optimization objective, selectively applying the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network to complete the parameter update; wherein, the parameters of the language inference model remain frozen during the parameter update process.

[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A real-time transfer adaptive learning method for multimodal intelligent agents, characterized in that, include: Step S1: Dynamically encode the multidimensional time-series data stream using a continuous-time encoder to obtain a time-series state representation sequence; Step S2: Based on the semantic mapping network, the retrieved knowledge text, the received natural language instructions, and the semantic temporal features based on the temporal state representation sequence transformation are concatenated to obtain a unified modality input sequence. Step S3: Input the unified modal input sequence into the language reasoning model, perform cross-modal information fusion reasoning through the self-attention mechanism, and output structured decision results; Step S4: Capture external correction feedback for the structured decision result, construct a multi-task loss function based on the difference between the external correction feedback and the structured decision result, and weight the multi-task loss function using heuristic weighting factors to obtain the final optimization objective; Step S5: Perform gradient backpropagation on the final optimization objective, selectively applying the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network to complete the parameter update; wherein, the parameters of the language inference model remain frozen during the parameter update process.

2. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 1, characterized in that: In step S1, the method for obtaining the temporal state representation sequence includes: Step S11: Obtain a multi-dimensional time-series data stream within the time window, wherein the multi-dimensional time-series data stream contains multiple types of measurement indicators from multiple devices; Step S12: Dynamic state encoding of the multidimensional time-series data stream through a continuous-time encoder includes: processing the multidimensional time-series data stream step by step through a closed continuous-time neural network, calculating the neuron state update based on the closed-form solution of the liquid time constant differential equation, and obtaining the time-series state representation sequence, wherein each time-series state representation corresponds to a dynamic state encoding of one time step.

3. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 2, characterized in that: In step S12, the expression for the state update of the closed continuous-time neural network is: in, For time step index, For the first Neuron states at each time step For the first Input data at each time step, This is the set of trainable parameters for a closed continuous-time neural network. It is the state transition function based on the closed solution of the differential equation of liquid time constant.

4. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 1, characterized in that: The method for obtaining the unified modal input sequence in step S2 is as follows: S21: Input the sequence of temporal state representations into a semantic mapping network, and map each temporal state representation to a semantic vector space compatible with the language reasoning model through linear projection transformation to obtain the semantic temporal features; S22: Using natural language commands as query vectors, retrieve multiple knowledge text fragments from the knowledge base based on the similarity of the retrieved texts. S23: The natural language instructions, multiple knowledge text fragments, and the semantic temporal features are concatenated to form a unified modal input sequence.

5. The real-time transfer adaptive learning method for multimodal agents according to claim 4, characterized in that: When the temporal state representation sequence is input into the semantic mapping network, the network structure is a single fully connected linear layer containing a weight matrix and a bias vector. Then, the dimension of the temporal state representation is recorded as the original dimension, and the hidden layer dimension of the language inference model is recorded as the target dimension. The purpose of the semantic mapping network is to project the state representation from the original dimension space to the target dimension space.

6. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 1, characterized in that: In step S4, the method for obtaining the final optimization objective includes: Step S41: Receive correction feedback data for the structured decision results; Step S42: For the diagnostic task corresponding to the diagnostic conclusion of the structured decision result, calculate the distance metric between the diagnostic conclusion and the corrected diagnostic conclusion in the semantic space using a contrastive learning loss strategy to obtain a diagnostic loss term; for the risk assessment task corresponding to the risk score of the structured decision result, calculate the mean square error between the risk score and the corrected risk score to obtain a risk loss term; for the scheme generation task corresponding to the processing scheme of the structured decision result, calculate the cosine similarity loss between the sentence vector representations of the processing scheme and the corrected processing scheme to obtain a scheme loss term. Step S43: Perform a weighted summation of the diagnostic loss term, the risk loss term, and the solution loss term to obtain the multi-task loss function; Step S44: Calculate the heuristic weighting factor based on the correction magnitude of the correction feedback data; Step S45: Multiply the heuristic weighting factor by the multi-task loss function to obtain the final optimization objective.

7. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 6, characterized in that: The expression for the heuristic weighting factor is: in, As a heuristic weighting factor, For indicator functions, This is the revised diagnostic conclusion. It is an empty set. The risk score output by the model, For the revised risk score, The preset first weighting coefficient, This is the preset second weighting coefficient.

8. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 6, characterized in that: The expression for the final optimization objective is: in, For the ultimate optimization goal, For multi-task loss function, To diagnose the loss, The weighting coefficients corresponding to the diagnostic loss. For risk loss, The weighting coefficients corresponding to risk losses. For the loss of the plan, These are the weighting coefficients corresponding to the loss of the proposed solution.

9. The real-time transfer adaptive learning method for multimodal intelligent agents according to claim 1, characterized in that: In step S5, the method for selectively applying the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network is as follows: The gradient signal starts from the final optimization target, passes through the output layer of the language inference model, and reaches the unified modality input sequence. At the position of the unified modality input sequence, the gradient splits into two paths: the gradient at the text word embedding position terminates its propagation due to the freezing of the language inference model, while the gradient at the semantic temporal feature position continues to propagate forward into the semantic mapping network; subsequently, the gradient calculates the partial derivatives of the weight matrix and bias vector in the semantic mapping network to obtain the gradient values ​​of the semantic mapping network parameters and stores them; the gradient continues to propagate from the semantic mapping network to the temporal state representation sequence, and then enters the continuous-time encoder. In the encoder, the partial derivatives of the weight matrix involved in the state transition function are calculated to obtain the gradient values ​​of the continuous-time encoder parameters and store them.

10. A real-time transfer adaptive learning system for multimodal intelligent agents, characterized in that, include: The encoding module is used to dynamically encode the state of a multidimensional time-series data stream using a continuous-time encoder to obtain a time-series state representation sequence. The splicing module is configured as a semantic mapping network to splice the retrieved knowledge text, the received natural language instructions, and the semantic temporal features based on the temporal state representation sequence transformation to obtain a unified modality input sequence. The reasoning module is configured as a language reasoning model to receive the unified modal input sequence, perform fusion reasoning on cross-modal information through a self-attention mechanism, and output structured decision results. A construction module is used to capture external correction feedback for the structured decision result, construct a multi-task loss function based on the difference between the external correction feedback and the structured decision result, and weight the multi-task loss function through heuristic weighting factors to obtain the final optimization objective; The training module is used to perform gradient backpropagation on the final optimization objective, selectively applying the gradient signal to the parameters of the continuous-time encoder and the semantic mapping network to complete the parameter update; wherein, the parameters of the language inference model remain frozen during the parameter update process.

Images