Multi-modal network state detection method based on large model alignment

Abstract

The invention discloses a multi-modal network state detection method based on large model alignment, which is used for processing multi-modal data of a communication network. The method comprises the steps that multi-modal data such as a time sequence and a text are converted into a unified semantic space through blocking and embedding, multi-modal features are aligned through cross-modal reprogramming by utilizing a text prototype library and an attention mechanism, and an injection task instruction and domain knowledge are enhanced by setting a dynamic prompt prefix so as to activate the LLM reasoning ability; then, LLM trunk model parameters are frozen, only an input and output adaptation layer is trained, and a lightweight optimal adaptation model is constructed; and generating a network state prediction result based on the adaptation layer. According to the method, dynamic alignment of multi-modal network data is achieved, the generated joint semantic representation can be further used for LLM reasoning, the purpose of more accurately sensing the network state is achieved, network state reasoning under few-sample and zero-sample scenes is supported, and the method is suitable for a heterogeneous network environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data processing in communication networks, and particularly relates to a multimodal network state detection method based on large model alignment. Background Art

[0002] In recent years, with the expansion of network scale and the improvement of intelligent requirements, the multimodal characteristics of network state data have become increasingly prominent, and multimodal network state analysis has become a key ability. Traditional network state analysis methods, such as statistics-based traffic monitoring and rule-based log parsing, only model single data modality and are difficult to capture multimodal correlations, resulting in high fault location latency and misjudgment rate, and semantic disconnection such as sudden traffic increase and log warnings; existing methods such as traffic prediction of the traffic prediction method based on convolutional neural network and long short-term memory network (Traffic Prediction Method Based on Convolutional Neural Network and Long Short-Term Memory), topology analysis of the network topology analysis method based on graph neural network (Network Topology Analysis Method Based on Graph Neural Network), etc. need to design independent models for different modalities and integrate the results through a post-fusion strategy, suffering from problems of computational redundancy and weak cross-modal semantic alignment ability.

[0003] Large Language Models (LLMs) demonstrate powerful cross-task generalization capabilities. LLMs perform excellently in the field of natural language processing. Literature 1 (Ziyang Ma, Wen Wu, Zhisheng Zheng, Yiwei Guo, Qian Chen, Shiliang Zhang, and Xie Chen. Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition. arXiv preprint arXiv:2309.10294, 2023.) points out that with the improvement of LLM architectures and training techniques, they have obtained broader knowledge in different modalities such as vision, speech, and text. Leveraging this knowledge enables collaborative work that integrates different data types. In contrast, traditional tools lack methods for jointly leveraging multiple knowledge bases. Additionally, Literature 2 (Suvir Mirchandani, Fei Xia, Pete Florence, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng, et al. Large language models as general pattern machines. In Proceedings of the 7th Annual Conference on Robot Learning, 2023.) points out that LLMs exhibit complex reasoning and pattern recognition capabilities. In contrast, most existing non-LLM methods are based on statistics and lack inherent reasoning abilities. Leveraging these skills, LLMs provide a promising approach to making multi-modal network state analysis more general, efficient, collaborative, and accessible.

[0004] Applying LLMs to multi-modal network state analysis depends on the effective alignment of time series data and natural language patterns. However, this is a challenging task. Because LLMs operate based on discrete tokens, while time series data is inherently continuous, and the spatio-temporal evolution characteristics of network states are difficult to model through static alignment methods. Additionally, LLMs do not possess the knowledge and reasoning abilities required to interpret time series patterns during the pre-training process.

[0005] Generally speaking, the existing technologies cannot efficiently achieve the unified semantic representation and dynamic alignment of multimodal network data, resulting in low accuracy of network state perception, weak cross-modal reasoning ability, and high resource consumption. Moreover, the pre-training of LLM does not naturally include the knowledge and reasoning ability to analyze time series patterns. Therefore, there is an urgent need for an efficient multimodal alignment framework to fully activate the knowledge within the LLM, enabling it to achieve general time series prediction with high precision, data efficiency, and task independence, and transferring the general capabilities of the LLM to network state analysis. Summary of the Invention

[0006] To achieve the dynamic alignment of multimodal network data, transfer the general capabilities of the LLM to network state analysis, and achieve the purpose of more accurately perceiving the network state, the present invention proposes a multimodal network state detection method based on large model alignment (Multimodal State Representation with LLM Alignment, abbreviated as MSR-LLA). Through multimodal reprogramming, dynamic spatio-temporal alignment, and lightweight adaptation technologies, network traffic, topology, and log data are mapped to the semantic space of the LLM to obtain network states with high precision and low latency, and then perform state analysis and autonomous decision-making.

[0007] The multimodal network state detection method based on large model alignment of the present invention includes the following steps:

[0008] Step 1: Collect multimodal time series data of network states, and the input conversion module preprocesses and aligns and encodes the multimodal time series data. The multimodal time series data of the network states includes network traffic feature data, device state data, and log text.

[0009] The preprocessing includes: dividing the multimodal time series data into n single-modal time series data according to the modality, where n is the number of modalities; performing standardization processing on the data of each modality, and the numerical data after standardization processing satisfies a mean of 0 and a standard deviation of 1, and converting the text into word vector embedding representation. Align each single-modal time series data according to the timestamp.

[0010] After data preprocessing, perform data segmentation and chunking. Divide the standardized single-modal time series data of each modality into overlapping or non-overlapping local segments (Patches), and the segment length L p is set according to the periodicity of the network state. Form multimodal input chunks from the single-modal time series segment data aligned by timestamp. The p-th input chunk contains the numerical multimodal time series data and text word vectors corresponding to the corresponding time period, p ∈ [1, P], where P is the number of segments; convert the numerical multimodal time series data of the p-th input chunk into embedding features through the embedding layer where d m is the embedding feature dimension.

[0011] Further, perform cross-modal embedding alignment. For each input block, project the embedding features into query vectors, project the text word vectors into key vectors and value vectors, align the numerical segments and text word vectors through the multi-head cross-attention mechanism, aggregate the outputs of multiple attention heads to obtain Z, and linearly project Z to the input dimension of the large language model LLM to obtain Then embed the dynamic prompt prefix E prompt and concatenate them to obtain the aligned multi-modal input O.

[0012] Step 2: Use an open-source LLM as the backbone model, keep its pre-trained parameters frozen, and only train the text prototype library and the output projection layer. The language vectors in the text prototype library are used to represent the network state. Then construct a dynamic prompt prefix Prompt and convert it into an embedded representation E prompt , and then E prompt and concatenate them to obtain the aligned multi-modal input O.

[0013] The dynamic prompt prefix consists of three parts: context description, statistical features, and task instructions. The context description refers to adding network domain knowledge. The task instructions refer to clarifying the task objective for the LLM. The statistical features refer to calculating the trend and periodic features of the input data. Concatenate the context description, statistical features, and task instructions in natural language form to obtain the dynamic prompt prefix Prompt.

[0014] Step 3: Input the multi-modal input O into the LLM. After being processed by the self-attention mechanism of the LLM, generate a hidden state that fuses semantics The output projection module discards the dynamic prompt prefix part, extracts the hidden state related to the network state For perform a flattening operation and a linear projection to obtain the network state prediction result.

[0015] The method of the present invention also sets a task extension module, and calls the corresponding task extension module according to the network state prediction result and the task objective. The task extension module includes: an anomaly detection module: this module calculates the anomaly score of the prediction error Y represents the true future time series value, represents the predicted network state time series value; compare the anomaly score s with the set threshold to determine whether the output triggers an anomaly warning. Root cause analysis module: By visualizing the attention weights of each input block, locate the modality and time segment that cause the anomaly, associate with the log warning, and infer the root cause in combination with the dynamic alignment result of the multi-modal data.

[0016] The method of the present invention jointly optimizes the prediction model of the main task and the detection modules of multiple tasks. The main task is the prediction of the multi-modal network state time series, and the multiple tasks refer to jointly optimizing other related tasks on the basis of the main task, such as anomaly detection, root cause analysis, etc. During training, the mean squared error loss or the mean absolute error loss is calculated for the main task, the cross-entropy loss is calculated for the multiple tasks, and the total loss is calculated by weighted summation of the losses of the main task and the multiple tasks. During training, the LLM backbone is frozen to keep the parameters isolated, and only the parameters of the text prototype library and the input transformation module and the output projection layer are updated. The method of the present invention forms an incremental data set by periodically collecting new multi-modal time series data of the network state, and updates the text prototype library based on 10% of the historical language vectors.

[0017] The method of the present invention is implemented as an edge-cloud collaborative architecture; an input transformation module is deployed on the edge side to process the collected multi-modal time series data of the network state in real time and generate aligned embeddings; an LLM and an output projection module are deployed in the cloud to receive the embeddings and run the LLM inference service to return the prediction results of the network state time series.

[0018] The advantages and positive effects of the present invention are as follows:

[0019] (1) The method of the present invention realizes the multi-modal reprogramming technology, supports multi-modal inputs such as time series and text, converts network data into text prototypes that can be understood by the LLM, breaks through the limitation of single data, and activates its cross-modal reasoning ability. The method of the present invention realizes the dynamic alignment of multi-modal network data by performing cross-modal impulse coding on network state data of different modalities, and further uses the generated joint semantic representation for the LLM to achieve the purpose of more accurately perceiving the network state.

[0020] (2) The method of the present invention realizes the dynamic prompt alignment mechanism, and realizes multi-modal dynamic association modeling through task instruction prefixes and spatio-temporal attention. The method of the present invention activates the LLM's understanding ability of network state patterns by introducing a text prototype library and using the multi-head cross-attention mechanism to align numerical multi-modal data segments with text word vectors.

[0021] (3) The method of the present invention has strong generalization ability. By fine-tuning the text prototype library, statistical features of the target scenario are injected into the dynamic prompt prefix, and the pre-trained prototype library and the general dynamic prompt prefix template are reused during cross-domain migration, without additional training, supporting network state inference in few-shot and zero-shot scenarios, and being applicable to heterogeneous network environments.

[0022] (4) The method of the present invention is deployed efficiently and time-savingly. Only the parameters of the adaptation layer (accounting for 0.2% of the total parameters) need to be trained to adapt to resource-constrained scenarios. In contrast, current existing methods usually require a large amount of in-domain data. Description of the Drawings

[0023] Figure 1 It is a schematic diagram of the overall process of the multi-modal network state detection method based on large model alignment of the present invention;

[0024] Figure 2 It is a schematic diagram of the cross-modal reprogramming process in the method of the present invention. Specific embodiments

[0025] The present invention will be further described in detail below with reference to embodiments and drawings.

[0026] A multi-modal network state detection method MSR-LLA based on large model alignment proposed by the present invention converts multi-modal data such as time series and text into a unified semantic space through chunking and embedding; uses a text prototype library and an attention mechanism to align multi-modal features through cross-modal reprogramming; and dynamically prompts to enhance the injection of task instructions and domain knowledge to activate the reasoning ability of the LLM; then freezes the LLM and performs lightweight adaptation by only training the input-output adaptation layer, keeping the backbone model parameters frozen; finally, generates a prediction result based on the adaptation layer to achieve intelligent perception and decision support for network status.

[0027] As Figure 1 shown, the end-to-end implementation framework of the method of the present invention includes an input layer, an inference layer, and an output layer. The input layer deploys an input conversion module, including a time series chunking module, a text structure descriptor, and a cross-modal reprogramming module, to realize real-time processing of raw collected data and generate an embedded representation of time-aligned multi-modal input; the cross-modal reprogramming module adopts a multi-head attention mechanism, calculates the modality alignment weight by dynamically screening text prototypes, and eliminates the dimensional difference based on linear projection. The input layer and the inference layer share a frozen pre-trained LLM, and it is assumed that multi-modal data can achieve semantic alignment through a text prototype library; the inference layer includes an LLM model and a dynamic prompt prefix generator; the dynamic prompt prefix generator integrates task instructions, historical statistical features, and domain knowledge to construct a context-aware semantic enhanced input; then, an optimal adaptation layer model is constructed, and the input-output adaptation parameters are optimized by minimizing the mean square error (MSE) of the prediction result, keeping the LLM backbone network frozen, and performing model training; finally, based on the reprogrammed multi-modal representation and dynamic prompt, a network state prediction result is generated. The output layer consists of a lightweight projection module, which flattens and linearly projects the prediction output of the LLM to output a network state prediction sequence.

[0028] Step 1: Multi-modal network state data collection and alignment encoding.

[0029] Step 101: Data collection and modality division. The network status data collected in the present invention includes network traffic data, device status data, and log text information. The network status data is multi-modal time-series data, including network traffic characteristics such as network bandwidth, throughput, latency, etc.; device status such as CPU utilization rate, memory occupancy rate; log text such as warning messages, operation logs. The collected multi-modal time-series network status data is subjected to modality division and timestamp alignment. Let the multi-modal time-series data of the collected network status be X total ∈R N×T , where n is the number of modalities, that is, the number of network status variables, T is the collection duration, and R represents the real number space. The multi-modal time-series data is divided into modalities and the timestamps are ensured to be aligned.

[0030] Step 102: Numerical data normalization and segmentation.

[0031] After modality division, the multi-variable time-series data is divided into N single-variable time-series data X∈R T . The time-series data of each modality is normalized. For numerical data such as network traffic, CPU utilization rate, etc., which are single-variable time-series data, reversible instance normalization (RevIN) is used to eliminate distribution shift, and the formula is:

[0032]

[0033] where, after normalizing X, X norm is obtained, and μ and σ are the mean and standard deviation of X respectively.

[0034] For text data such as warning messages, operation logs, they are converted into vector representations through a pre-trained word embedding model (such as BERT) of a text structure descriptor to retain semantic information.

[0035] The normalized single-variable time-series data is divided into overlapping or non-overlapping local segments (Patches), and the segment length L p is set according to the network status periodicity. In the embodiments of the present invention, a time series chunking module is used to divide the single-variable time-series data into overlapping segments of length L p to generate local semantic blocks where P is the number of divided segments, and the calculation formula is: S is the sliding step, L p is the time step of each segment, and T is the total length of the original time-series data collected.

[0036] In the embodiments of the present invention, the multi-modal data is aligned and segmented to ensure timestamp consistency, forming multi-modal input blocks. In the embedding layer, an image feature encoder is used to convert each single-modal data of the numerical type in each input block into an embedding representation, and the multi-variable data of the numerical type in each input block forms an embedding feature p ∈ [1, P], d m is the dimension of the embedded feature. An input block contains the embedded feature X of multimodal data corresponding to numerical data p and text word vectors.

[0037] It should be understood that there is no fixed order for the processing of segmentation and data standardization in the method of the present invention, and it can be flexibly adjusted according to the characteristics of the data and the processing requirements.

[0038] Step 103, text log embedding and cross-modal alignment.

[0039] Figure 2 is the process of the cross-modal reprogramming mechanism of the present invention. In the embodiment of the present invention, a pre-trained language model is used to convert log text into word vectors V log . For each input block, a linear projection layer is used to map each modal segment to a unified latent space to generate an embedded vector The embedded feature converted from numerical multimodal data is mapped into a query vector, the word vector is projected into key vectors and value vectors, and the numerical segment and the text word vector V are aligned through a multi-head cross-attention mechanism log to generate a joint semantic representation. The specific formula is:

[0040]

[0041] where T represents transpose. The meanings of the characters in the formula are as follows:

[0042] k represents different attention heads in the multi-head attention mechanism, and each head focuses on a different semantic subspace. For example, head 1 focuses on time correlation and head 2 focuses on log keyword matching; h represents the number of heads in the multi-head attention mechanism;

[0043] Q k represents the query vector of the k-th attention head, which is derived from the embedded representation of the numerical data of the input block;

[0044] represents the original block X p the intermediate vector after linear embedding mapping;

[0045] d k represents the dimension of each head, d k = d m / h, d m is the dimension of the model hidden layer;

[0046] represents the trainable projection matrices of the query vector, key vector and value vector in the cross-modal attention mechanism, which are the core components of the multi-head attention mechanism and are used to achieve semantic alignment between time series blocks and text modalities;

[0047] K k and V k represent the key vector and value vector of the k-th head, both of which are derived from the independent projection of the text word vector V log .

[0048] Aggregate the multi-head output Z k and linearly project it to the input dimension D of the LLM. The specific formula is as follows:

[0049]

[0050] where Concat represents the concatenation operation; W O represents the output projection matrix, W proj represents the trainable linear projection matrix, which is used to map the intermediate representation after cross-modal alignment to the input dimension of the LLM, D represents the hidden layer dimension of the frozen LLM backbone model, that is, the feature dimension of the input / output vector of the Transformer layer of the LLM.

[0051] Concatenate the dynamic prompt prefix embedding with the fused to obtain the aligned multi-modal input (abbreviation ), and the specific formula is as follows:

[0052]

[0053] where E prompt represents the vector sequence encoded by the Prompt generated in step 202 through a pre-trained language model (such as BERT), and L prompt represents the number of tokens of the prompt prefix.

[0054] Step 2: Alignment of the large model and dynamic prompt injection.

[0055] Step 201: Freeze the LLM backbone and lightweight adaptation. In the embodiment of the present invention, an open-source LLM is used as the backbone model, and its pre-trained parameters are frozen, and only the text prototype library and the output projection layer are trained to ensure the efficiency of knowledge transfer. The text prototype library consists of V' learnable language vectors, where V' << V and V is the number of language vectors in the original text prototype library. The language vectors in the text prototype library of the present invention are used to represent network states, such as "sudden increase in traffic" and "equipment overload".

[0056] Step 202: Construct a dynamic prompt prefix (Prompt-as-Prefix). The dynamic prompt prefix consists of three parts: context description, statistical features, and task instructions. By concatenating the prompt with the aligned multi-modal embedding, the LLM is guided to focus on the task-related semantics.

[0057] The context description enhances the LLM's understanding of the task by injecting network domain knowledge such as "core router traffic prediction". For example, adding the network domain knowledge "traffic surge warning" and describing the data source as "core router traffic data".

[0058] Statistical features refer to calculating the trends and periodicities of the input data, such as using a moving average to fit the trend, calculating autocorrelation lags, key lags, etc.

[0059] The task instruction is to clarify the prediction target, such as "predict the bandwidth utilization rate in the next 1 hour", to guide the inference direction of the LLM.

[0060] Concatenate the context description, statistical features, and task instruction in natural language form to obtain the dynamic prompt prefix Prompt:

[0061]

[0062] Taking the task of "predicting the bandwidth utilization rate in the next 1 hour" as an example, each part of the dynamic prompt prefix is as follows:

[0063] [Context description]: Core router traffic data: The bandwidth has suddenly increased to 95% in the past 5 minutes, triggering a threshold warning.

[0064] [Statistical features]: The average bandwidth growth rate in the past 1 hour is 5% per minute; the traffic conforms to the daily cycle pattern, and the current is in the historical peak interval.

[0065] [Task instruction]: Predict the bandwidth utilization rate in the next 1 hour.

[0066] Step 203: Concatenate the multi-modal input sequences.

[0067] As in step 103, concatenate the embedding vectors of the dynamic prompt prefix with the aligned multi-modal embeddings Sequentially concatenate to obtain O, input it into the LLM, and after being processed by the self-attention mechanism of the LLM, generate a hidden state with fused semantics

[0068] Step three: Network state representation and multi-task output.

[0069] Step 301: Perform hidden state decoding and prediction. Discard the prompt part in the LLM output and extract the hidden state related to the network state Output the prediction results for the next H steps through a flattening operation and a linear projection layer As follows:

[0070]

[0071] Among them, Flatten() is the flattening operation, Wo is the projection matrix, and b o is the bias term.

[0072] Step 302, Multitask Extension Module. Call the corresponding task extension module according to the network status prediction result and the task objective for task detection. In the embodiment of the present invention, an anomaly detection module and a root cause analysis module are provided.

[0073] Anomaly Detection Module: Stack a classification head and calculate the anomaly score of the prediction error Y represents the supervision signal, that is, the true future time series value, represents the predicted network status time series value, and determines whether to trigger an anomaly warning by comparing s with the set threshold.

[0074] Root Cause Analysis Module: Through attention weight visualization, locate the key modes during the anomaly period, such as the association between high CPU utilization and log warnings.

[0075] Taking "predicting the bandwidth utilization in the next 1 hour" as an example:

[0076] Continuously output the predicted value For example: represents the bandwidth utilization in the next 1 hour.

[0077] Anomaly Detection Module: Generate by calculating the prediction error, that is, the difference between the real-time observed value Y and the predicted value and stack the classification head to output the anomaly probability. If s > θ, and the set threshold θ = 0.8, then trigger an anomaly alarm and push it to the operation and maintenance platform.

[0078] Root Cause Analysis Module: For example, the attention weight shows that the association weight between the 144th block (14:00 - 14:05) and the log word "Link congestion" is 0.85. Combining with the fact that the CPU utilization rate suddenly increases to 95% synchronously and the traffic surge period overlaps with the backup task execution time, it is determined that the root cause is "unthrottled background backup task".

[0079] Step Four, Efficient Training and Resource Optimization. The main task uses the mean square error (MSE) or the mean absolute error (MAE); in the multi-task scenario, cross-entropy loss is introduced to jointly optimize the prediction and classification objectives. Secondly, formulate an efficient training strategy. Keep the parameters isolated, only update the input transformation and output projection layer parameters, and keep the LLM backbone frozen; accelerate quantization, and use dynamic quantization during deployment to compress the model memory occupancy and adapt to edge device inference. Further, consider few-shot and zero-shot adaptation. Few-shot means fine-tuning the text prototype library based on 10% of the historical data and injecting the statistical features of the target scenario into the prompt prefix. Zero-shot means that when migrating across domains, such as from the enterprise network to the 5G core network, reuse the pre-trained prototype library and the general prompt template without additional training.

[0080] Step 401: Design the loss function and perform joint optimization. The method of the present invention includes a main task and a multi-task scenario.

[0081] The main task is the temporal prediction of the multi-modal network state, such as predicting modal data such as bandwidth utilization and device CPU load in a specified future time period. Mean Square Error (MSE) or Mean Absolute Error (MAE) is used during the training of the main task. In the embodiments of the present invention, the Mean Square Error loss n represents the total number of samples, and Y i 、 and

[0082] respectively represent the actual network state value and the predicted network state value of the i-th sample. CE The multi-task scenario refers to jointly optimizing other related tasks on the basis of the main task, such as anomaly detection, root cause analysis, etc. By sharing the hidden state representation of the backbone LLM and stacking different task heads (such as classification heads), joint training is achieved. The multi-task scenario introduces the cross-entropy loss L total and jointly optimizes the prediction and anomaly detection objectives. The total loss L

[0083] L total = λ1L MSE + λ2L CE

[0084] where λ1 and λ2 are balance coefficients.

[0085] Taking "predicting the bandwidth utilization in the next 1 hour" as an example, the main task and multi-tasks are as follows:

[0086] Main task: Predict the bandwidth utilization in the next 1 hour.

[0087] Joint multi-tasks:

[0088] Anomaly detection: Calculate the real-time anomaly probability based on the prediction error and output the anomaly score through the classification head.

[0089] Root cause analysis: Use the attention weight visualization technology to locate the key modalities that cause anomalies.

[0090] Step 402: Develop an efficient training strategy.

[0091] During training, the LLM backbone remains frozen, parameter isolation is maintained, and only the parameters of the text prototype library, embedding layer, and output projection layer are updated. By periodically collecting new temporal data of the network state, an incremental dataset is formed to update the text prototype library. The parameter quantities of the embedding layer and the output projection layer only account for 0.2% of the total parameter quantities.

[0092] Accelerate quantization. During deployment, adopt dynamic quantization to compress the memory occupation of the model and adapt to inference on edge devices. In the embodiments of the present invention, FP16 quantization is adopted during deployment, compressing the model memory to 50% of the original size and adapting to real-time inference on edge devices.

[0093] Furthermore, consider few-shot and zero-shot adaptation. Few-shot means fine-tuning the text prototype library based on 10% of the historical data and injecting the statistical features of the target scenario into the dynamic prompt prefix. Zero-shot means that when performing cross-domain migration, such as from the enterprise network to the 5G core network, reusing the pre-trained prototype library and the general dynamic prompt prefix template without additional training.

[0094] Step Five: Edge-cloud collaborative deployment and dynamic update. Deploy an edge-cloud collaborative architecture. On the edge side, implement a lightweight input conversion module to process the raw data in real time and generate aligned embeddings. On the cloud side, run the LLM inference service, receive the embeddings and return the prediction results, supporting high-concurrency requests. The dynamic update mechanism periodically collects new data and incrementally updates the text prototype library and the statistical feature library to adapt to the drift of the network state distribution.

[0095] Step 501: Construct a hierarchical inference architecture as follows:

[0096] On the edge side: The lightweight module performs data preprocessing, segment division, and preliminary embedding, generating low-dimensional features and transmitting them to the cloud.

[0097] On the cloud side: The LLM completes complex inference and returns the prediction results and anomaly scores to the edge device.

[0098] Step 502: Update the text prototype library using the incremental update mechanism.

[0099] Periodically collect new data and update the text prototype library as follows:

[0100] E′ new = αE′ old +(1 - α)E′ update

[0101] where α is the smoothing coefficient, E′ old represents the old text prototype embedding matrix before update, E′ new represents the new text prototype embedding matrix after update, and E′ update is the prototype adjustment driven by new data.

[0102] In summary, through cross-modal alignment, dynamic prompt engineering, and large model knowledge transfer, the method of the present invention can not only achieve efficient representation and prediction of multi-modal network states, but also reconstruct the intelligent network operation and maintenance paradigm through the reverse control of prediction results, having practical value and technological innovation in the field of intelligent network operation and maintenance. In practical applications, by analyzing network state data in real time and predicting future trends, it can provide accurate decision-making support for actual network operation and maintenance: dynamically adjusting resource allocation based on prediction results to achieve on-demand scheduling of core resources such as bandwidth and computing power, avoiding congestion and improving utilization; quickly locating the root cause of faults by combining anomaly detection scores and linking with an automated repair system to shorten service interruption time; at the same time, continuously learning network state changes to adaptively optimize operation and maintenance strategies, significantly reducing the cost of manual intervention, promoting the transformation of network operation and maintenance towards intelligence and predictive mode, and comprehensively improving stability and energy efficiency ratio.

[0103] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A multi-modal network state detection method based on large model alignment, characterized in that Including: Step 1: Collect multi-modal time-series data of network status, and the input conversion module preprocesses and aligns and encodes the data. The multi-modal time-series data of the network status includes network traffic feature data, device status data, and log text. The preprocessing includes: dividing the multi-modal time-series data into N single-modal time-series data according to the modality, where N is the number of modalities; performing standardization processing on the data of each modality, and the numerical data after standardization processing satisfies a mean of 0 and a standard deviation of 1, and converting the text into a word vector representation. Segment the standardized single-modal time-series data in an overlapping or non-overlapping manner, with the segment length being L p Set periodically according to the network state; form multi-modal input blocks from the single-modal time-series segment data with aligned timestamps. The p-th input block contains the digital multi-modal time-series data and text word vectors for the corresponding time period, where p ∈ [1, P] and P is the number of segments; convert the digital multi-modal time-series data of the p-th input block into embedding features X p ; for each input block, project the embedding features into query vectors, project the text word vectors into key vectors and value vectors, align the numerical segments and text word vectors through the multi-head cross-attention mechanism, aggregate the outputs of multiple attention heads to obtain Z, and linearly project Z to the input dimension of the large language model LLM to obtain the embedding Step 2: Use the large language model LLM as the prediction model for the network state time series. Freeze the pre-trained parameters of the LLM backbone network and only train the text prototype library and the output projection module. The language vectors in the text prototype library are used to represent the network state; construct a dynamic prompt prefix Prompt and convert it into an embedding representation E prompt , and then E prompt and are concatenated to obtain the aligned multi-modal input O; The dynamic prompt prefix is composed of context description, statistical features, and task instructions; the context description refers to adding network domain knowledge; the statistical features refer to the trend and periodic features of the calculated input data; the task instructions refer to the clear task objectives for the LLM; the context description, statistical features, and task instructions are concatenated in the form of natural language to obtain the dynamic prompt prefix Prompt. Step 3: Input the multi-modal input O into the LLM. After being processed by the self-attention mechanism of the LLM, a fused semantic hidden state is generated. The output projection module Discard the dynamic prompt prefix part and extract the hidden state related to the network state. For Perform a flattening operation and a linear projection to obtain the prediction result of the network state time series.

2. The method according to claim 1, characterized in that, In the step 1, the modalities included in the multi-modal time-series data are: network bandwidth, network throughput, network latency, CPU utilization rate, memory occupancy rate, warning information, and operation logs.

3. The method according to claim 1 or 2, characterized in that, In the step 1, aligning the numerical segment of each input block with the text word vector includes: Let the log text of the p-th input block be converted into a word vector V log , and the embedding feature X of the numerical multi-modal data in this block p is linearly embedded into an intermediate vector Align the numerical segment with the text word vector V through the multi-head cross-attention mechanism log , as follows: Among them, Softmax() is the activation function, and the superscript represents transpose, h represents the number of heads of the multi-head attention mechanism, and d k represents the dimension of each head, d k = d m / h, d m is the dimension of the embedded feature, Q k represents the query vector of the k-th attention head, K k and V k represent the key vector and value vector of the k-th head respectively, and are the projection matrices of the query vector, key vector, and value vector respectively; Z k is the output of the k-th attention head; Aggregate multi-head output Z k And linearly project it to the input dimension D of the large language model LLM to obtain As follows: Among them, Concat represents the concatenation operation, and W O represents the output projection matrix; W proj represents the trainable linear projection matrix; D represents the hidden layer dimension of the frozen LLM backbone model.

4. The method according to claim 1, wherein In the step 2, the statistical feature calculation includes: using a moving average to fit the trend of the input data and calculating the autocorrelation lag term.

5. The method according to claim 1, wherein In the third step described above, the multi-modal input O is processed by the LLM to obtain where L prompt represents the number of tokens of the dynamic prompt prefix, and the hidden state for extract the hidden state where 6. The method according to claim 1, wherein The method also sets a task extension module, which calls the corresponding detection module according to the network status prediction result and the task objective; the task extension module includes the following detection modules: Anomaly detection module: Calculate the anomaly score of the prediction error Y represents the true future time series value, represents the predicted network state time series value; Compare the anomaly score s with the set threshold to determine whether the output triggers an anomaly warning; Root cause analysis module: By visualizing the attention weights of each input block, locate the modality and time segment that cause the anomaly, associate with the log warning, and infer the root cause in combination with the dynamic alignment result of the multi-modal data.

7. The method according to claim 1 or 6, characterized in that The method jointly optimizes the prediction model of the main task and the detection module of the multi-task. The main task is the prediction of the multi-modal network status time series, and the multi-task is the detection task executed on top of the main task; during training, calculate the mean square error loss or mean absolute error loss for the main task, calculate the cross-entropy loss for the multi-task, and calculate the total loss by weighted summation of the losses; when training the main task, freeze the pre-trained parameters of the LLM backbone network and only update the parameters of the text prototype library, input conversion module, and output projection module.

8. The method according to claim 1, wherein The method is deployed to implement an edge-cloud collaborative architecture; the input conversion module is deployed on the edge side to process the collected multi-modal time-series data of the network status in real time and generate aligned embeddings. The LLM and the output projection module are deployed in the cloud, receive the embeddings and run the LLM inference service, and return the prediction result of the network status time series.

9. The method according to claim 7, wherein The method forms an incremental data set by periodically collecting new multi-modal time-series data of the network status, and updates the text prototype library based on 10% of the historical language vectors.

Images