Telecommunication network-oriented multi-modal pre-training large model construction method and device

Abstract

The invention discloses a telecommunication network-oriented multi-modal pre-training large model construction method and device, and the method comprises the steps: carrying out the unified Token processing of multi-source heterogeneous network data, and forming a discrete sequence in a shared semantic space; constructing a dynamic heterogeneous graph based on the sequence, and learning the unified representation of the fused spatio-temporal context in combination with space attention and time Transform; and pre-training is carried out by utilizing self-supervision tasks such as contrast learning and mask reconstruction, so that universal characterization which can be generalized to various operation and maintenance tasks is obtained. According to the scheme, the method can effectively extract the key features of the telecommunication network operation data, generates high-quality implicit vector representation, and provides powerful general representation support for subsequent diversified downstream tasks.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and telecommunications networks, and in particular to a method and apparatus for constructing a multimodal pre-trained large model for telecommunications networks. Background Technology

[0002] As a core infrastructure for the digital transformation of society, telecommunications networks have evolved from narrowband circuit switching in the fixed-line telephone era to 5G / 6G interconnected systems, and the services they support have expanded from voice and SMS to high-real-time scenarios such as high-definition video, cloud gaming, and connected vehicles. With network scale expanding from tens of millions of nodes to billions of terminals, traditional management models based on manual configuration and static rules face the challenges of low latency, high reliability, and high energy efficiency in operation and maintenance. Among related technologies, a technical system for traffic prediction, fault correlation analysis, and resource allocation has been constructed through the collaborative operation of traditional machine learning models such as Support Vector Machines (SVM) and decision trees with deep neural networks. Specifically, this system covers the entire process from traffic time-series modeling to explicit topology modeling, including key aspects such as graph neural networks (GNN) optimization of link load balancing and federated learning for data privacy protection. However, existing technologies still generally rely on single-modal data and a large number of manually labeled samples, making it difficult to achieve a unified semantic understanding of complex network behavior.

[0003] Existing multimodal large-scale modeling methods directly employ weakly coupled multimodal designs without specifically optimizing for the high dynamism, strong structure, and security sensitivity of telecommunications networks. This can lead to significant limitations in topology evolution modeling, cross-modal alignment, and edge deployment. Specifically, telecommunications network operational data includes multimodal information such as traffic sequences, alarm texts, and topology graph structures, but existing models lack a unified semantic benchmark, making them prone to mode collapse or spurious correlations. While dynamic heterogeneous graph modeling can explicitly characterize network structure, traditional models with billions of parameters struggle to meet the millisecond-level inference requirements at the edge. Consequently, existing technologies remain insufficient in addressing frequent network topology changes, overcoming catastrophic forgetting, and ensuring privacy compliance, impacting the reliability and sustainability of intelligent operation and maintenance decisions. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for constructing a multimodal pre-trained large model for telecommunications networks.

[0006] Another objective of this invention is to provide a device for constructing multimodal pre-trained large models for telecommunications networks.

[0007] The third objective of this invention is to provide a computer device.

[0008] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a method for constructing a multimodal pre-trained large model for telecommunications networks, comprising: S1 maps multi-source heterogeneous operational data in the telecommunications network into discrete token sequences with shared semantic structures through unified tokenization processing. S2, constructing a dynamic heterogeneous graph containing spatiotemporal topological information based on discrete token sequences; S3 mines topological interaction dependencies in the spatial dimension of dynamic heterogeneous graphs through neighborhood aggregation, captures evolutionary patterns in the temporal dimension through temporal modeling, and introduces an external memory mechanism to suppress anomalous noise, so as to generate a unified representation that integrates spatiotemporal context. S4. Self-supervised pre-training is performed using the unified representation to learn a general network representation applicable to a variety of downstream tasks through contrastive learning and mask reconstruction tasks.

[0010] In one embodiment of the present invention, S1 includes: Adaptive time window segmentation is performed on multi-source data to ensure that the time granularity is synchronized with business fluctuations; Within a local slice, a segmented aggregation approximation is performed, dividing the numerical sequence into several equal-length sub-intervals, and calculating the mean or median of each sub-interval as a representative value; then, based on a preset discretization threshold set or an equal-frequency / equal-width binning strategy, the representative value is mapped to a symbol in a finite symbol set, thereby compressing the original numerical sequence into a symbol string of uniform standard and extracting key local features. The design of the embedding layer updates the feature map based on the direction of the neural network, so that symbols with similar semantics or numerical values ​​are represented more closely in the embedding space; The data stream identifier, data stream characteristics, and global identifier are concatenated to generate a unified standard token sequence, which serves as the starting point for the entire process.

[0011] In one embodiment of the present invention, S2 includes: In the dynamic graph, type labels are added to nodes and edges to distinguish the various roles and relationships of base stations, links, services, and alarms; The edge weights are dynamically updated based on information about load changes and fault events, reflecting the strength of interaction and health status in real time. Output the topology snapshot sequence in chronological order to ensure that subsequent models can effectively identify the interaction structure.

[0012] In one embodiment of the present invention, S3 includes: By leveraging spatial attention to aggregate neighborhood information within the same snapshot, local topological dependencies can be learned. An external graph memory mechanism is introduced to smoothly update node representations and reduce the interference of sudden anomalies on prediction. The stacked temporal Transformer processes snapshot sequences to capture long-range temporal correlations; at the same time, it integrates GCN layers to achieve spatial-temporal collaboration, and finally generates a unified latent vector for subsequent pre-training.

[0013] In one embodiment of the present invention, S4 includes: A contrastive learning task is constructed, and the loss is estimated using information-noise contrast to improve the model's discriminative power; Set up reconstruction tasks to restore random masks or future fragments, and enhance information completion and generation capabilities; By employing a multi-task weighted mechanism to jointly optimize the individual losses, a highly generalizable pre-trained large model is obtained for use in telecommunications networks.

[0014] In one embodiment of the present invention, it further includes: By using general network characterization, we can jointly predict the traffic load, node failure probability, and link resource distribution of a telecommunications network at multiple future time steps to obtain joint prediction results. Based on the joint prediction results, under multi-dimensional constraints of bandwidth capacity, end-to-end latency, and quality of service, the optimal routing path and load balancing strategy are solved, and scheduling control instructions for network device configuration are generated to achieve elastic allocation of resources.

[0015] In one embodiment of the present invention, it further includes: Collect the actual network operation status data returned after the execution of the scheduling and control instructions, including real traffic, alarm events and performance indicators, and convert them into a standardized feedback token sequence according to the tokenization rules; The feedback token sequence is injected into the original multi-source data input stream as an incremental training sample to participate in the online fine-tuning or self-supervised reconstruction task of the model, so as to form an adaptive iterative closed loop.

[0016] To achieve the above objectives, a second aspect of the present invention provides a multimodal pre-trained large model construction apparatus for telecommunications networks, comprising: The data preprocessing module is used to map multi-source heterogeneous operational data in the telecommunications network into discrete token sequences with shared semantic structures through unified tokenization processing. The dynamic heterogeneous graph construction module is used to construct dynamic heterogeneous graphs containing spatiotemporal topological information based on discrete token sequences. The spatiotemporal co-coding module is used to mine topological interaction dependencies in the spatial dimension of dynamic heterogeneous graphs through neighborhood aggregation, capture evolutionary patterns in the temporal dimension through temporal modeling, and introduce an external memory mechanism to suppress anomalous noise, so as to generate a unified representation that integrates spatiotemporal context. The network scheduling strategy generation module is used to perform self-supervised pre-training using the unified representation, and learn a general network representation applicable to a variety of downstream tasks through comparative learning and mask reconstruction tasks.

[0017] This invention provides a method and apparatus for constructing a multimodal pre-trained large model for telecommunications networks, which can realize unified representation and dynamic interactive modeling of multimodal data in telecommunications networks, effectively overcome the problems of data heterogeneity and labeling dependency, and improve the accuracy and real-time performance of network anomaly detection and intelligent operation and maintenance.

[0018] To achieve the above objectives, a third aspect of this application provides a computer device comprising a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory, for implementing a method for constructing a multimodal pre-trained large model for telecommunications networks as described in the first aspect embodiment.

[0019] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for constructing a multimodal pre-trained large model for telecommunications networks as described in the first aspect embodiment.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for constructing a multimodal pre-trained large model for telecommunications networks according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a method for constructing a multimodal pre-trained large model for telecommunications networks according to an embodiment of the present invention; Figure 3 This is a diagram illustrating the weight iteration process during the construction of a dynamic heterogeneous graph according to an embodiment of the present invention; Figure 4 This is a flowchart of the unified hidden representation based on spatiotemporal Transformer-GCN encoding according to an embodiment of the present invention; Figure 5 This is a flowchart of model training according to an embodiment of the present invention; Figure 6This is a complete flowchart of multi-step adaptive learning according to an embodiment of the present invention; Figure 7 This is a structural diagram of a multimodal pre-trained large model construction device for telecommunications networks according to an embodiment of the present invention; Figure 8 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following description, with reference to the accompanying drawings, describes a method and apparatus for constructing a multimodal pre-trained large model for telecommunications networks, according to an embodiment of the present invention.

[0025] Figure 1 This is a flowchart of a method for constructing a multimodal pre-trained large model for telecommunications networks according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1 maps multi-source heterogeneous operational data in the telecommunications network into discrete token sequences with shared semantic structures through unified tokenization processing. S2, constructing a dynamic heterogeneous graph containing spatiotemporal topological information based on discrete token sequences; S3 mines topological interaction dependencies in the spatial dimension of dynamic heterogeneous graphs through neighborhood aggregation, captures evolutionary patterns in the temporal dimension through temporal modeling, and introduces an external memory mechanism to suppress anomalous noise, so as to generate a unified representation that integrates spatiotemporal context. S4. Self-supervised pre-training is performed using the unified representation to learn a general network representation applicable to a variety of downstream tasks through contrastive learning and mask reconstruction tasks.

[0026] This invention proposes another method for constructing multimodal pre-trained large models for telecommunications networks, such as... Figure 2 As shown, follow these steps: Step 1: Tokenize the multi-source data in the telecommunications network, transforming the original, diverse network operation data into a unified discrete sequence, and designing a unified minimum semantic unit for subsequent graph modeling.

[0027] Step 1.1: Perform adaptive time window segmentation on multi-source data to ensure that the time granularity is synchronized with business fluctuations.

[0028] Step 1.2: Perform segmented aggregation approximation within the local slice to compress the numerical sequence into a standardized symbol string and extract key local features.

[0029] Step 1.3: Design an embedding layer to update the feature map based on the neural network direction, so that the representations of similar features are closer.

[0030] Step 1.4: Concatenate the data stream identifier, data stream feature, and global identifier to generate a unified standard token sequence, which serves as the starting point for the entire process input.

[0031] Step 2: Based on the token sequence obtained in Step 1, construct a dynamic heterogeneous graph, use multiple relationship graphs to accurately depict the structure and service interaction of the telecommunications network, and provide an explicit topological context for spatiotemporal coding.

[0032] Step 2.1: Label the nodes and edges in the dynamic graph with type labels to distinguish the various roles and relationships of base stations, links, services, and alarms.

[0033] Step 2.2: Dynamically update edge weights based on load changes and fault event information to reflect the strength of interaction and health status in real time.

[0034] Step 2.3: Output the topology snapshot sequence in chronological order to ensure that subsequent models can effectively identify the interaction structure.

[0035] Step 3: For the graph structure containing spatiotemporal topological information obtained in Step 2, deeply mine the interaction dependencies in both spatial and temporal dimensions, suppress noise based on external memory, and achieve a high-quality unified hidden representation.

[0036] Step 3.1: Use spatial attention to aggregate neighborhood information within the same snapshot and learn local topological dependencies.

[0037] Step 3.2: Introduce an external graph memory mechanism to smoothly update node representations and reduce the interference of sudden anomalies on prediction.

[0038] Step 3.3: Stacked temporal Transformers process snapshot sequences to capture long-range temporal correlations; at the same time, GCN layers are fused to achieve spatial-temporal collaboration, and finally a unified latent vector is generated for subsequent pre-training.

[0039] Step 4: Train the model based on the encoding output from Step 3. The purpose is to enable the model to simultaneously perform discrimination, completion, and generation under unlabeled or weakly labeled conditions, providing a general representation for diverse downstream tasks.

[0040] Step 4.1: Construct a contrastive learning task and use information-noise contrastive estimation to improve the model's discriminative power.

[0041] Step 4.2: Set up a reconstruction task to restore random masks or future fragments, and enhance information completion and generation capabilities.

[0042] Step 4.3: Employ a multi-task weighted mechanism to jointly optimize the individual losses, ultimately obtaining a pre-trained large model with strong generalization capabilities for use in telecommunications networks.

[0043] Step 5: Based on steps 1-4, the model inference is mapped to network scheduling decisions driven by constraint optimization, and the execution feedback is re-injected into the data stream to form an adaptive iterative process that combines prediction, control and relearning, thereby continuously improving operation and maintenance efficiency.

[0044] Step 5.1: Use the pre-trained large model obtained in Step 4 to jointly predict the traffic, failure probability and load distribution for several future steps.

[0045] Step 5.2: Solve for the optimal routing and load balancing strategies under the constraints of bandwidth, latency, and Quality of Service (QoS) to achieve elastic resource allocation.

[0046] Step 5.3: Distribute the strategy in real time and collect operational feedback; the feedback data flows back into Step 1 to drive the model to learn online and close the entire intelligent operation and maintenance link.

[0047] In one embodiment of the present invention, step one, multi-source data tokenization, includes: Telecommunication network monitoring data is diverse, including traffic data, alarm information, link logs, key performance indicators, etc. These data have varying original formats and cannot be directly used for unified model training. To achieve effective integration of multi-source data and establish a unified data representation, this step proposes a method for tokenizing multi-source data, transforming heterogeneous telecommunications network operation data into discrete sequences that can be used for computation, providing a foundation for subsequent modeling. Assuming the input data is a high-speed, real-time acquired multi-source monitoring stream, covering base station traffic, link KPIs, fault logs, and real-time alarms. To uniformly process sequences of different formats, the present invention first employs an adaptive time window segmentation method. Assuming... For any continuous data stream, its local variance is By window size Adjust the length of the time window to vary with business fluctuations:

[0048] Where σ² represents the local variance of the current data stream within a preset time interval.

[0049] Calculate the feature values ​​within each time window As shown in formula (2):

[0050] in, Indicates the first time within the time window The values ​​of the sampling points, where N represents the total number of sampling points within the time window.

[0051] To reduce data dimensionality and redundancy, within each time window, the PAA (Pattern of Analysis) is used to divide the window into equal-width segments, and the mean of each segment is calculated to obtain more representative features. :

[0052] in, These are samples within a window. Next, we will analyze the features within each time window. This involves a unified tokenization mapping. Specifically, it involves mapping all window features at the current moment. Perform Gaussian normalization to ensure that the normalized features have a mean of 0 and a variance of 1.

[0053] Where μ is the mean of the feature and σ is the standard deviation of the feature.

[0054] The Gaussian normalization is used to adjust the features within the window to a zero-mean, unit-variance distribution.

[0055] Then, this step designs an embedding layer to map each feature to a high-dimensional space, so that similar data stream features can be closer together in the embedding space. The transformation process of the embedding layer is shown in Equation (5):

[0056] Wherein, the dimension of the embedding vector is , It is an embedding vector representing data stream features. The mapping result in high-dimensional space. This is the embedding matrix, representing the transformation weight matrix from the input feature space to the embedding space, which is optimized through backpropagation in the neural network. Each time the network propagates forward, the embedding layer will... Mapped to a high-dimensional space, and during backpropagation, Optimize based on the loss function.

[0057] Finally, the embedding vectors obtained for each data feature are uniformly tokenized and encoded:

[0058] in, The identifier of the data stream is used to distinguish the data source corresponding to the token, such as alarm stream or traffic stream; The formula is output through the embedding layer, representing the characteristics of the current data stream; It is a global identifier used to distinguish different data sources or different tasks. At this point, all multi-source heterogeneous data has been transformed into a token stream with the same dimension and distribution for subsequent network training and inference.

[0059] In one embodiment of the present invention, step two, dynamic heterogeneous graph construction, includes: based on the unified token sequence obtained in step one, the present invention further constructs a graph model capable of representing the network structure and temporal dynamics. Specifically, this step proposes a dynamic heterogeneous graph generation method based on token sequences to accurately describe the time-varying interaction relationships between network nodes and links. Since tokens themselves lack explicit structural associations, in order to allow the model to explicitly capture interaction relationships, this method maps them to a multi-relationship graph. First, assign roles and relationship labels to nodes and edges: nodes can be categorized as base stations, links, services, and alarms; edge types cover various semantics such as physical connection, service mapping, and causal influence. Specific design definitions are shown in Tables 1 and 2.

[0060] Table 1. Definitions of Telecommunication Network Nodes in the Dynamic Diagram

[0061] Table 2. Edge type definitions in dynamic graphs

[0062] When the network load changes or a fault occurs, the edge weights are updated based on time according to equation (7):

[0063] in, Geographical distance, To increase traffic, It reflects the health status of the link or service at time t.

[0064] To further characterize the temporal evolution of the network, this invention generates a sequence of topological snapshots at time intervals, wherein the network changes are tracked based on spatiotemporal coding:

[0065] in, It is the current Topology snapshots output at any time yes The set of nodes at any given moment; yes The set of edges at each moment; yes The timestamps of time nodes and edges record their state at that moment. Figure 3 The iterative process of network snapshots is shown, where the node graph is used to distinguish node types, and the thickness of the connecting edges reflects the magnitude of the weights.

[0066] In one embodiment of the present invention, step three, spatiotemporal Transformer-GCN encoding, includes: based on the dynamic heterogeneous graph generated in step two, this step further mines the interaction relationships and dynamic patterns between spatial nodes in the telecommunications network. Specifically, this step proposes a deep encoding framework that integrates Transformer and graph convolutional networks, obtaining a high-quality unified hidden representation of the network state through deep feature extraction in both spatial and temporal dimensions. The goal of this step is to extract the dynamic graph snapshot sequence generated in step two. Encoding as a unified tensor representation This method captures spatial neighborhood dependencies, characterizes long-range dynamics across time, and buffers anomalous impulses through external memory. In the following formulas, LN represents the layer normalization operation, A represents the adjacency mask matrix of the graph, ⊙ represents element-wise multiplication, D represents the node degree matrix, and GELU is the nonlinear activation function. The process flow is as follows: Figure 4 As shown.

[0067] First, input embedding and layer normalization are performed for any node. Map its type to a type vector Then, based on the Token vector generated in step one Element-by-element addition:

[0068] in Layer normalization is used to mitigate gradient explosion and vanishing. Further spatial multi-head attention calculations are performed for single-frame snapshots. ,make Among them, the first The weights of each attention head are:

[0069] in, Represents the normalization function. This indicates the dimension of the key vector in the attention head, used to scale the dot product result; This is an adjacency mask matrix used to shield attention calculations between non-adjacent nodes; This represents element-wise multiplication (Hadamard product).

[0070] The spatial representation of the nodes is obtained after aggregation:

[0071] in, Indicates at time The node spatial feature representation obtained after fusion via a multi-head attention mechanism in the lower space. Indicates the first The spatial attention weight matrix corresponding to each attention head. Indicates the first The value vector matrix under each attention head Indicates all The outputs of each attention head are concatenated according to the feature dimension. It is a trainable linear mapping matrix used to perform linear transformations and dimension mappings on the concatenated multi-head features.

[0072] To smooth out abnormal pulses and reduce the impact of abnormal spikes, a gated memory unit is introduced:

[0073] in, This represents the Sigmoid activation function. This is the decay coefficient used to control the balance between historical memory and current input.

[0074] Further, time-based Transformer stacking is performed. Specifically, it involves stacking snapshot sequences across the same node. Sending it to the multi-head self-attention layer can yield:

[0075] in, This indicates multi-head self-attention computation. This represents the node time series feature representation obtained after time-dimensional Transformer encoding.

[0076] After the multi-head self-attention layer, two layers of feedforward and residual / LN are connected to output. To address the issue of local smoothing convergence, a two-layer normalized graph convolution is employed to further enhance local smoothness:

[0077] in, , It is the identity matrix; This is the corresponding node degree matrix; The activation function is non-linear, and the final output is a uniform tensor.

[0078] in, Indicates the number of nodes. This represents the dimension of node features. The final output is a node-level unified representation tensor. .

[0079] In one embodiment of the present invention, step four involves model training as follows: Figure 5 As shown, this step further improves the model's high-quality representation learning and generalization performance based on the spatiotemporal Transformer-GCN deep encoding completed in step three. Considering that the cost of collecting large-scale high-quality labeled data is extremely high in the actual operation of telecommunications networks, and that labeled data faces challenges of timeliness and dynamic changes, this invention proposes a self-supervised learning mechanism that does not require manual labeling. By designing reasonable auxiliary tasks, the model is guided to autonomously learn the intrinsic structure and spatiotemporal evolution pattern of network states, thereby obtaining high-quality and universal network representations. Specifically, this step mainly designs two types of complementary self-supervised tasks: (1) Discriminative task: based on graph contrast learning, improves the ability of the representation to distinguish different network states; (2) Generative task: based on random mask reconstruction, strengthens the model's state reconstruction and prediction capabilities.

[0080] First, the core idea of ​​graph contrastive learning is to enable the model to learn to recognize the differences between different network topology states. Specifically, this invention adopts the information noise contrastive loss function based on InfoNCE. Two different data augmentation operations are performed on the input dynamic graph sequence: (1) Node dropping: randomly removing a certain proportion of nodes. (2) Time period masking: randomly masking certain consecutive time segments. Two augmented views are generated through the above two operations, and positive samples are generated. Meanwhile, several negative samples were randomly selected from other unrelated batches. Subsequently, the InfoNCE loss function is defined as shown in equation (15):

[0081] in, Cosine similarity is used to measure the similarity between the state representations of two networks. The temperature coefficient is used to smooth the gradient optimization process. By calculating the minimum loss in equation (15), the model can effectively distinguish different topological states and improve the model's sensitivity to anomalies or sudden events.

[0082] For the generation task, this step considers the lack of certain key information. To improve the model's robustness in dealing with information gaps, the generation task is reconstructed based on a random mask. Specifically, the model's input feature sequence for each batch... A set of elements in a random mask of a certain proportion The model is required to reconstruct the original values ​​of the masked portion based on the unmasked information. The reconstruction loss function is defined as:

[0083] in, For true representation, This represents the reconstructed predicted value of the model. By optimizing the above loss, the model is forced to learn the inherent continuity and regularity of the network's spatiotemporal state, thereby enhancing its robust predictive ability against missing information.

[0084] Considering that some high-performance computing devices (e.g., centralized base stations) exist in actual telecommunications network deployment environments for training models, this step designs a knowledge distillation task to further enhance the overall system performance. Guided by the teacher model, this task strengthens the current model's understanding of more complex network features. The knowledge distillation loss is defined as:

[0085] in, KL divergence is used to measure the difference between the student model representation and the teacher model representation. The softening temperature coefficient is typically set to a value of [value to be filled in]. or .

[0086] Finally, a joint optimization strategy is adopted to combine the above three losses, and the overall training objective is defined as the weighted loss:

[0087] in, , and The importance of the discrimination, reconstruction, and distillation tasks is controlled separately, and adaptive updates are performed based on the multidimensional gradient balancing method. The self-supervised pre-trained model of the spatiotemporal state of the telecommunications network constructed in this step can effectively improve the model's generalization ability and anomaly detection sensitivity, significantly reduce the dependence on manually labeled data, and provide key technical support for intelligent decision-making and dynamic adjustment in subsequent network operation and maintenance.

[0088] In one embodiment of the present invention, step five, iterative reasoning and scheduling coordination, includes: This step, based on the multimodal pre-trained models obtained in steps one through four, constructs a complete iterative process from state prediction to resource scheduling, covering four core components: multi-step joint prediction, constraint optimization solution, reinforcement feedback update, and online adaptive learning. The specific process is as follows: Figure 6 As shown.

[0089] First, based on the unified hidden representation provided by the pre-trained model trained in step four. This invention uses a sequence decoder to predict the future. The network state at time t. Let the prediction target be network traffic. Link failure probability Link load distribution The multi-step prediction process is shown in equation (19):

[0090] in, This represents the sequence decoding function based on prior noise injection; k represents the prediction time step; This indicates the introduction of prior Gaussian noise to enhance the robustness and generalization ability of the model predictions. Based on the prediction results, this invention transforms these predictions into practical network scheduling strategies. In practical network scheduling scenarios, bandwidth is typically a key consideration. Delay Service quality Multiple practical constraints exist. Therefore, this invention constructs an optimization model with the objective function being to minimize the deviation between the predicted load and the target load:

[0091] To further improve the effectiveness of solving the above optimization problem, this invention introduces the Lagrange relaxation method, which transforms the constrained optimization problem into an unconstrained problem. The specific definition of the Lagrange function is as follows:

[0092] For decision variables By alternating between projected gradient descent (as shown in Equation (22)) and Lagrange multipliers The subgradient ascent method (as shown in equation (23)) is used to gradually approximate the optimal solution.

[0093]

[0094] in, and Let be the learning rate. The optimal policy obtained after the above optimization steps. This will serve as the final network scheduling execution scheme.

[0095] Optimal Strategy The network management system issues commands to network devices for execution in real time, and obtains execution feedback in real time by monitoring the operating status of the network devices. Feedback results It is used not only to evaluate the effectiveness of the current strategy, but also as an important basis for online updating of the pre-trained large model of the telecommunications network.

[0096] Considering the changing and uncertainties of the actual deployment environment of telecommunications networks, this step utilizes the feedback update mechanism based on reinforcement learning, taking into account the actual network benefits after the policy is implemented. As a reward signal, the decision network is further fine-tuned using the policy gradient method:

[0097] in The number of sampling trajectories, Let be the probability distribution function of the current policy. This reinforced feedback mechanism aims to enable the model to quickly adapt to the dynamic changes of the network, further improving the intelligence and robustness of decision-making. Execution logs and the latest network monitoring data after each round of scheduling are collected in real time and stored in a data buffer. Subsequently, the pre-trained large model is continuously updated through online fine-tuning using exponential moving averages.

[0098] in This indicates a minor update based on the latest feedback data. Control the fusion ratio of historical model weights to newly updated model weights. Through online update mode, the pre-trained large model can continuously evolve during network operation, automatically adapting to changes in network state and ensuring long-term decision performance.

[0099] This invention proposes a method for constructing a multimodal pre-trained large model for telecommunications networks. Through the synergy of multiple stages, it enables unified modeling and effective representation of multimodal data in telecommunications networks, thereby improving the model's prediction accuracy and practicality. Specifically: First, the multi-source data tokenization method effectively overcomes the challenge of heterogeneous data fusion, providing a unified input representation for model construction and supporting its development. Second, dynamic heterogeneous graph modeling accurately captures network topology and temporal dynamic changes, which is expected to enhance the model's perception of network structure changes and thus improve the ability to identify abnormal events. The designed spatiotemporal Transformer-GCN encoding method has strong feature extraction capabilities, which helps improve the accuracy of node anomaly prediction and shorten the response time to network anomalies. The introduced model training mechanism can fully utilize unlabeled or weakly labeled data, enhance the model's generalization ability, reduce reliance on manual annotation, and improve the flexibility of practical applications. Finally, the overall iterative inference-scheduling integrated mechanism helps to achieve dynamic and efficient allocation of network resources, promotes network load balancing, and supports long-term performance stability through the model's continuous online learning capability. In summary, the method of this invention provides an implementable intelligent operation and maintenance technical solution for anomaly detection and resource scheduling in telecommunications networks.

[0100] To achieve the above embodiments, such as Figure 7 As shown, this embodiment also provides a multimodal pre-trained large model construction device 10 for telecommunications networks, including: The data preprocessing module 100 is used to map multi-source heterogeneous operating data in the telecommunications network into discrete token sequences with shared semantic structures through unified tokenization processing. The dynamic heterogeneous graph construction module 200 is used to construct a dynamic heterogeneous graph containing spatiotemporal topological information based on discrete token sequences. The spatiotemporal co-coding module 300 is used to mine topological interaction dependencies in the spatial dimension of dynamic heterogeneous graphs through neighborhood aggregation, capture evolutionary patterns in the temporal dimension through temporal modeling, and introduce an external memory mechanism to suppress anomalous noise, so as to generate a unified representation that integrates spatiotemporal context. The network scheduling strategy generation module 400 is used to perform self-supervised pre-training using the unified representation, and learn a general network representation applicable to a variety of downstream tasks through comparative learning and mask reconstruction tasks.

[0101] This invention provides a multimodal pre-trained large model construction device for telecommunications networks, which can realize unified representation and dynamic interactive modeling of multimodal data in telecommunications networks, effectively overcome the problems of data heterogeneity and labeling dependence, and improve the accuracy and real-time performance of network anomaly detection and intelligent operation and maintenance.

[0102] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 8 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the multimodal pre-trained large model construction method for telecommunications networks described above.

[0103] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for constructing a multimodal pre-trained large model for telecommunications networks as described in the foregoing embodiments.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for constructing a multimodal pre-trained large model for telecommunications networks, characterized in that, include: S1 maps multi-source heterogeneous operational data in the telecommunications network into discrete token sequences with shared semantic structures through unified tokenization processing. S2, constructing a dynamic heterogeneous graph containing spatiotemporal topological information based on discrete token sequences; S3 mines topological interaction dependencies in the spatial dimension of dynamic heterogeneous graphs through neighborhood aggregation, captures evolutionary patterns in the temporal dimension through temporal modeling, and introduces an external memory mechanism to suppress anomalous noise, so as to generate a unified representation that integrates spatiotemporal context. S4. Self-supervised pre-training is performed using the unified representation to learn a general network representation applicable to a variety of downstream tasks through contrastive learning and mask reconstruction tasks.

2. The method as described in claim 1, characterized in that, After learning a general network representation applicable to multiple downstream tasks, the method further includes: By using general network characterization, we can jointly predict the traffic load, node failure probability, and link resource distribution of a telecommunications network at multiple future time steps to obtain joint prediction results. Based on the joint prediction results, under multi-dimensional constraints of bandwidth capacity, end-to-end latency, and quality of service, the optimal routing path and load balancing strategy are solved, and scheduling control instructions for network device configuration are generated to achieve elastic allocation of resources.

3. The method as described in claim 2, characterized in that, After generating scheduling control instructions for network device configuration, the method further includes: Collect the actual network operation status data returned after the execution of the scheduling and control instructions, including real traffic, alarm events and performance indicators, and convert them into a standardized feedback token sequence according to the tokenization rules; The feedback token sequence is injected into the original multi-source data input stream as an incremental training sample to participate in the online fine-tuning or self-supervised reconstruction task of the model, so as to form an adaptive iterative closed loop.

4. The method as described in claim 1, characterized in that, S1 includes: Adaptive time window segmentation is performed on multi-source data to ensure that the time granularity is synchronized with business fluctuations; Within a local slice, a segmented aggregation approximation is performed, dividing the numerical sequence into several equal-length sub-intervals, and calculating the mean or median of each sub-interval as a representative value; then, based on a preset discretization threshold set or an equal-frequency / equal-width binning strategy, the representative value is mapped to a symbol in a finite symbol set, thereby compressing the original numerical sequence into a symbol string of a unified standard and extracting key local features. The design of the embedding layer updates the feature map based on the direction of the neural network, so that symbols with similar semantics or numerical values ​​are represented more closely in the embedding space; The data stream identifier, data stream characteristics, and global identifier are concatenated to generate a unified standard token sequence, which serves as the starting point for the entire process.

5. The method as described in claim 1, characterized in that, The S2 includes: In the dynamic graph, type labels are added to nodes and edges to distinguish the various roles and relationships of base stations, links, services, and alarms; The edge weights are dynamically updated based on information about load changes and fault events, reflecting the strength of interaction and health status in real time. Output the topology snapshot sequence in chronological order to ensure that subsequent models can effectively identify the interaction structure.

6. The method as described in claim 1, characterized in that, The S3 includes: By leveraging spatial attention to aggregate neighborhood information within the same snapshot, local topological dependencies can be learned. An external graph memory mechanism is introduced to smoothly update node representations and reduce the interference of sudden anomalies on prediction. The stacked temporal Transformer processes snapshot sequences to capture long-range temporal correlations; at the same time, it integrates GCN layers to achieve spatial-temporal collaboration, and finally generates a unified latent vector for subsequent pre-training.

7. The method as described in claim 1, characterized in that, The S4 includes: A contrastive learning task is constructed, and the loss is estimated using information-noise contrast to improve the model's discriminative power; Set up reconstruction tasks to restore random masks or future fragments, and enhance information completion and generation capabilities; By employing a multi-task weighted mechanism to jointly optimize the individual losses, a highly generalizable pre-trained large model is obtained for use in telecommunications networks.

8. A device for constructing a multimodal pre-trained large model for telecommunications networks, characterized in that, include: The data preprocessing module is used to map multi-source heterogeneous operational data in the telecommunications network into discrete token sequences with shared semantic structures through unified tokenization processing. The dynamic heterogeneous graph construction module is used to construct dynamic heterogeneous graphs containing spatiotemporal topological information based on discrete token sequences. The spatiotemporal co-coding module is used to mine topological interaction dependencies in the spatial dimension of dynamic heterogeneous graphs through neighborhood aggregation, capture evolutionary patterns in the temporal dimension through temporal modeling, and introduce an external memory mechanism to suppress anomalous noise, so as to generate a unified representation that integrates spatiotemporal context. The network scheduling strategy generation module is used to perform self-supervised pre-training using the unified representation, and learn a general network representation applicable to a variety of downstream tasks through comparative learning and mask reconstruction tasks.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a method for constructing a multimodal pre-trained large model for telecommunications networks as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a method for constructing a multimodal pre-trained large model for telecommunications networks as described in any one of claims 1-7.

Images