A picture-text multi-modal fusion core network intelligent autonomy method and system

Abstract

This invention provides an intelligent autonomous method and system for a multimodal graph-text fusion core network, relating to the fields of mobile communication networks and artificial intelligence. Based on a spaceborne platform, this method constructs a unified multimodal fusion model to receive natural language instructions and a dynamic network topology graph, extracting graph structure features and text semantic features respectively. It utilizes a cross-attention mechanism to achieve bidirectional feature fusion between graph and text modalities, evaluates the dominance of each modality, and adaptively adjusts the configuration of self-attention and cross-modal attention heads. It combines MoE dynamic routing to select the target output head to generate task instructions. After secure verification in an isolated environment by a deterministic execution engine, it is executed through an atomic operation execution layer. Furthermore, a hierarchical federated learning framework is introduced to aggregate the parameter increments of various satellite models on the ground, achieving multi-satellite collaborative evolution. It also realizes graph-text joint inference, low-power multi-task processing, secure and reliable execution, and continuous model evolution, significantly improving the level of intelligence, reliability, and security.

Description

Technical Field

[0001] This invention relates to the fields of mobile communication networks and artificial intelligence technology, and in particular to an intelligent autonomous method and system for a multimodal fusion core network of graphics and text. Background Technology

[0002] With the development of 5G-Advanced and 6G non-terrestrial networks (NTN), satellite deployment of core network functions has become a key technological path to achieve seamless global coverage. In the TS 23.288 standard, 3GPP systematically defines various analysis scenarios for Network Data Analysis Function (NWDAF), including five major types: network performance analysis, anomaly analysis, user experience analysis, service experience analysis, and network load analysis, specifically covering multiple sub-scenarios such as prediction, diagnosis, optimization, and generation.

[0003] In related technologies, single-scenario AI models based on the standard NWDAF follow the 3GPP TS 23.288 specification and can deploy dedicated AI models for specific analysis scenarios (such as service experience analysis and network performance prediction). However, the independent training and deployment of each scenario model leads to: weak model generalization ability, inability to transfer knowledge across scenarios, low resource utilization, resource waste due to multiple models running in parallel, difficulty in co-evolution, and a lack of knowledge sharing mechanisms between models. Research has found that for different scenarios, the main focus is still on supporting diverse generalization capabilities such as prediction, diagnosis, optimization, and generation based on text, relationships, and time. Using a general large model is significant for improving the model's generalization ability, but current satellite core network operation and maintenance mainly relies on ground telemetry and remote control, periodically collecting status data through the OAM system, with the ground center performing analysis and decision-making. This approach suffers from severe latency, cannot meet real-time requirements, and the large amount of raw data transmission consumes valuable satellite-to-ground link resources. Furthermore, for satellite core networks in high-speed mobile scenarios, there is a lack of timely and effective prediction, analysis, and decision-making management capabilities to address various space challenges.

[0004] Directly applying general-purpose models (such as BERT and GPT variants based on the Transformer model) to core network operations and maintenance presents serious problems, primarily: a lack of domain knowledge, meaning the general-purpose model lacks understanding of professional knowledge such as the 3GPP protocol stack, network topology, and signaling state machines; excessively high computational resource requirements, even after pruning and quantization, the computational complexity of the standard Transformer's global attention mechanism is still O(n²), making it difficult to meet the real-time requirements of satellite systems; poor adaptability to graph structure data, meaning the core network is essentially a graph structure (network elements are nodes, interfaces are edges), while the Transformer is a native sequence model, which is insufficient in capturing graph structure information. While it excels at natural language understanding and generation, it has several inherent defects when handling structured relationship reasoning such as network topology and fault propagation. For example, topology information loss, i.e., forcibly serializing graph structure data results in the loss of connections between nodes; computational redundancy, i.e., performing global attention calculations on all node pairs in the network graph, while the actual impact is usually limited to a local area; and weak relationship reasoning ability, i.e., difficulty in understanding topology-based causal relationships such as "upstream failure leads to downstream congestion". While graph neural network-based analysis solutions excel in graph structure reasoning, they suffer from limitations in text understanding and generation capabilities. For example, they have weak natural language processing capabilities, making it difficult to understand natural language commands from operations personnel; limited configuration generation capabilities, failing to generate grammatically correct and readable configuration scripts; and difficulties in generating report documents, making it hard to automatically generate structured fault analysis reports. Simultaneous deployment of multiple models, such as using GNN models for graph analysis and Transformer models for text processing, while performing well in their respective tasks, suffers from serious problems. These include model fragmentation (multiple models are maintained independently, preventing knowledge sharing); resource waste (each model requires independent memory and computing resources); complex coordination (inconsistent outputs between models require complex coordination logic); and poor generalization ability (difficulty in handling complex scenarios requiring joint graph-text reasoning).

[0005] Furthermore, the spaceborne environment faces significant challenges due to resource constraints: severely limited computing, storage, and energy, and prolonged and unstable communication with ground operations centers. For example, resource consumption is enormous; traditional AI models (especially large models) incur huge computational and storage overhead, making them unsuitable for deployment and efficient operation on resource-constrained spaceborne platforms. Real-time performance is poor; all data must be transmitted to the ground for processing before decision-making instructions are transmitted back to the satellite. Limited by long latency and unstable space-to-ground links, this fails to meet real-time requirements such as rapid fault self-healing (e.g., millisecond / second-level). Collaboration is weak; each satellite node operates independently, lacking effective multi-satellite collaborative learning and knowledge-sharing mechanisms, resulting in limited capabilities for single-satellite AI models and an inability to learn and evolve from the experience of other satellites. Business coupling is high; different business scenarios (such as fault prediction and strategy generation) typically use independent models and data processing flows, leading to redundant resource consumption and complex model management.

[0006] If a lightweight machine learning solution is adopted directly, that is, using traditional machine learning algorithms such as decision trees and random forests, although the resource constraints are met, the ability to handle complex scenarios is sacrificed. That is, it cannot handle multimodal data fusion; it lacks dynamic adaptability, that is, static models are difficult to cope with network topology changes and business model evolution; and it lacks natural language interaction capabilities, making it unable to understand the natural language commands of operation and maintenance personnel.

[0007] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0008] This invention provides a method and system for intelligent autonomous operation of a multimodal fusion core network, which can solve the technical problems that related technologies are difficult to perform well in multiple types of tasks and are difficult to reduce resource consumption.

[0009] According to a first aspect of the present invention, a method for intelligent autonomy of a multimodal image-text fusion core network is provided, comprising: According to the selected working mode, the input data is preprocessed to obtain multiple input feature information, wherein the working mode includes one of fully automatic monitoring mode, ground command mode and hybrid mode; Based on the aforementioned working mode and the attention mechanism of the multimodal fusion unified model, the input feature information is processed to obtain the output feature information; Based on the output feature information, select the target output head of the multimodal fusion unified model; The output feature information is processed through the target output head to obtain task instructions; The task instructions are parsed through the preprocessing layer of the deterministic execution engine to obtain the task steps; The task steps are executed using a deterministic execution engine to obtain the execution result.

[0010] According to the present invention, when the working mode is fully automatic monitoring mode, the input data includes network element status data; Based on the selected working mode, the input data is preprocessed to obtain multiple input feature information, including: Based on the network element status data and the traffic attribute information of the data interface, a dynamic network topology diagram is constructed. The graph coding layer of the input layer of the multimodal fusion unified model is used to process the nodes of the dynamic network topology graph to obtain the initial node feature information of the dynamic network topology graph. The initial node feature information overlay image position is encoded to obtain node feature information; Edge relationship feature information is obtained by processing the edges of the dynamic network topology graph through the edge relationship embedding layer of the input layer of the multimodal fusion unified model. The input feature information is obtained based on node feature information and edge relationship feature information; Alternatively, the input data may include ground text input data; Based on the selected working mode, the input data is preprocessed to obtain multiple input feature information, including: The text encoding layer of the input layer of the multimodal fusion unified model processes the word vectors corresponding to multiple word segments of the ground text input data to obtain initial word segmentation feature information. Text position encoding is added to the initial word segmentation feature information to obtain the input feature information.

[0011] According to the present invention, based on the working mode and the attention mechanism of the multimodal fusion unified model, the input feature information is processed to obtain the output feature information, including: The attention mechanism is used to fuse the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data to obtain the first output feature information. The attention mechanism is used to fuse the input feature information corresponding to the ground text input data with the input feature information corresponding to the dynamic network topology graph to obtain the second output feature information. The output feature information is obtained based on the first output feature information and the second output feature information.

[0012] According to the present invention, the first output feature information is obtained by fusing the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data through the attention mechanism, including: Through the attention mechanism, the first key matrix and the first weight matrix of the input feature information corresponding to the dynamic network topology graph are obtained, and the first query matrix of the input feature information corresponding to the ground text input data is obtained through the attention mechanism. Based on the first query matrix, the first key-value matrix, and the first weight matrix, the first output feature information is obtained.

[0013] According to the present invention, the input feature information corresponding to the ground text input data is fused to the input feature information corresponding to the dynamic network topology graph through the attention mechanism to obtain the second output feature information, including: Through the attention mechanism, the second key matrix and the second weight matrix of the input feature information corresponding to the ground text input data are obtained, and the second query matrix of the input feature information corresponding to the dynamic network topology graph is obtained through the attention mechanism. Based on the second query matrix, the second key matrix, and the second weight matrix, the second output feature information is obtained.

[0014] According to the present invention, obtaining the output feature information based on the first output feature information and the second output feature information includes: Obtain the first information entropy of the input feature information corresponding to the ground text input data, and obtain the second information entropy of the input feature information corresponding to the dynamic network topology graph; Based on the first information entropy and the second information entropy, the text dominance score and the graph dominance score are determined respectively; The text value contribution and graph value contribution are determined based on the static contribution ratio related to the task type, as well as the text dominance score and graph dominance score. The fusion mode is determined based on the contribution of text value and graph value. Based on the fusion mode, determine the number of text self-attention heads, the number of graph self-attention heads, and the number of cross-modal attention heads; Output feature information is obtained based on the number of text self-attention heads, the number of graph self-attention heads, and the number of cross-modal attention heads, as well as the first output feature information and the second output feature information.

[0015] According to the present invention, selecting the target output head of the multimodal fusion unified model based on the output feature information includes: The basic probabilities of each output head are obtained by using the first activation function and output feature information of the task type classifier; The refined probabilities of each output head are obtained by using the second activation function of the task type classifier and the input data. The base probability and the refined probability are weighted and fused to obtain the fused probability of each output head; The decision probability of each output head is obtained by using the first activation function of the task type classifier and the fusion probability. The target output head is selected based on the decision probability.

[0016] According to the present invention, the method further includes: Determine the task's criticality level based on the input feature information; High-criticality tasks use half-precision floating-point numbers, medium-criticality tasks use mixed precision, and low-criticality tasks use 8-bit integer quantization precision.

[0017] According to the present invention, the task steps are executed by a deterministic execution engine to obtain the execution result, including: The task steps are executed in an isolated simulation environment to obtain the simulation results; If the simulation results are safe, the task steps are executed through the atomic operation execution layer of the deterministic execution engine to obtain the execution results; Generate a contingency plan for reverse operations; When the simulation execution result is risky, the rollback mechanism will be automatically triggered according to the aforementioned reverse operation plan.

[0018] According to the present invention, the method further includes: The basic model is pre-trained using the processor in the ground command center to obtain the pre-trained basic model. Knowledge distillation is performed on the pre-trained base model to construct a multimodal unified fusion model; The multimodal unified fusion model was uploaded to the satellite; By using preset training objectives and training resources, incremental training is performed on the multimodal unified fusion model stored in multiple satellites to obtain an optimized multimodal unified fusion model. Obtain the parameter difference information between the optimized multimodal unified fusion model and the multimodal unified fusion model; Each satellite uploads encrypted parameter increments processed with differential privacy, and the ground aggregation center performs aggregation calculations in the encrypted state to obtain the updated model set; The trained multimodal unified fusion model is distributed according to the differences in satellite type.

[0019] According to the present invention, a pre-trained basic model is obtained by pre-training a basic model through a processor in a ground command center, including: By processing multimodal training data through a basic model, textual semantic information, graph node feature information, and training task instructions are obtained. The loss function is obtained based on the annotation information, the text semantic information, the graph node feature information, and the training task instructions; The base model is pre-trained based on the loss function to obtain the pre-trained base model.

[0020] According to the present invention, a loss function is obtained based on the annotation information, the text semantic information, the graph node feature information, and the training task instructions, including: Based on the annotation information and the text semantic information, a semantic prediction loss function is obtained; Based on the annotation information and the graph node feature information, a graph feature prediction loss function is obtained; Based on the graph node feature information and text semantic information, the alignment loss function is obtained; Based on the annotation information and training task instructions, the task prediction loss function is obtained; The loss function is obtained based on the semantic prediction loss function, the graph feature prediction loss function, the alignment loss function, and the task prediction loss function.

[0021] According to the present invention, each satellite uploads encrypted parameter increments processed by differential privacy, and the ground aggregation center performs aggregation calculations in encrypted form to obtain an updated model set, including: For the parameters of the text encoding layer in the multimodal fusion unified model, a global averaging strategy is used for aggregation; For the parameters of the graph coding layer and edge relation embedding layer of the multimodal fusion unified model, a global averaging strategy is used for aggregation; The parameters of the output head of the multimodal fusion unified model are aggregated in a personalized manner according to the task type.

[0022] According to a second aspect of the present invention, a graphics-text multimodal fusion core network intelligent autonomous system is provided, comprising: The preprocessing module is used to preprocess the input data according to the selected working mode to obtain multiple input feature information, wherein the working mode includes one of fully automatic monitoring mode, ground command mode and hybrid mode; The output feature module is used to process the input feature information according to the working mode and the attention mechanism of the multimodal fusion unified model to obtain the output feature information; The selection module is used to select the target output head of the multimodal fusion unified model based on the output feature information. The task instruction module is used to process the output feature information through the target output head to obtain task instructions; The task step module is used to parse task instructions and obtain task steps through the preprocessing layer of the deterministic execution engine; The execution result module is used to execute the task steps through a deterministic execution engine and obtain the execution result.

[0023] According to a third aspect of the present invention, a smart autonomous device for a multimodal graphics fusion core network is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the smart autonomous method for the multimodal graphics fusion core network.

[0024] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the intelligent autonomous method for a multimodal fusion core network of graphics and text.

[0025] By adopting the above technical solution, the present invention can achieve the following technical effects: The proposed intelligent autonomous core network method for graph-text multimodal fusion in this application can introduce natural language as input data, freeing ground maintenance personnel from complex command-line configuration of onboard equipment, reducing the maintenance threshold and human error risk of the NTN core network. By preprocessing the input data and using the attention mechanism of the unified multimodal fusion model, the multimodal input data is fused, thereby transforming the multimodal input information into unified output feature information. This adapts to multimodal tasks such as graph structure reasoning, natural language understanding and generation, while avoiding the resource waste and coordination difficulties caused by deploying dedicated models, avoiding modal fragmentation, realizing knowledge sharing and transfer between text understanding capabilities and graph analysis capabilities, improving the model's generalization ability and the collaboration between satellite nodes.

[0026] Furthermore, the intelligent autonomous method for the image-text multimodal fusion core network proposed in this application uses an independent multimodal fusion unified model processing approach, which can reduce resource consumption, improve processing real-time performance, adapt to the environment of strictly limited on-board resources, realize efficient model deployment and real-time inference, and improve the reliability, accuracy and security of AI decision-making.

[0027] The graph-text multimodal fusion core network intelligent autonomous method of this application can achieve unified embedding processing of multimodal data through an embedding layer, enabling the unified multimodal fusion model to be compatible with graph data (network topology, signaling flow), text data (configuration scripts, operation and maintenance logs, natural language commands), and time series data (performance indicators, load time series), providing a basic architecture for deep fusion and deep interactive reasoning of multimodal data. Therefore, without building a separate model for each individual task or adding a dedicated time series module, it utilizes the native time series capabilities of Transformer and GNN time series graph representation to achieve full coverage support for diverse core network operation and maintenance tasks such as analysis, diagnosis, prediction, generation, optimization, governance, and decision-making, avoiding model fragmentation, improving generalization ability, and reducing resource consumption of parallel operation of multiple models.

[0028] The intelligent autonomous method for the graph-text multimodal fusion core network proposed in this application can also fuse the input feature information of the two modalities through the use of a cross-attention mechanism twice, thereby obtaining richer fused feature information. This is beneficial for multimodal feature representation, enhances the applicability of the model, improves the generalization ability of the model, and realizes knowledge sharing and transfer between text understanding ability and graph analysis ability. This allows the domain knowledge learned by the model from the text to improve the graph analysis ability, and vice versa, enabling the model to adapt to more types of tasks.

[0029] The intelligent autonomous method for the graph-text multimodal fusion core network in this application can dynamically evaluate the dominance of text and graph structures in a task based on the information entropy of the input feature information. Based on this dominance, it determines the fusion mode and allocates attention heads, thereby determining the information content of the output feature information corresponding to various input data. This makes the information content more closely match the task type, providing richer and more effective reference information for task execution and improving the accuracy of task execution. Furthermore, by selecting target output heads, it can achieve computation compatible with multiple tasks under the strictly limited computing, storage, and power resources of spaceborne networks. Through the MoE architecture, appropriate expert models can be selected for computation without activating all neural networks, significantly saving computing resources. Moreover, when selecting target output heads, selection can be based on both the characteristics of the output feature information itself and the characteristics of the input data. For specific tasks, keywords can be predefined in the input data to assist in selection, improving the flexibility and accuracy of target output head selection and making the target output heads more suitable for specific tasks.

[0030] The intelligent autonomous method for the multimodal fusion core network proposed in this application can reliably transform the probabilistic inference results of the unified multimodal fusion model into deterministic network operations, ensuring system security. All AI inference task instructions are executed only after security verification, greatly guaranteeing the validity of the results. Feedback on the results further enhances the model's learning ability. Furthermore, under strict resource constraints, through thorough layered decoupling, probabilistic intelligent inference and deterministic network operations are separated, ensuring both the flexibility of intelligent decision-making and the reliability of network operations. During training, ground-based full training, knowledge distillation, and layered federated learning can be used to keep the single model simple, reduce computational resource consumption, and achieve continuous model evolution without on-board full retraining, while protecting the local data privacy and business characteristics of each satellite.

[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort. Figure 1 An exemplary flowchart of an intelligent autonomous method for a multimodal fusion core network based on an embodiment of the present invention is shown. Figure 2 An exemplary schematic diagram of the operational architecture of the intelligent autonomous method for a multimodal fusion core network according to an embodiment of the present invention is shown. Figure 3 A schematic diagram of a multimodal fusion unified model according to an embodiment of the present invention is shown as an example; Figure 4 A schematic diagram of a deterministic execution engine according to an embodiment of the present invention is shown as an example; Figure 5 An exemplary schematic diagram of training a multimodal unified fusion model according to an embodiment of the present invention is shown; Figure 6 An exemplary schematic diagram illustrating the application of the intelligent autonomous method for a multimodal fusion core network according to an embodiment of the present invention is shown. Figure 7 A block diagram of an intelligent autonomous system for a multimodal fusion core network based on a text and image representation according to an embodiment of the present invention is shown as an example. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0035] Figure 1 An exemplary flowchart illustrates a method for intelligent autonomy in a multimodal fusion core network according to an embodiment of the present invention, the method comprising: Step S1: Preprocess the input data according to the selected working mode to obtain multiple input feature information, wherein the working mode includes one of fully automatic monitoring mode, ground command mode and hybrid mode; Step S2: Based on the working mode and the attention mechanism of the multimodal fusion unified model, the input feature information is processed to obtain the output feature information; Step S3: Select the target output head of the multimodal fusion unified model based on the output feature information; Step S4: Process the output feature information through the target output head to obtain task instructions; Step S5: The task instructions are parsed through the preprocessing layer of the deterministic execution engine to obtain the task steps; Step S6: The task steps are executed using a deterministic execution engine to obtain the execution result.

[0036] The intelligent autonomous method for multimodal fusion core network according to embodiments of the present invention can introduce natural language as input data, freeing ground maintenance personnel from complex command-line configuration of spaceborne equipment, reducing the maintenance threshold and human error risk of NTN core network. By preprocessing the input data and using the attention mechanism of the multimodal fusion unified model, the multimodal input data is fused, thereby transforming the multimodal input information into unified output feature information. This adapts to multimodal tasks such as graph structure reasoning, natural language understanding and generation, and avoids the resource waste and coordination difficulties caused by deploying dedicated models. It avoids modal fragmentation, realizes knowledge sharing and transfer between text understanding and graph analysis capabilities, improves the generalization ability of the model and the synergy between satellite nodes. Furthermore, the processing method using an independent multimodal fusion unified model can reduce resource consumption, improve processing real-time performance, adapt to environments with strictly limited spaceborne resources, realize efficient deployment and real-time reasoning of the model, and improve the reliability, accuracy and security of AI decision-making.

[0037] According to one embodiment of the present invention, in view of the objective constraints such as the diversification of tasks and the strict limitation of spaceborne resources, the technical problems of natural language text recognition and generation are solved based on the lightweight Transformer architecture. The graph neural network (such as HGNN) is used to analyze the relationship state diagram of core network elements, the signaling flow relationship diagram and similar diagrams that can be represented as relationship diagrams. The multimodal fusion method is adopted to avoid the problems of resource waste, coordination difficulties and weak generalization ability caused by parallel single models. In the case of spaceborne resources, the expert model is dynamically activated by combining MoE to reduce the number of inference parameters, thereby reducing resource consumption and improving inference speed and real-time performance.

[0038] Figure 2 An exemplary schematic diagram of the operational architecture of the intelligent autonomous method for a multimodal fusion core network based on an embodiment of the present invention is shown.

[0039] According to one embodiment of the present invention, the core network functions may include Operation Management and Maintenance (OAM), Authentication Management Function (AMF), Message Format Standard (SMF), Policy Control Function (PCF), Unified Data Management (UDM), and User Plane Data Processing Function (UPF), etc., and can generate multimodal data such as configuration management data, interface data, log alarm data, policy information, software management data, user terminal context, device management data, session context, performance statistics data, other contexts, and other OAM data. Some of the data can be composed of graph structures, while some of the data is in text form.

[0040] According to one embodiment of the present invention, the input data can be of various types, such as text, graph structures, time series data, etc. Without explicitly specifying the task type, it can be adaptively processed through multimodal fusion. After unified preprocessing (e.g., processing using encoding layers), data from different modalities with the same meaning can obtain unified feature information, thereby aiding the model's understanding and unified processing of multimodal information. This eliminates the need for multiple parallel single-task models, enabling accurate processing and inference of different modalities in various tasks. Furthermore, based on the sequence modeling capabilities of Transformer, in addition to processing text and graph structures themselves, it can also process time-series graph structures combined with time series, further enriching task types and improving model generalization ability. Moreover, deep interaction of feature information from different modalities can be achieved at multiple network layers, allowing for full fusion of various information and richer, more accurate feature representation. This enhances and expands the capabilities of NWDAF, adding powerful intelligent analysis and autonomous execution capabilities to standardized NWDAF through multimodal fusion AI technology, while maintaining full compatibility with the existing 3GPP core network architecture. It can perform a variety of NWDAF tasks, such as analysis, diagnosis, prediction, generation (e.g., configuration strategy), optimization, governance, and decision-making.

[0041] In the examples, for network performance analysis tasks, graph modalities can be used to analyze topology load distribution, and text modalities can be used to generate performance reports. For user experience analysis tasks, graph modalities can be used for UE-QoS correlation analysis, and text modalities can be used to generate descriptions of experience issues. For abnormal behavior detection tasks, graph modalities can be used for abnormal pattern recognition, and text modalities can be used for anomaly description and root cause analysis. For congestion analysis tasks, graph modalities can be used for traffic hotspot detection, and text modalities can be used to output congestion mitigation suggestions. For quality of service prediction tasks, graph modalities can be used for time-series prediction, and text modalities can be used to output interpretations of the prediction results. For service assurance analysis tasks, graph modalities can be used for SLA achievement analysis, and text modalities can be used to generate assurance measures.

[0042] Figure 3 A schematic diagram of a multimodal fusion unified model according to an embodiment of the present invention is shown as an example.

[0043] According to one embodiment of the present invention, the multimodal fusion unified model is deployed on a high-reliability computing module of a satellite platform, comprising a dedicated AI acceleration chip (GPU), a general-purpose processor (CPU), and a programmable logic unit (FPGA).

[0044] According to one embodiment of the present invention, in step S1, the multimodal data includes text data and graph structure data. The text data sources include OAM (Operation, Administration, and Maintenance) data interfaces, which can collect network element status, performance indicators, and alarm information; 3GPP interfaces, which can collect signaling messages and dynamic context information from standard interfaces such as N1, N2, and N4; service data interfaces, which can collect user plane traffic characteristics and service quality indicators; and user manuals, command manuals, fault work orders, and operation and maintenance manuals. The graph structure data sources include network topology diagrams, load characteristic diagrams, signaling sequence diagrams, resource dependency diagrams, and fault propagation diagrams. The graph structures include the following types: raw data graph structures, such as real-time monitoring data flow graphs, real-time device status change graphs, and business flow tracing graphs; feature extraction graph structures, such as load feature graphs (mean, trend, fluctuation), anomaly pattern graphs (outliers, mutation points), and correlation analysis graphs (device correlation, business correlation); pattern recognition graph structures, such as load pattern graphs (high / medium / low load patterns), time pattern graphs (periodicity, trend), which model history, current, and future through time series graphs, and spatial pattern graphs (regional distribution, hotspot distribution), which model the relationships and influences between devices through topology graphs; and decision support graph structures, such as condition satisfaction graphs (the state of satisfaction of various conditions), constraint compliance graphs (the degree of compliance of various constraints), and solution evaluation graphs (the evaluation results of each solution).

[0045] According to one embodiment of the present invention, multimodal data can be preprocessed, for example, signaling data, performance data, configuration data, etc., to obtain a graph structure, or text data can be obtained through log alarms and configuration requirements. Multimodal input data can be preprocessed to obtain input feature information, which can be used for analysis to obtain analysis results and execution strategies. A multimodal fusion unified model is used to analyze and reason about multimodal data and obtain execution strategies. A deterministic execution engine can be used to execute the execution strategies. In the example, the functions of the multimodal fusion unified model include language understanding, intent analysis, obtaining analysis results, and outputting strategies. Its architecture includes a multimodal fusion unified model obtained by organizing and fusing Transformer, GNN, and MoE in a specific way, which can be used to analyze and reason about text data and graph structures, and output analysis results and execution strategies.

[0046] According to one embodiment of the present invention, the operating modes include a fully automatic monitoring mode, a ground command mode, and a hybrid mode.

[0047] According to an embodiment of the present invention, when the working mode is fully automatic monitoring mode, the input data includes network element status data; according to the selected working mode, the input data is preprocessed to obtain multiple input feature information, including: constructing a dynamic network topology graph based on the network element status data and the traffic attribute information of the data interface; processing the nodes of the dynamic network topology graph through the graph coding layer of the input layer of the multimodal fusion unified model to obtain initial node feature information of the dynamic network topology graph; superimposing graph position encoding on the initial node feature information to obtain node feature information; processing the edges of the dynamic network topology graph through the edge relationship embedding layer of the input layer of the multimodal fusion unified model to obtain edge relationship feature information; and obtaining the input feature information based on the node feature information and the edge relationship feature information.

[0048] According to one embodiment of the present invention, in fully automatic monitoring mode, the input data is a graph structure. OAM, AMF, SMF, PCF, UDM, and UPF can be identified as nodes in the graph structure, and the network element status data of these core network elements (e.g., CPU usage, memory usage, number of sessions, etc.) are used as node features. The data interfaces between these core network elements are used as edges in the graph structure, and the traffic attribute information of the data interfaces (e.g., latency, link quality, etc.) are used as edge features. A dynamic network topology graph can then be constructed based on the node and node features, as well as the edge and edge features. In fully automatic monitoring mode, periodic monitoring can be performed, for example, constructing a dynamic network topology graph every 5 seconds (this time is only an example and can be configured to other times as needed), and processing it through a multimodal fusion unified model. Alternatively, a trigger event can be set, for example, constructing a dynamic network topology graph when a set key indicator (e.g., one or more network element status data exceed a threshold). The multimodal fusion unified model can combine the current dynamic network topology graph and the dynamic network topology graphs from the last 30 minutes for comprehensive analysis to obtain analysis results and output strategies. For example, this working mode can be used for tasks such as fault prediction, performance optimization, anomaly detection, and resource scheduling. It can analyze whether the core network functions are abnormal and the impact of the anomalies (i.e., the scope of fault propagation and the impact on services), and generate repair strategies, such as generating preventive self-healing solutions.

[0049] According to one embodiment of the present invention, after obtaining a dynamic network topology graph, the nodes of the dynamic network topology graph can be processed through the graph encoding layer of the input layer of a multimodal fusion unified model to obtain initial node feature information of the dynamic network topology graph. For example, the embedding layer may include a network hierarchy based on a hypergraph model, which can process the dynamic network topology graph to obtain initial node feature information for each node. After superimposing graph position encoding on the initial node feature information, node feature information can be obtained. Further, the edges of the dynamic network topology graph can also be processed through an edge relationship embedding layer to obtain edge relationship feature information. For example, based on the edge features, the relationship between nodes and edges can be determined, thereby aggregating node feature information into edges to obtain edge relationship feature information. Further, according to the connection relationship between each node and each edge, the edge relationship feature information is distributed to the nodes. After fully connected and activation processing, the input feature information corresponding to each node can be obtained, i.e., the... Figure 3 The graph information transmission feature sequence.

[0050] According to an embodiment of the present invention, when the working mode is ground command mode, the input data includes ground text input data; according to the selected working mode, the input data is preprocessed to obtain multiple input feature information, including: processing the word vectors corresponding to multiple word segments of the ground text input data through the text encoding layer of the input layer of the multimodal fusion unified model to obtain initial word segmentation feature information; adding text position encoding to the initial word segmentation feature information to obtain the input feature information.

[0051] According to one embodiment of the present invention, in the ground command mode, the input data is ground text input data, such as signaling data transmitted from the ground. After segmenting the ground text input data, word embedding can be performed on each segment to obtain word vectors. The word vectors are then mapped through a text encoding layer to obtain initial segmentation feature information. Text position encoding is then added to obtain the input feature information, i.e., Figure 3 The text feature sequence in the data. The multimodal fusion unified model can analyze and reason about the input feature information by combining the dynamic network topology map of the current and / or historical moments (e.g., analyzing the core network elements involved in the ground text input data), and obtain analysis results and output strategies. For example, this working mode can be used for service activation, policy configuration, problem query, and manual intervention guidance.

[0052] According to an embodiment of the present invention, in the hybrid mode, the input data includes the input data of the above two modes, and the method of obtaining the input feature information is the same as that of the above two modes. That is, the features of the text token and the graph node can be embedded through the embedding layer to obtain their respective input feature information.

[0053] In this way, unified embedding processing of multimodal data can be achieved through the embedding layer, enabling the unified multimodal fusion model to be compatible with graph data (network topology, signaling flow), text data (configuration scripts, operation and maintenance logs, natural language commands), and time series data (performance indicators, load time series), providing the infrastructure for deep fusion and deep interactive reasoning of multimodal data. Therefore, without building a separate model for each individual task or adding a dedicated time series module, it leverages the native time series capabilities of Transformer and GNN time series graph representation to achieve full coverage support for diverse core network operation and maintenance tasks such as analysis, diagnosis, prediction, generation, optimization, governance, and decision-making, avoiding model fragmentation, improving generalization ability, and reducing the resource consumption of running multiple models in parallel.

[0054] According to an embodiment of the present invention, in step S2, it can be achieved through... Figure 3 The attention mechanism of the hierarchical transformer in the model processes the input feature information, thereby achieving deep fusion and inference of multimodal data. If the input data only includes ground text input data, the corresponding input feature information is the input feature information corresponding to the text. This can be processed through the text self-attention mechanism to obtain the output feature information. Feature extraction and temporal enhancement can be performed on the input feature information of the ground text input data to obtain the output feature information that reflects the semantic and logical characteristics of the ground text input data. If the input data only includes network element state data, the input feature information is the input feature information corresponding to the dynamic network topology graph. This can be processed through the graph information transfer attention mechanism (still a self-attention mechanism) to obtain the output feature information. Temporal feature extraction and spatiotemporal attention feature extraction can be performed on the dynamic network topology graph to enhance its temporal and logical characteristics, facilitating analysis and inference.

[0055] According to one embodiment of the present invention, if the input data is multimodal data, that is, the input feature information includes both input feature information corresponding to text and input feature information corresponding to the dynamic network topology graph, a cross-modal attention mechanism can be used for processing. This cross-modal attention mechanism is a cross-attention mechanism, which can be used to fuse multimodal input feature information. Furthermore, while using this cross-attention mechanism, the aforementioned self-attention mechanism can also be used to process the input feature information corresponding to the ground text input data and the input feature information corresponding to the dynamic network topology graph, respectively, to obtain their respective output feature information, which is then combined with the output feature information obtained by the cross-attention mechanism for subsequent processing.

[0056] According to an embodiment of the present invention, input feature information is processed to obtain output feature information based on the attention mechanism of the working mode and the multimodal fusion unified model. This includes: fusing input feature information corresponding to the dynamic network topology graph with input feature information corresponding to ground text input data through the attention mechanism to obtain first output feature information; fusing input feature information corresponding to ground text input data with input feature information corresponding to the dynamic network topology graph through the attention mechanism to obtain second output feature information; and obtaining the output feature information based on the first output feature information and the second output feature information. When the input data is multimodal data, the attention mechanism is the aforementioned cross-attention mechanism, which can perform feature fusion on the input feature information corresponding to the multimodal input data. In the example, the two types of input information can be fused together, that is, the input feature information corresponding to the dynamic network topology graph is fused with the input feature information corresponding to the ground text input data, and the input feature information corresponding to the ground text input data is fused with the input feature information corresponding to the dynamic network topology graph to obtain fused feature information from both aspects, i.e., output feature information.

[0057] According to one embodiment of the present invention, the first output feature information is obtained by fusing the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data through the attention mechanism. The method includes: obtaining a first key-value matrix and a first weight matrix of the input feature information corresponding to the dynamic network topology graph through the attention mechanism, and obtaining a first query matrix of the input feature information corresponding to the ground text input data through the attention mechanism; and obtaining the first output feature information based on the first query matrix, the first key-value matrix and the first weight matrix.

[0058] According to one embodiment of the present invention, for each attention head of the attention mechanism, a first key-value matrix can be obtained by performing matrix multiplication between the first key-value projection matrix of the attention head and the input feature information corresponding to the dynamic network topology graph. Similarly, a first weight matrix can be obtained by performing matrix multiplication between the first weight projection matrix of the attention head and the input feature information corresponding to the dynamic network topology graph. Furthermore, a first query matrix can be obtained by performing matrix multiplication between the first query matrix and the first key-value matrix, dividing by the square root of the dimension of each key-value vector in the first key-value matrix, and then processing through a softmax activation function. The result is then multiplied by the first weight matrix. Each vector in the resulting matrix is ​​the first output feature information corresponding to the ground text input data. In other words, by fusing the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data to obtain the first output feature information, the semantic and logical understanding of the ground text input data can be enhanced through the features of the dynamic network topology graph.

[0059] According to one embodiment of the present invention, the second output feature information is obtained by fusing the input feature information corresponding to the ground text input data with the input feature information corresponding to the dynamic network topology graph through the attention mechanism. This includes: obtaining a second key-value matrix and a second weight matrix of the input feature information corresponding to the ground text input data through the attention mechanism, and obtaining a second query matrix of the input feature information corresponding to the dynamic network topology graph through the attention mechanism; and obtaining the second output feature information based on the second query matrix, the second key-value matrix, and the second weight matrix.

[0060] According to one embodiment of the present invention, for each attention head of the attention mechanism, a second key-value matrix can be obtained by performing matrix multiplication between the second key-value projection matrix of the attention head and the input feature information corresponding to the ground text input data. Similarly, a second weight matrix can be obtained by performing matrix multiplication between the second weight projection matrix of the attention head and the input feature information corresponding to the ground text input data. Furthermore, a second query matrix can be obtained by performing matrix multiplication between the second query matrix and the second key-value matrix, dividing by the square root of the dimension of each key-value vector in the second key-value matrix, and then processing through a softmax activation function. The result is then multiplied by the second weight matrix. Each vector in the resulting matrix is ​​the second output feature information corresponding to the dynamic network topology. In other words, the second output feature information is obtained by fusing the input feature information corresponding to the ground text input data with the input feature information corresponding to the dynamic network topology.

[0061] In this way, the input feature information of the two modalities can be fused together through the use of cross-attention mechanism, resulting in richer fused feature information. This is beneficial for multimodal feature representation, enhances the applicability of the model, improves the model's generalization ability, and realizes knowledge sharing and transfer between text understanding ability and graph analysis ability. This allows the domain knowledge learned by the model from the text to improve graph analysis ability, and vice versa, enabling the model to adapt to more types of tasks.

[0062] According to an embodiment of the present invention, obtaining the output feature information based on the first output feature information and the second output feature information includes: obtaining a first information entropy of the input feature information corresponding to the ground text input data, and obtaining a second information entropy of the input feature information corresponding to the dynamic network topology graph; determining the text dominance score and the graph dominance score based on the first information entropy and the second information entropy, respectively; determining the text value contribution and the graph value contribution based on the static contribution ratio related to the task type, as well as the text dominance score and the graph dominance score; determining the fusion mode based on the text value contribution and the graph value contribution; determining the number of text self-attention heads, the number of graph self-attention heads, and the number of cross-modal attention heads based on the fusion mode; and obtaining the output feature information based on the number of text self-attention heads, the number of graph self-attention heads, the number of cross-modal attention heads, the first output feature information, and the second output feature information.

[0063] According to an embodiment of the present invention, as described above, each attention head can perform the above processing to obtain first output feature information and second output feature information, thereby determining the number of attention heads for self-attention mechanism and cross-attention mechanism. Based on different attention heads, the above processing is performed respectively to obtain the output feature information of multiple attention heads corresponding to ground text input data, the output feature information of multiple attention heads corresponding to dynamic network topology graph, and the above-mentioned cross-modal output feature information of multiple attention heads.

[0064] According to an embodiment of the present invention, obtaining the output feature information based on the first output feature information and the second output feature information includes: obtaining the first information entropy of the input feature information corresponding to the ground text input data, obtaining the second information entropy of the input feature information corresponding to the dynamic network topology graph, and determining the text dominance score and the graph dominance score according to the following formulas (1) and (2). I text = α1× H text_norm (h t ) + α2× D sem (1) I graph = β1× H graph_norm (hg ) + β2× Q norm (2) Among them, I text H represents the text dominance score. text_norm (h t ) is the first information entropy, calculated as the Shannon entropy after normalizing the text embedding vector; I graph H represents the dominance score of the graph. graph_norm (h g D is the second information entropy, calculated as the Shannon entropy after normalizing the graph embedding vectors; sem Semantic density, defined as the ratio of the number of non-stop words to the total number of words; Q norm The modularity of a graph reflects its network structure information. Mapping the modularity Q to the [0,1] interval yields the normalized graph structure information; α1, α... 2、 β 1、 β2 is a weighting coefficient that satisfies α1+α2=1 and β1+β2=1.

[0065] According to an embodiment of the present invention, the above text dominance score and graph dominance score can be normalized before subsequent calculations to ensure that the order of magnitude of each parameter is uniform. In the example, the following formulas (3) and (4) can be used for normalization to obtain the normalized text information density I. textnorm Information density of graph I graphnorm The normalization method uses standard Min-max normalization, such as: I text_norm = clamp((I text - L text ) / (U text - L text ),0,1)(3) I graph_norm = clamp((I graph - L graph ) / (U graph - L graph ),0,1)(4) Among them, U text and L text The text dominance score is I. text Statistical maximum and minimum values ​​on the multimodal training dataset; U graph and L graph Dominance score I graph The statistical maximum and minimum values ​​on the multimodal training dataset; clamp(x,0,1) means restricting x to the interval [0,1].

[0066] According to one embodiment of the present invention, the static contribution ratio related to the task type is defined as a preset ratio (such as one set based on historical task data analysis or domain expert experience), conforming to V text （i） + V graph （i） = 1, for example, for a fault diagnosis task, the static contribution ratio V of the text is 1. text （i） =0.4, the static contribution ratio V of the graph structure graph （i） =0.6; For configuration generation tasks, the static contribution ratio V of the text is 0.6. text （i） =0.7, the static contribution ratio V of the graph structure graph （i） =0.3; For performance prediction tasks, the static contribution ratio V of the text is... text （i） =0.3, the static contribution ratio V of the graph structure graph （i） =0.7; For report generation tasks, the static contribution ratio V of the text is 0.7. text （i） =0.8, the static contribution ratio V of the graph structure graph （i） h =0.2; For resource optimization tasks, the static contribution ratio V of the text is... text （i） =0.4, the static contribution ratio V of the graph structure graph （i） =0.6.

[0067] According to one embodiment of the present invention, a "task prior selection mechanism" is introduced to calculate the value contribution of all candidate tasks in parallel, and to calculate the corresponding text value contribution I for each candidate task type i. text value （i） Hetu Value Contribution I graph value （i） : I text value （i） = I textnorm ×V text （i） (5) I graph value （i） = I graphnorm ×V graph （i） (6) Using all the task results obtained from formulas (5) and (6), construct the task adaptation score function: S（i） = w1.I text value （i） + w2.I graph value （i） Select the task with the highest score (argmax) i S （i） As the "dominant task path", and with its corresponding I text value （i） and I graph value （i） The contribution of the branch as the optimal text value (I) text value ) and graph value contribution (I graph value This is used for subsequent attention head allocation.

[0068] Subsequently, based on the I issued by this task text value and I graph value Continue executing formulas (7) to (11) to obtain the optimal attention head allocation scheme and its fusion feature representation for the task. This process is executed in parallel for all candidate tasks, generating fusion path scores for multiple tasks, which serve as one of the reference bases for subsequent task routing.

[0069] According to one embodiment of the present invention, a textual bias score and a graph bias score can be determined based on the textual value contribution and the graph value contribution, which are used to determine the dominant fusion method. In the example, the textual bias score T can be determined according to the following formulas (7) and (8). Dominance Preference score G Dominance And satisfy T Dominance + G Dominance = 1 when I text value +I graph value ≤ 10 -6 T Dominance =G Dominance = 0.5.

[0070] T Dominance = (I text value / (I text value + I graph value (7) G Dominance = (I graph value / (I text value + I graph value (8) If the text bias score is greater than the graph bias score, the fusion mode is a text-dominated fusion mode; if the text bias score is less than the graph bias score, the fusion mode is a graph-dominated fusion mode; if the text bias score is equal to the graph bias score, the fusion mode is a balanced fusion mode.

[0071] According to one embodiment of the present invention, the number N of text self-attention heads can be dynamically determined based on text valoresistive scores and graph valoresistive scores. text The number of attention heads N in the graph graph and the number of cross-modal attention heads N cross That is, determining the allocation of attention heads. In the example, the number of text self-attention heads, the number of graph self-attention heads, and the number of cross-modal attention heads can be determined according to the following formulas (9), (10), and (11). N text = max(1, floor(B × sigmoid(k × (T Dominance - 0.5)))(9) N graph = max(1, floor(B × sigmoid(k ×(G Dominance - 0.5))))(10) N cross = max(1, B - N text - N graph (11) Where B is the preset total number of attention heads in the multimodal fusion unified model, with a value of 8 or 16; k is an adaptive adjustment coefficient used to control the steepness of the sigmoid function; sigmoid(.) is the standard sigmoid activation function, and floor(.) represents the floor function; max(1,.) ensures that at least one self-attention head is retained; when the number of cross-modal attention heads is less than 1, the number of text self-attention heads and graph self-attention heads are compressed proportionally to ensure that the number of cross-modal attention heads is at least 1, so as to guarantee the intermodal interaction capability. After the above allocation of attention heads, multiple text self-attention heads can be used to obtain the output feature information corresponding to the ground text input data, multiple graph self-attention heads can be used to obtain the output feature information corresponding to the dynamic network topology graph, and multiple cross-modal attention heads can be used to obtain the above cross-modal output feature information, thereby obtaining the final output feature information.

[0072] In this way, the dominance of text and graph structures in a task can be dynamically evaluated based on the information entropy of the input feature information. Based on this dominance, a fusion mode can be determined and the number of attention heads can be allocated. This determines the information content of the output feature information corresponding to various input data, making the information content more compatible with the task type. This provides richer and more effective reference information for task execution and improves the accuracy of task execution.

[0073] According to an embodiment of the present invention, after obtaining the output feature information, the target output head of each output feature information can be selected through the dynamic task routing layer MoE, that is, an expert model suitable for processing the output feature information can be selected, and the output feature information can be appropriately processed to obtain output information suitable for task execution, thereby obtaining appropriate task instructions.

[0074] According to an embodiment of the present invention, in step S3, selecting the target output head of the multimodal fusion unified model based on the output feature information includes: obtaining the basic probability of each output head through the first activation function of the task type classifier and the output feature information; obtaining the refined probability of each output head through the second activation function of the task type classifier and the input data; weighting and fusing the basic probability and the refined probability to obtain the fusion probability of each output head; obtaining the decision probability of each output head through the first activation function of the task type classifier and the fusion probability; and selecting the target output head based on the decision probability.

[0075] According to one embodiment of the present invention, each task may correspond to one output head, and the base probability of each output head may be determined according to the following formula (12): P(task i | output feature information) = softmax(s) j ), s j = w1 （i） × K j + w2 （i） ×S j +w3 （i） ×C j (12) Where P(task i|output feature information) is the base probability that the target output head of the output feature information is the output head corresponding to the i-th task. K j The keyword score is derived from the keyword weight table of task i. For example, it can be obtained by searching the input text data to see if any keywords in the keyword weight table of task i exist. If they do, the weights corresponding to these keywords are determined, and specific operations (e.g., averaging, maximizing, etc.) are performed based on these weights to obtain the keyword score. j Let be the semantic similarity, representing the matching degree between the output feature information and task i. For example, the routing model in the Dynamic Task Routing Layer (MoE) (such as the gated network in MoE) can be used to process the output feature information to obtain semantic features for comparison. Then, the semantic features are compared with the preset semantic features of the i-th task to solve for similarity (e.g., solving for cosine similarity) to obtain the semantic similarity i. jFor context relevance, this reflects the matching degree between the input text data and the static keywords matching task i. For example, the matching degree between the text data and the static keyword "time", the matching degree between the text data and the static keyword "network status", and the matching degree between the text data and the static keyword "permissions" can be calculated. The matching score between the text and each static keyword can be calculated separately (e.g., based on TF-IDF or word frequency), and then the results are summed according to preset weights (e.g., the weights are preset static weights of 0.4, 0.3, and 0.3 respectively) to obtain the context relevance. w1 （i） w2 （i） w3 （i） Preset weights specific to task i, satisfying w1 （i） +w2 （i） +w3 （i） =1, and can be flexibly set according to task characteristics (for example, fault diagnosis tasks focus more on keywords, while report generation tasks rely more on semantic similarity). All components K j S j C j Before weighting, all weights must be normalized to the [0,1] interval using Min-max normalization to eliminate dimensional differences and ensure the effectiveness of the weights; w1 can be calculated for each task. （i） × K j + w2 （i） ×S j +w3 （i） ×C j The calculation results are processed by the first activation function (softmax activation function) to obtain the basic probability that the target output head of the output feature information is the output head corresponding to the i-th task. Based on the same method, the basic probability that the target output head of the output feature information is the output head corresponding to each task can be calculated.

[0076] According to one embodiment of the present invention, the refinement probability of each output head can be obtained based on a second activation function (sigmoid activation function) and input data. In the example, the fault diagnosis task probability P can be determined according to the following formulas (13), (14), and (15), respectively. fault Configure the probability P of generating tasks config Performance prediction task probability P perf , P fault =sigmoid(2×number of faulty words + 1.5×number of abnormal words + 0.3×urgency)(13) P config =sigmoid(1.8×number of configuration words + 1.2×number of strategy words + 0.8×number of activated words)(14) P perf=sigmoid(1.5×number of performance words + 1.0×number of predicted words + 0.7×number of trending words) (15) The number of fault words, abnormal words, urgency words, configuration words, strategy words, activation words, performance words, prediction words, and trend words can all be set through historical data statistics or expert experience. Based on this method, the probabilities of other types of tasks can also be obtained, which can be used as refined probabilities for the output headers corresponding to various tasks.

[0077] According to one embodiment of the present invention, the base probability and refinement probability of each output head can be weighted and summed. For example, according to formula (16), the following fusion probability is obtained. P final (task i )=λ×P(task i|output feature information)+(1-λ)×Px(task i (16) Among them, P final (task i Let P(task_i|output_feature_information) be the fusion probability of the i-th task, and let P(task_i|output_feature_information) be the base probability of the i-th task. i ) represents the refinement probability of the i-th task, and λ is the fusion weight, with a value range of [0,1]. It is dynamically adjusted according to the integrity of the input modality and is calculated as follows: λ = λ0x The ratio of the number of current input modalities to the total number of modalities, where λ0 can be set to a fixed value (such as 0.6). When there is only one modal input, increase the weight of the path corresponding to that modality.

[0078] Furthermore, after obtaining the fusion probability of each task, the fusion probability of each task can be processed by the first activation function (softmax activation function) to obtain the decision probability of the output head corresponding to each task. For example, the decision probability of the output head can be obtained by formula (17): P decision (task i = softmax(P) final (task1),P final (task2),...,P final (task n )) i (17) Based on the decision probability, select the target output head. Choose one or two output heads with the highest decision probability as the target output head. If multiple target output heads are selected, use the decision probability as the expert weight and sum the output results of each target output head in a weighted manner to obtain the final result of processing the output feature information.

[0079] According to one embodiment of the present invention, based on the above method of filtering target output heads, not only can filtering be performed through output feature information, but also by predefining keywords (such as "configuration," "fault," etc.) in the input data to improve filtering flexibility. In the example, for a time-series prediction task, the input text can include "prediction" and time-related words, or the graph structure can include time-series features, etc., to increase the probability of the output head corresponding to the time-series prediction task being selected. For a fault diagnosis task, the input text can include keywords such as "fault" and "analysis." For a configuration generation task, the input text can include keywords such as "configuration" and "generation."

[0080] According to an embodiment of the present invention, in step S4, the output feature information is processed through the target output head to obtain task instructions. The output layer includes the aforementioned multiple output heads (only the target output head is activated) and a shared projection layer. Different output heads can be customized according to the corresponding task type. For example, the output head corresponding to the fault diagnosis task is a fully connected network containing a Softmax activation function, which can process the output feature information and output the probability distribution and impact assessment score of the root cause category; the output head corresponding to the configuration generation task is a lightweight Transformer decoder, which can process the output feature information, generate a structured JSON configuration sequence in an autoregressive manner, and also perform 3GPP compliance checks; the output head corresponding to the performance prediction task consists of a fully connected layer and a temporal convolutional layer, which can process the output feature information and output the predicted value of performance indicators over a future period of time, for example, outputting the future K-step prediction (load / QoS / anomaly), and conforming to the TS 23.288 standard; the output head corresponding to the report generation task is a deeper Transformer decoder, which is used to process the output feature information and generate a long natural language operation and maintenance report. The output header corresponding to the optimization suggestion task can output a resource allocation optimization scheme. Furthermore, the results from the above multiple modalities can be processed through a shared projection layer, projecting the results of different modalities onto the task common space to obtain task instructions. Task instructions may include information such as task type, task chain, confidence level, and constraints. Furthermore, the output headers can be grouped; for example, each output header can be a separate expert model. Therefore, during grouping, they can be divided into text analysis expert groups, graph analysis expert groups, and cross-modal fusion expert groups. Thus, a dynamic task routing layer can select appropriate expert groups to participate in task execution, while unselected expert groups will not be activated, thereby saving computational resources.

[0081] In this way, by selecting a target output head, computation compatible with multiple tasks can be achieved under the strictly limited computing, storage, and power resources on space. Through the MoE architecture, appropriate expert models can be selected for computation without activating all neural networks, significantly saving computational resources. Furthermore, when selecting a target output head, selection can be based on both the characteristics of the output feature information itself and the characteristics of the input data. For specific tasks, predefined keywords in the input data can assist in the selection, improving the flexibility and accuracy of target output head selection and making the target output head more suitable for specific tasks.

[0082] In the example, the system can generate task instructions periodically or based on event triggers. For instance, in fully automatic monitoring mode, network element status data, performance indicators, alarm information, etc., can be collected through OAM to construct nodes, edges, etc., containing a dynamic network topology graph. Furthermore, this graph structure includes a time dimension, reflecting the changing trends of the network status. The constructed dynamic network topology graph is input into a multimodal fusion unified model. Since there is no text input, the model primarily relies on the dynamic network topology graph for analysis and can combine pre-trained knowledge (such as typical fault modes stored in the model) for further analysis. Model outputs may include: fault prediction results and confidence levels; performance anomaly detection and root cause analysis; resource optimization suggestions; recommended self-healing operation schemes, etc. Further, high-confidence task instructions can be generated and automatically executed through a deterministic execution engine. The deterministic execution engine can automatically perform corresponding operations, such as resource adjustment and redundancy switching.

[0083] In the example, under ground command mode, the multimodal fusion unified model performs deep analysis of the input text, not only identifying the intent type (such as configuration generation, fault query, etc.) but also extracting structured information, including target entities, expected operations, and constraints. Based on the parsed intent, relevant subgraphs are retrieved from the current network state graph. For example, if the intent involves a specific network element, the subgraph consisting of that network element and its k-hop neighbors is extracted. Furthermore, an attention mechanism can be used to inject the semantic features of the intent into the graph analysis process, guiding the model to focus on network parts related to the intent. For example, when querying "Why is video service lagging?", the model will focus more on network elements and links related to video service. Joint reasoning is performed by combining textual intent and graph structure to generate solutions that satisfy the intent, which may include configuration scripts, operation instructions, analysis reports, etc. Furthermore, task instructions can be generated, which, after security verification, are executed by a deterministic execution engine, and the execution results are fed back to ground operators.

[0084] In the example, in hybrid mode, the model continuously monitors network status and automatically detects potential problems, such as performance degradation trends and resource bottlenecks. For detected problems, the model generates preliminary analysis reports and solution suggestions, including multiple alternatives and their expected effects. These solutions are sent to ground operations personnel for review. Personnel can modify, supplement, or reject the suggestions. The manually approved solutions are securely executed by a deterministic execution engine, and the execution process and results are recorded for subsequent learning.

[0085] According to one embodiment of the present invention, the method further includes: determining the task criticality level based on input feature information; high-criticality tasks use half-precision floating-point numbers, medium-criticality tasks use mixed precision, and low-criticality tasks use 8-bit integer quantization precision.

[0086] According to one embodiment of the present invention, the task criticality level can be manually specified or determined based on input feature information (such as keyword weights). For example, the task criticality level can be determined by processing the input feature information through a separately designed neural network model. Half-precision floating-point numbers are used for tasks with high criticality levels, mixed precision is used for tasks with medium criticality levels, and 8-bit integer precision is used for tasks with low criticality levels. Seamless transitions between different precision levels are achieved by automatically inserting precision conversion operators at runtime.

[0087] Figure 4 A schematic diagram of a deterministic execution engine according to an embodiment of the present invention is shown as an example.

[0088] According to one embodiment of the present invention, in step S5, the task instructions are parsed through the preprocessing layer of the deterministic execution engine to obtain the task steps.

[0089] According to one embodiment of the present invention, to ensure the secure and reliable conversion of AI inference results into network operations, the deterministic execution engine completely separates the probabilistic AI inference (i.e., the task instructions output by the multimodal fusion unified model) from the deterministic execution engine. The multimodal fusion unified model outputs task instructions containing confidence levels, and the deterministic execution engine module is responsible for converting the task instructions into secure operations. Specifically, the processing layer of the deterministic execution engine can parse and verify the task instructions, receive and parse task instructions from the multimodal fusion unified model, perform protocol compliance, task semantic security, permission and resource availability checks, and conduct risk assessment and classification.

[0090] According to an embodiment of the present invention, in step S6, the task steps are executed by a deterministic execution engine to obtain execution results, including: executing the task steps in an isolated simulation environment (containing a lightweight network simulator and model) to obtain simulated execution results; if the simulated execution results are safe, executing the task steps by the atomic operation execution layer of the deterministic execution engine to obtain execution results; generating a reverse operation plan; and automatically triggering a rollback mechanism according to the reverse operation plan when there is a risk in the simulated execution results.

[0091] In the example, the secure execution layer can be configured with a fully isolated simulation environment in which tasks can be pre-executed. This layer includes a network simulator, an impact prediction module based on a lightweight prediction model, and a rollback plan generator to simulate operational outcomes and generate reverse operation plans to obtain the simulated execution results. The reverse operation plans are used to roll back if there are risks in the simulated execution results.

[0092] According to one embodiment of the present invention, under the condition that the simulation execution result is secure, the task steps are executed through the atomic operation execution layer of the deterministic execution engine to obtain the execution result. The atomic operation execution layer includes multiple dedicated executors, such as configuration executors, signaling analyzers, resource conditioning executors, fault analyzers, and other action executors. Each executor is responsible for mapping high-level task instructions into specific network device operation command sequences that possess atomicity, rollback capability, and monitorability.

[0093] According to one embodiment of the present invention, the deterministic execution engine may further include a result verification and feedback layer, which can be used to compare the actual operation effect with the expected goal, evaluate the performance impact, and extract knowledge in a structured manner from successful or failed experiences, and feed it back to the multimodal fusion unified model for model optimization, forming a closed-loop learning.

[0094] According to one embodiment of the present invention, the multimodal fusion unified model can be deployed on a central board, and the various actuators of the deterministic execution engine can be distributed and deployed on processing units directly connected to each network element, thereby reducing internal communication latency. On the other hand, the deterministic execution engine can be replaced by a lightweight agent based on reinforcement learning to find a better execution strategy when the network environment changes dynamically (such as sudden changes in link quality).

[0095] In this way, the probabilistic reasoning results of a multimodal fusion unified model can be reliably transformed into deterministic network operations, ensuring system security. All AI inference task instructions are executed only after security verification, greatly guaranteeing the validity of the results. Simultaneously, feedback on the results further enhances the model's learning ability. Furthermore, under strict resource constraints, through thorough layered decoupling, probabilistic intelligent reasoning is separated from deterministic network operations, ensuring both the flexibility of intelligent decision-making and the reliability of network operations.

[0096] According to one embodiment of the present invention, the above process transforms the black-box AI inference process into a transparent, traceable, and step-by-step decision logic chain. By embedding structured inference steps into the model, the model's thinking and inference process becomes visible, understandable, and verifiable. In the example, the output of each layer of the Transformer serves as the input of the next layer, decomposing end-to-end inference into an interpretable, progressive process. During cross-modal fusion, the attention mechanism can be represented as the steps of text-guided graph structure analysis and the steps of graph structure analysis results influencing text generation, thus clearly demonstrating the collaborative work of the two modalities. During MoE expert routing decisions, the reasons for selecting the most relevant target output head can be recorded. When outputting task instructions, the verification process of the solution can be demonstrated, and the checking process of all constraints can be recorded, thus proving that the task instructions have been fully verified. In environments with strictly limited spaceborne resources and extremely high reliability requirements, the above decision logic chain makes the model's push and execution credible, reliable, and available. This enables the generation of reliable engineering solutions, providing key support for the practical application of intelligent autonomy in spaceborne core networks.

[0097] Figure 5 An exemplary schematic diagram of training a multimodal unified fusion model according to an embodiment of the present invention is shown.

[0098] According to an embodiment of the present invention, the training steps of the above-mentioned multimodal unified fusion model may include: pre-training a base model through a processor in a ground command center to obtain a pre-trained base model; performing knowledge distillation on the pre-trained base model to construct a multimodal unified fusion model; uploading the multimodal unified fusion model to a satellite; incrementally training the multimodal unified fusion models stored in multiple satellites using preset training objectives and training resources to obtain an optimized multimodal unified fusion model; obtaining parameter difference information between the optimized multimodal unified fusion model and the multimodal unified fusion model; each satellite uploading encrypted parameter increments processed by differential privacy, and the ground aggregation center performing aggregation calculations in encrypted state to obtain an updated model set; and distributing the trained multimodal unified fusion models according to satellite type differences.

[0099] This aggregation process can be performed using a secure aggregation protocol to protect local data privacy. Each satellite uploads encrypted parameter increments processed with differential privacy. The ground aggregation center performs aggregation calculations in encrypted form to obtain an updated model set. Specifically, this includes: aggregating the text encoding layer parameters of the multimodal fusion unified model using a global averaging strategy to achieve network-wide sharing of language understanding capabilities; aggregating the graph encoding layer and edge relationship embedding layer parameters of the multimodal fusion unified model using a global averaging strategy to adapt to graph analysis needs in different orbital environments; and aggregating the parameters of the output header of the multimodal fusion unified model according to task type to support differentiated business optimization.

[0100] According to one embodiment of the present invention, a pre-trained basic model is obtained by pre-training a basic model through a processor in a ground command center, including: processing multimodal training data through the basic model to obtain text semantic information, graph node feature information, and training task instructions; obtaining a loss function based on the annotation information, the text semantic information, the graph node feature information, and the training task instructions; and pre-training the basic model based on the loss function to obtain a pre-trained basic model.

[0101] In the example, during the ground-based pre-training phase, centralized, large-scale pre-training can be performed to generate a base model. A text corpus containing core network 3GPP specifications, configuration manuals, fault reports, and operation and maintenance manuals is constructed, along with a graph corpus containing network topology, signaling flowcharts, resource dependency graphs, and fault propagation path graphs. Text-graph aligned sample pairs can be created, and six types of pre-training tasks can be designed: masked language modeling, masked node prediction, edge prediction, graph-text matching, graph description generation, and text-to-graph generation. Self-supervised training can be performed for various tasks. Furthermore, through knowledge distillation, the base model can be used as a teacher model to transfer knowledge to a lightweight student model, which is the multimodal unified fusion model. This lightweight multimodal unified fusion model can then be injected into a satellite via a space-to-ground link.

[0102] In the example, for different tasks, the base model can obtain textual semantic information, graph node feature information, and training task instructions. Based on the differences between these information and the sample annotation information, a loss function can be obtained. For example, for the differences between textual semantic information and annotation information, and between graph node feature information and annotation information, a cross-entropy loss function can be used. For the differences between training task instructions and annotation information, various types of loss functions can also be used. For example, for the fault prediction task, the loss function is a binary cross-entropy loss function; for the configuration generation task, the loss function is a sequence-to-sequence cross-entropy loss, and a loss term for configuration syntax compliance can also be added; for the performance prediction task, for time series prediction, a smoothed L1 loss function can be used, which is more robust to outliers. Furthermore, a cross-modal alignment loss function can be introduced. For example, for input data from different modalities with the same content, the similarity difference between their input feature information can be used as a cross-modal alignment loss function. By weighted summing of the above multiple loss functions, the loss function of the base model can be obtained. Then, the parameters of the base model can be adjusted by gradient descent to perform pre-training and obtain the pre-trained base model. During pre-training, an optimizer (such as Adam) and a cosine annealing hot-start learning rate scheduler can be used to perform multi-task joint training on the unified model. In each training step, the various loss functions mentioned above are dynamically weighted and combined according to the label type contained in the batch data, and gradient pruning is implemented to stabilize the training.

[0103] According to one embodiment of the present invention, the pre-trained base model is compressed through four steps: structured pruning, knowledge distillation, quantized perceptual training, and hardware-aware optimization, generating a lightweight version suitable for onboard AI chips, namely, a multimodal fusion unified model. Furthermore, under the condition of strictly limited onboard resources, during onboard inference, the model's inference accuracy and batch size are dynamically adjusted based on real-time monitored CPU and memory usage.

[0104] According to one embodiment of the present invention, during on-board incremental training, a multimodal unified fusion model stored in multiple satellites can be incrementally trained using preset training objectives and resources to obtain an optimized multimodal unified fusion model. In the example, satellite A is used to focus on training specific fault data and fault mode data under high-concurrency scenarios, thereby enabling model optimization tailored to the characteristics of mobile terminals. Satellite B (VIP customer) is used for performance prediction learning, optimizing performance prediction for IoT services. Satellite C (broadband internet) is used to focus on training for dynamic allocation optimization of bandwidth resources. Furthermore, an incremental learning module containing an experience replay buffer and a continuous learning strategy is deployed on-board. When new experiences are generated, historical data can be sampled from the buffer and mixed with new data to form a batch for training. Elastic weight consolidation technology is used to constrain changes in important parameters, protecting old knowledge from being forgotten while acquiring new knowledge. The parameter increments generated by learning are processed for differential privacy and are prepared for uploading via federated learning.

[0105] According to one embodiment of the present invention, the parameter difference information of the optimized multimodal unified fusion model and the multimodal unified fusion model, namely, the encrypted ΔδA, ΔδB, and ΔδC, can be obtained. The parameter difference information can then be aggregated using a hierarchical federated averaging algorithm to obtain an updated model set.

[0106] In the example, parameter differences can be securely exchanged and aggregated via inter-satellite links. Alternatively, the ground center can aggregate the collected parameters and use a hierarchical federated averaging algorithm to aggregate different parameter layers, obtaining an updated model set. A secure aggregation protocol can be employed, with each satellite uploading encrypted parameter increments, and the ground aggregation center performing aggregation calculations in encrypted form, protecting the data privacy of each satellite.

[0107] In the example, the parameters of the text encoding layer are aggregated using a global averaging strategy, enabling all satellites to share language understanding capabilities; the parameters of the graph encoding layer and the edge relation embedding layer are aggregated using a global averaging strategy, which can optimize graph analysis capabilities according to orbital characteristics and adapt to graph analysis needs in different orbital environments; the parameters of the output header can be aggregated in a personalized manner according to the service type, so as to adapt different output headers to different service characteristics.

[0108] Furthermore, different model variants can be distributed according to the differences in satellite types. That is, the various trained multimodal unified fusion models in the model set are distributed to the corresponding satellites. For example, the basic model part shared by all satellites is distributed as a model variant optimized for high-concurrency communication for satellite A, and a model variant optimized for communication in business experience scenarios for satellite B.

[0109] According to one embodiment of the present invention, model validation and deployment can be performed, that is, the model is rigorously validated before updating, and performance comparison tests are conducted between the old and new models. For example, adversarial example and security tests, inference latency and accuracy performance tests are performed. Furthermore, differentiated distribution can be implemented, that is, the new model is deployed in batches and stages to reduce risk. In addition, version management functionality can be provided, that is, historical versions can be saved and version control can be performed. Deployment status can also be obtained in a timely manner, that is, the model update status of each satellite, and a fast rollback mechanism is maintained, that is, the system has the ability to quickly revert to the old version.

[0110] In this way, through full ground training, knowledge distillation, and hierarchical federated learning, the single model remains simple, reduces the consumption of computing resources, and can achieve continuous model evolution without full on-board retraining, while protecting the local data privacy and business characteristics of each satellite.

[0111] The following examples illustrate the specific implementation of this application: Figure 6 An exemplary schematic diagram illustrates the application of the intelligent autonomous method for a multimodal fusion core network based on an embodiment of the present invention.

[0112] According to one embodiment of the present invention, in the multimodal input stage, the input data includes natural language descriptions and graph structures. Maintenance personnel can input natural language fault descriptions, including key symptom information, such as a slow response time and high CPU utilization in a certain network element (NF). The graph structure includes a network topology diagram, such as the current network topology diagram, showing the connection relationships between the faulty NF and related NFs, and may also include a performance monitoring graph, such as a graph structure formed from real-time performance monitoring data, including metrics such as CPU and memory usage.

[0113] According to an embodiment of the present invention, in the fusion model processing stage, the above-mentioned multimodal unified fusion model is used for processing, which can convert natural language descriptions into text feature vector sequences and convert nodes in the network topology graph into graph feature vectors, both of which can be used as input feature information.

[0114] Furthermore, through text self-attention, self-attention calculation is performed within the text feature vector sequence to capture text semantics and obtain output feature information. Through graph message passing, message passing occurs between graph nodes to update node representations, and output feature information is obtained through the self-attention mechanism. Through a cross-modal attention mechanism, the text feature vector sequence and graph feature vectors are bidirectionally fused using a bidirectional cross-attention mechanism to obtain the output feature vector. Furthermore, this bidirectional cross-attention mechanism can bidirectionally fuse the input feature information corresponding to the graph structure and the input feature information corresponding to the ground text input data, or it can bidirectionally fuse the output feature information corresponding to the graph structure after self-attention processing and the output feature information corresponding to the ground text input data; this invention does not limit this. After processing by the above attention mechanisms, the fused output feature information can be obtained.

[0115] According to one embodiment of the present invention, in the task identification and solution generation stage, an expert model that matches the output feature information can be found through dynamic routing. In the example, the task type corresponding to the output feature information can be identified. For example, if the task is identified as a fault diagnosis task, then the output head corresponding to fault diagnosis is activated. A solution is generated through this output head, and specific inference steps are generated. For example, the first step of the generated solution is "migrate the business to SMF02", and the second step of the generated solution is "restart SMF01". The confidence level of the solution can also be given. For example, a confidence level of 0.92 means that the model's confidence score for the solution is 92%.

[0116] According to one embodiment of the present invention, during the execution and feedback phase, a restart operation can first be simulated in a secure sandbox to predict the impact of the operation on network services and automatically generate a rollback plan in case of operation failure. After confirming the safety of the operation, the operation can be formally executed to gracefully restart the faulty NF and obtain the execution results. For example, the recovery status of the faulty NF can be monitored in real time, and the total time from diagnosis to recovery completion (e.g., 350ms) can be monitored. The service impact during the execution process can also be monitored, and the execution results can be collected. For example, after successful execution, the overall status identifier of the execution results can be obtained, and the execution results can be fed back to the multimodal unified fusion model for feedback training.

[0117] According to one embodiment of the present invention, during the above execution process, the semantic information in the text helps the graph structure analysis to accurately locate the target, and the graph structure information helps to generate accurate text instructions. Therefore, combining the information from the two modalities allows for more accurate joint reasoning and provides richer interpretable information, thereby improving the interpretability of the model.

[0118] Furthermore, the method of this application can be applied to a variety of representative scenarios, such as fault prediction and self-healing scenarios, which can be used for predictive network element fault self-healing; policy configuration and optimization scenarios, which can intelligently generate service policy configurations; resource management and energy saving scenarios, which can generate and execute network energy saving policies; and emergency response and protection scenarios, such as emergency response to satellite-to-ground link interruptions.

[0119] According to one embodiment of the present invention, in a scenario of predictive network element fault self-healing, a fully automatic monitoring mode can be used. In this mode, the input data includes a dynamic network topology diagram. The entire system can be periodically monitored, for example, by triggering inference every 5 minutes, or by using event triggering, such as triggering inference when a key indicator exceeds a threshold.

[0120] According to one embodiment of the present invention, the nodes of the dynamic network topology graph include core network elements, such as SMF, UPF, AMF, etc., and the characteristic information of the nodes can be obtained based on the state attributes of the network elements (e.g., CPU, memory, number of sessions). The edges of the dynamic network topology graph include interface connections, such as N1, N2, N4, etc., and the characteristic information of the edges can be obtained through traffic attributes (e.g., latency, link quality, etc.).

[0121] According to one embodiment of the present invention, based on the performance trend sequence of the most recent 30 minutes, the system autonomously triggers a processing flow without manual text input. This flow first constructs a spatiotemporal graph containing SMF01 and its associated network elements based on real-time monitoring data. Then, it detects abnormal patterns, such as the memory leak trend of SMF01, using a graph neural network. Based on the topology analysis, it analyzes the fault propagation range and service impact, thereby generating a preventative self-healing plan for migration, restart, and verification. This plan is automatically executed after verification in a security sandbox. The output includes a fault prediction report indicating that SMF01 will fail within 15±5 minutes with a confidence level of 92%. The system can implement self-healing based on this report using pooled reliability or a primary / backup solution. The execution result shows a total execution time of 350 milliseconds and a service interruption time of 0 milliseconds.

[0122] According to one embodiment of the present invention, in a scenario of predictive network element fault self-healing, a ground command mode can be used. The system receives the manually input text command "SMF01 memory usage continuously exceeds 85%, analyze the risk and handle it" and generates an SMF01 related topology and performance timing diagram in real time. The processing flow first performs deep intent parsing to extract the target network element, risk indicators, and handling requirements; then, through intent-guided graph analysis, the text command guides the system to focus on memory-related indicators and timing trends; next, multimodal joint diagnosis is performed, combining the risk level indicated in the text with the topology structure in the diagram for comprehensive impact analysis; finally, a detailed analysis report and solution are generated for manual confirmation. Corresponding monitoring events can also be generated according to user needs, such as periodic monitoring (e.g., every 5 minutes) or event triggering.

[0123] According to one embodiment of the present invention, in a scenario of predictive network element fault self-healing, a hybrid mode can be used. The system first detects an anomaly in SMF01, but with a moderate confidence level (75%), and provides a suggestion: "An abnormal memory trend has been detected in SMF01. Preventive maintenance is recommended, with a confidence level of 75%." Subsequently, ground personnel review the suggestion, supplement information or adjust the plan, and finally output an optimized plan based on human experience.

[0124] According to one embodiment of the present invention, in a policy configuration and optimization scenario, a ground command mode can be used. Ground personnel input the text "Enable 4K video service guarantee for VIP user XXX", thereby retrieving the graph structure and obtaining the topology of the graph structure (including the path from gNB to AMF, SMF, and UPF) and the status of resources along the path. The processing flow includes service intent parsing to identify the user level as VIP (highest priority), the service type as 4K video (high bandwidth, low latency), and the guarantee requirements as resource reservation and QoS guarantee; it can also perform topology resource analysis, such as checking the available bandwidth of the path, estimating end-to-end latency, and verifying the remaining network element resources; furthermore, it can also perform intelligent policy generation, such as automatically generating 3GPP standard PCC policies containing parameters such as 5QI, ARP, and bandwidth and performing automatic compliance verification; in addition, it can also perform secure deployment, which is issued and executed after passing the security sandbox verification. The output results are standardized PCC policy configuration (JSON format), deployment plan issued through standard interfaces (such as OAM), and service quality assurance monitoring scheme. In addition, the trigger condition for the autonomous monitoring mode is to automatically identify VIP users online based on user subscription information. The input data is the user subscription information graph structure and the current network status graph structure. Real-time monitoring automatically matches the VIP user policy template and optimizes it in combination with the real-time network status.

[0125] According to one embodiment of the present invention, in resource management and energy-saving scenarios, a fully automatic monitoring mode can be used. The trigger condition is that the network-wide service load remains below 30% for one hour, and the system predicts that this low-load state will continue for the next two hours. Input data includes graph structures, such as a network-wide load distribution heatmap (nodes are network elements, attributes are load rates), a device power consumption correlation graph (nodes are physical devices, edges represent energy consumption relationships), and a service prediction time-series graph (load prediction for the next two hours). No text data is input; decisions are made autonomously based entirely on graph structure analysis. The processing flow includes: identifying load patterns through graph clustering analysis to locate low-load areas; assessing the energy-saving potential and impact of each device; performing security constraint checks to ensure that energy-saving operations do not affect critical services and system redundancy; generating progressive energy-saving schemes from light to moderate to deep; and automatically executing energy-saving operations according to the plan. The output is a tiered energy-saving plan table, with an expected overall power consumption reduction of 25%, and includes an intelligent recovery strategy based on proactive wake-up based on service prediction.

[0126] According to one embodiment of the present invention, in resource management and energy-saving scenarios, a ground command mode can be used, triggered by a manually input text command such as "Currently in a low-peak business period, execute energy-saving strategy." This allows for manual setting of the specific intensity and time window for energy saving, upon which the system generates a customized energy-saving plan. In hybrid mode, the system first suggests: "Low load mode detected, energy saving recommended, saving 20% ​​of energy consumption." Subsequently, manual confirmation is performed to review the scope and intensity of energy saving and supplement any business-specific requirements. The collaborative feature of this mode lies in its ability to combine human business knowledge to ultimately form an optimized energy-saving plan.

[0127] According to one embodiment of the present invention, in emergency response and support scenarios, a fully automatic monitoring mode can be used. In this mode, the triggering condition is that the satellite-to-ground link signal loss lasts for more than 30 seconds, or the bit error rate is greater than 10%. - ³. Input data includes graph structure data, such as the affected service topology graph structure (nodes represent affected service flows, edges represent dependencies), the backup path availability graph structure (nodes represent backup ground stations, edges represent connectivity), and the service priority labeling graph structure (nodes represent services, attributes represent priority levels). No text data input is allowed. The processing flow is as follows: automatic event classification based on impact scope and duration; intelligent retrieval of the most suitable contingency plan from the contingency plan library; real-time status adaptation and dynamic adjustment of contingency plan parameters based on the current network status; automated execution of emergency operation sequences; and real-time monitoring of recovery effects and service impact. Output results are the emergency operation execution sequence and a report for archiving and review.

[0128] According to one embodiment of the present invention, in an emergency response and support scenario, in ground command mode, the processing flow is triggered by manually inputting the text "Satellite-to-ground link interrupted, execute emergency plan," allowing manual specification of the emergency level and key support services, upon which the system generates a targeted plan. In hybrid mode, the system first detects a link anomaly, but is unsure whether it is a permanent interruption, and provides the suggestion: "Satellite-to-ground link anomaly detected, it is recommended to activate Level 2 emergency response"; subsequently, manual decision-making is made to confirm the emergency level and operational scope. This enables rapid human-machine collaborative decision-making in emergency situations.

[0129] Figure 7 An exemplary block diagram of an intelligent autonomous system for a multimodal fusion core network according to an embodiment of the present invention is shown. The system includes: The preprocessing module is used to preprocess the input data according to the selected working mode to obtain multiple input feature information, wherein the working mode includes one of fully automatic monitoring mode, ground command mode and hybrid mode; The output feature module is used to process the input feature information according to the working mode and the attention mechanism of the multimodal fusion unified model to obtain the output feature information; The selection module is used to select the target output head of the multimodal fusion unified model based on the output feature information. The task instruction module is used to process the output feature information through the target output head to obtain task instructions; The task step module is used to parse task instructions and obtain task steps through the preprocessing layer of the deterministic execution engine; The execution result module is used to execute the task steps through a deterministic execution engine and obtain the execution result.

[0130] According to one embodiment of the present invention, a smart autonomous device for a multimodal converged core network is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the smart autonomous method for the multimodal converged core network.

[0131] According to one embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the intelligent autonomous method of the graphics and text multimodal fusion core network.

[0132] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0133] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent autonomy of a picture-text multi-modal fusion core network, characterized in that, include: Based on the selected operating mode, the input data is preprocessed to obtain multiple input feature information. The operating mode includes one of a fully automatic monitoring mode, a ground command mode, and a hybrid mode. In the fully automatic monitoring mode, the input data includes network element status data. In the ground command mode, the input data includes ground text input data. In the hybrid mode, the input data includes both network element status data and ground text input data. Based on the network element status data and the traffic attribute information of the data interface, a dynamic network topology diagram is constructed. Based on the attention mechanism of the multimodal fusion unified model, the input feature information is processed to obtain the output feature information, including: The attention mechanism is used to fuse the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data to obtain the first output feature information. The attention mechanism is used to fuse the input feature information corresponding to the ground text input data with the input feature information corresponding to the dynamic network topology graph to obtain the second output feature information. The output feature information is obtained based on the first output feature information and the second output feature information; Based on the output feature information, the target output head of the multimodal fusion unified model is selected, including: The basic probabilities of each output head are obtained by using the first activation function and output feature information of the task type classifier; The refined probabilities of each output head are obtained by using the second activation function of the task type classifier and the input data. The base probability and the refined probability are weighted and fused to obtain the fused probability of each output head; The decision probability of each output head is obtained by using the first activation function of the task type classifier and the fusion probability. The target output head is selected based on the decision probability. The output feature information is processed through the target output head to obtain task instructions; The task instructions are parsed through the preprocessing layer of the deterministic execution engine to obtain the task steps; The task steps are executed using a deterministic execution engine to obtain the execution result.

2. The method according to claim 1, characterized in that, Based on the selected working mode, the input data is preprocessed to obtain multiple input feature information, including: The graph coding layer of the input layer of the multimodal fusion unified model is used to process the nodes of the dynamic network topology graph to obtain the initial node feature information of the dynamic network topology graph. The initial node feature information overlay image position is encoded to obtain node feature information; Edge relationship feature information is obtained by processing the edges of the dynamic network topology graph through the edge relationship embedding layer of the input layer of the multimodal fusion unified model. The input feature information is obtained based on node feature information and edge relationship feature information; or, The text encoding layer of the input layer of the multimodal fusion unified model processes the word vectors corresponding to multiple word segments of the ground text input data to obtain initial word segmentation feature information. Text position encoding is added to the initial word segmentation feature information to obtain the input feature information.

3. The method of claim 1, wherein, The attention mechanism is used to fuse the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data to obtain the first output feature information, including: Through the attention mechanism, the first key matrix and the first weight matrix of the input feature information corresponding to the dynamic network topology graph are obtained, and the first query matrix of the input feature information corresponding to the ground text input data is obtained through the attention mechanism. Based on the first query matrix, the first key-value matrix, and the first weight matrix, the first output feature information is obtained.

4. The method according to claim 1, characterized in that, The attention mechanism is used to fuse the input feature information corresponding to the ground text input data with the input feature information corresponding to the dynamic network topology graph to obtain the second output feature information, including: Through the attention mechanism, the second key matrix and the second weight matrix of the input feature information corresponding to the ground text input data are obtained, and the second query matrix of the input feature information corresponding to the dynamic network topology graph is obtained through the attention mechanism. Based on the second query matrix, the second key matrix, and the second weight matrix, the second output feature information is obtained.

5. The method according to claim 1, characterized in that, The output feature information is obtained based on the first output feature information and the second output feature information, including: Obtain the first information entropy of the input feature information corresponding to the ground text input data, and obtain the second information entropy of the input feature information corresponding to the dynamic network topology graph; Based on the first information entropy and the second information entropy, the text dominance score and the graph dominance score are determined respectively; The text value contribution and graph value contribution are determined based on the static contribution ratio related to the task type, as well as the text dominance score and graph dominance score. The fusion mode is determined based on the contribution of text value and graph value. Based on the fusion mode, determine the number of text self-attention heads, the number of graph self-attention heads, and the number of cross-modal attention heads; Output feature information is obtained based on the number of text self-attention heads, the number of graph self-attention heads, and the number of cross-modal attention heads, as well as the first output feature information and the second output feature information.

6. The method according to claim 1, characterized in that, The method further includes: Determine the task's criticality level based on the input feature information; High-criticality tasks use half-precision floating-point numbers, medium-criticality tasks use mixed precision, and low-criticality tasks use 8-bit integer quantization precision.

7. The method according to claim 1, characterized in that, The task steps are executed using a deterministic execution engine to obtain execution results, including: The task steps are executed in an isolated simulation environment to obtain the simulation results; If the simulation results are safe, the task steps are executed through the atomic operation execution layer of the deterministic execution engine to obtain the execution results; Generate a contingency plan for reverse operations; When the simulation execution result is risky, the rollback mechanism will be automatically triggered according to the aforementioned reverse operation plan.

8. The method according to claim 1, characterized in that, The method further includes: The basic model is pre-trained using the processor in the ground command center to obtain the pre-trained basic model. Knowledge distillation is performed on the pre-trained base model to construct a multimodal unified fusion model; The multimodal unified fusion model was uploaded to the satellite; By using preset training objectives and training resources, incremental training is performed on the multimodal unified fusion model stored in multiple satellites to obtain an optimized multimodal unified fusion model. Obtain the parameter difference information between the optimized multimodal unified fusion model and the multimodal unified fusion model; Each satellite uploads encrypted parameter increments processed with differential privacy, and the ground aggregation center performs aggregation calculations in the encrypted state to obtain the updated model set; The trained multimodal unified fusion model is distributed according to the differences in satellite type.

9. The method according to claim 8, characterized in that, The base model is pre-trained using the processor at the ground command center to obtain the pre-trained base model, which includes: By processing multimodal training data through a basic model, textual semantic information, graph node feature information, and training task instructions are obtained. The loss function is obtained based on the annotation information, the text semantic information, the graph node feature information, and the training task instructions; The base model is pre-trained based on the loss function to obtain the pre-trained base model.

10. The method according to claim 9, characterized in that, Based on the annotation information, the text semantic information, the graph node feature information, and the training task instructions, a loss function is obtained, including: Based on the annotation information and the text semantic information, a semantic prediction loss function is obtained; Based on the annotation information and the graph node feature information, a graph feature prediction loss function is obtained; Based on the graph node feature information and text semantic information, the alignment loss function is obtained; Based on the annotation information and training task instructions, the task prediction loss function is obtained; The loss function is obtained based on the semantic prediction loss function, the graph feature prediction loss function, the alignment loss function, and the task prediction loss function.

11. The method according to claim 8, characterized in that, Each satellite uploads encrypted parameter increments processed with differential privacy. The ground aggregation center performs aggregation calculations in the encrypted state to obtain the updated model set, including: For the parameters of the text encoding layer in the multimodal fusion unified model, a global averaging strategy is used for aggregation; For the parameters of the graph coding layer and edge relation embedding layer of the multimodal fusion unified model, a global averaging strategy is used for aggregation; The parameters of the output head of the multimodal fusion unified model are aggregated in a personalized manner according to the task type.

12. A multimodal image-text fusion core network intelligent autonomous system, characterized in that, include: A preprocessing module is used to preprocess the input data according to the selected working mode to obtain multiple input feature information. The working mode includes one of a fully automatic monitoring mode, a ground command mode, and a hybrid mode. In the fully automatic monitoring mode, the input data includes network element status data; in the ground command mode, the input data includes ground text input data; and in the hybrid mode, the input data includes both network element status data and ground text input data. Based on the network element status data and the traffic attribute information of the data interface, a dynamic network topology map is constructed. The output feature module processes the input feature information according to the attention mechanism of the multimodal fusion unified model to obtain output feature information, including: The attention mechanism is used to fuse the input feature information corresponding to the dynamic network topology graph with the input feature information corresponding to the ground text input data to obtain the first output feature information. The attention mechanism is used to fuse the input feature information corresponding to the ground text input data with the input feature information corresponding to the dynamic network topology graph to obtain the second output feature information. The output feature information is obtained based on the first output feature information and the second output feature information; The selection module is used to select the target output head of the multimodal fusion unified model based on the output feature information, including: The basic probabilities of each output head are obtained by using the first activation function and output feature information of the task type classifier; The refined probabilities of each output head are obtained by using the second activation function of the task type classifier and the input data. The base probability and the refined probability are weighted and fused to obtain the fused probability of each output head; The decision probability of each output head is obtained by using the first activation function of the task type classifier and the fusion probability. The target output head is selected based on the decision probability. The task instruction module is used to process the output feature information through the target output head to obtain task instructions; The task step module is used to parse task instructions and obtain task steps through the preprocessing layer of the deterministic execution engine; The execution result module is used to execute the task steps through a deterministic execution engine and obtain the execution result.

Images