Processing method and device based on joint pre-training, electronic equipment and product

Abstract

The invention provides a processing method and device based on joint pre-training, electronic equipment and a computer program product, and relates to the technical field of computers.The method comprises the steps that feature extraction is conducted on a network topological graph to obtain graph data representation, and text description of a task to be processed is converted into text embedding representation; calculating the correlation between the graph data representation and the text embedding representation through a graph neural network encoder, a text encoder and a cross-modal attention fusion module in the pre-training model to obtain a target graph data representation and a target text embedding representation, and mapping the target graph data representation and the target text embedding representation to a unified representation space; the pre-training model is obtained according to multi-task target training; and adjusting the pre-training model according to the to-be-processed task, inputting the target graph data representation, mapped to the unified representation space, of the to-be-processed task and the target text embedding representation into the adjusted pre-training model, and outputting an application result. The accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to a processing method based on joint pre-training, a processing device based on joint pre-training, an electronic device and a computer program product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, pre-training models have made breakthrough progress in natural language processing, computer vision and other fields, significantly improving the performance of various tasks.

[0003] However, the pre-training method in the related art mainly focuses on processing single modal data, network topology analysis management and natural language processing are usually treated as two independent fields, it is difficult to establish effective association between network topology elements and text description, and cross-modal understanding cannot be achieved. Moreover, a large amount of manually annotated training data is needed to establish the correspondence between the topology structure and the text description, which is high in cost and difficult to obtain data. It is difficult to capture the deep semantic relationship between the complex connection relationship between network devices and its text description, and it is difficult to fully utilize the complementary information between the two modal data to realize model training, and the accuracy is low when processing tasks through the model. SUMMARY

[0004] The purpose of the present disclosure is to provide a processing method based on joint pre-training, a processing device based on joint pre-training, an electronic device and a computer program product, thereby at least partially overcoming the problem of low accuracy due to the limitations and defects of the related art.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a processing method based on joint pre-training is provided, comprising: performing feature extraction on a network topology graph to obtain a graph data representation, and converting a text description of a to-be-processed task into a text embedding representation; calculating the relevance of the graph data representation and the text embedding representation through a graph neural network encoder, a text encoder and a cross-modal attention fusion module in a pre-training model, obtaining a target graph data representation and a target text embedding representation, and mapping the target graph data representation and the target text embedding representation to a unified representation space; the pre-training model is trained according to a multi-task target to obtain the graph neural network encoder, the text encoder and the cross-modal attention fusion module. adjusting the pre-trained model according to the to-be-processed task, inputting the mapped target graph data representation and the target text embedding representation of the to-be-processed task into the adjusted pre-trained model, and outputting an application result.

[0007] In an example embodiment of the present disclosure, the correlation between the graph data representation and the text embedding representation calculated by the graph neural network encoder, the text encoder, and the cross-modal attention fusion module in the pre-trained model includes: extracting a topological structure representation of the graph data representation by the graph neural network encoder; determining a text semantic representation of the text embedding representation based on the text encoder; calculating attention of the topological structure representation relative to the text semantic representation and attention of the text semantic representation relative to the topological structure representation according to the cross-modal attention fusion module, to obtain the target graph data representation and the target text embedding representation.

[0008] In an example embodiment of the present disclosure, the calculation of the attention of the topological structure representation relative to the text semantic representation and the attention of the text semantic representation relative to the topological structure representation according to the cross-modal attention fusion module to obtain the target graph data representation and the target text embedding representation includes: calculating a first attention distribution of each node in the topological structure representation relative to all text semantic representations, weighting and aggregating the text semantic representations according to the first attention distribution, and generating a context-enhanced text representation corresponding to each node as the target text embedding representation; for each text semantic representation, calculating an attention score of each text semantic representation relative to all nodes of the topological structure representation to obtain a second attention distribution and aggregation, and generating the target graph data representation.

[0009] In an example embodiment of the present disclosure, the method further includes: processing the sample graph data representation and the sample text embedding representation according to a multi-task objective to generate a training sample; training the graph neural network encoder, the text encoder, and the cross-modal attention fusion module based on a plurality of objective functions and the training sample to obtain the pre-trained model.

[0010] In an example embodiment of the present disclosure, the multi-task objective includes node-text contrast learning, masked language modeling, and masked node modeling.

[0011] In an example embodiment of the present disclosure, the adjusting of the pre-trained model according to the to-be-processed task includes: According to the type of the task to be processed, a task output layer is added to the pre-trained model to obtain an intermediate model; The intermediate model is fine-tuned using the labeled data of the task to be processed to obtain the adjusted pre-trained model.

[0012] In one exemplary embodiment of this disclosure, the step of extracting features from the network topology graph to obtain graph data representation includes: Graph neural networks abstract each network device in a network topology graph as a node in the graph, and the connection relationships between network devices as edges; The neighbor information of each node is aggregated, and the feature representation of the node is updated step by step to determine the graph data representation; the neighbor information includes one or more of the following: device type, connection status, and configuration parameters.

[0013] According to one aspect of this disclosure, a processing apparatus based on joint pre-training is provided, comprising: The feature extraction module is used to extract features from the network topology graph to obtain graph data representation, and to convert the text description of the task to be processed into a text embedding representation. The joint training module is used to calculate the correlation between the graph data representation and the text embedding representation through the graph neural network encoder, text encoder and cross-modal attention fusion module in the pre-trained model, to obtain the target graph data representation and the target text embedding representation, and to map the target graph data representation and the target text embedding representation to a unified representation space; the pre-trained model is obtained by training the graph neural network encoder, text encoder and cross-modal attention fusion module according to the multi-task objective; The task processing module is used to adjust the pre-trained model according to the task to be processed, input the target graph data representation and target text embedding representation of the task to be processed into the adjusted pre-trained model, and output the application results.

[0014] According to one aspect of this disclosure, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to execute any of the above-described joint pre-training-based processing methods by executing the executable instructions.

[0015] According to one aspect of this disclosure, a computer program product is provided, which, when executed by a processor, implements the processing method based on joint pre-training as described in any one of the preceding claims.

[0016] The technical solution provided in this disclosure, on the one hand, calculates the correlation between graph data representation and text embedding representation through the graph neural network encoder, text encoder, and cross-modal attention fusion module in the pre-trained model. This enables the establishment of an effective association between the network topology graph and the text description of the task to be processed, achieving cross-modal understanding. On the other hand, obtaining the target graph data representation and target text embedding representation through correlation calculation allows full utilization of the complementary information between the two modalities to achieve model training and improve accuracy. Model training through multi-task objectives avoids the problem of requiring a large amount of manually labeled training data, reducing the difficulty and cost of data acquisition. Furthermore, mapping the target graph data representation and target text embedding representation to a unified representation space based on the pre-trained model makes data from different modalities comparable and operable, providing a consistent semantic foundation for the task to be processed and improving the universality and accuracy of the processing task.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 The schematic diagram illustrates a flowchart of a processing method based on joint pre-training in an embodiment of this disclosure.

[0020] Figure 2 The schematic diagram illustrates a process for determining correlation in an embodiment of this disclosure.

[0021] Figure 3 This schematic diagram illustrates the overall process of performing downstream tasks in an embodiment of the present disclosure.

[0022] Figure 4 The diagram illustrates a block diagram of a processing apparatus based on joint pre-training according to an embodiment of the present disclosure.

[0023] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0025] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] With the rapid development of artificial intelligence technology, pre-trained models have made breakthroughs in many fields such as natural language processing and computer vision, significantly improving the performance of various tasks. Especially in the field of natural language processing, large-scale pre-trained models based on the Transformer architecture have demonstrated powerful language understanding and generation capabilities. However, existing pre-training methods mainly focus on processing single-modal data, either concentrating on text semantic analysis or emphasizing image or graph structure processing, lacking the ability to jointly model multimodal data, especially network topology, with natural language descriptions.

[0027] In current technological practices, network topology analysis and management, as well as natural language processing, are typically treated as two separate domains. The generation and analysis of network topology graphs primarily rely on graph theory algorithms and specialized network management tools. These tools can effectively display the connections and topological structure between network devices, but lack a deep semantic understanding of topological information. On the other hand, while natural language processing technology can understand and generate human language, it cannot directly understand and manipulate structured data such as network topology.

[0028] This separate processing approach makes it difficult to establish an effective association between network topology elements and text descriptions, thus failing to achieve true cross-modal understanding. Traditional methods require a large amount of manually labeled training data to establish the correspondence between topology structures and text descriptions, which is both costly and difficult to obtain on a large scale in practical applications. Furthermore, existing methods struggle to capture the deep semantic connections between the complex connectivity relationships between network devices and their text descriptions, failing to fully utilize the complementary information between the two modalities.

[0029] Currently, there is a lack of a unified representation learning framework to simultaneously encode graph structural information and textual semantic information. This severely limits system performance when dealing with application scenarios that require a joint understanding of network topology and textual descriptions. For example, in scenarios such as automated network management, intelligent question answering, and fault diagnosis, the system cannot simultaneously utilize structured and unstructured information for comprehensive analysis and decision-making.

[0030] Therefore, there is an urgent need for a novel pre-training method that can effectively integrate network topology graphs and natural language, which can both understand the structural features of network topology and process related text descriptions, thereby achieving true multimodal joint learning and providing stronger technical support for intelligent network management.

[0031] To address the technical problems existing in related technologies, this disclosure provides a processing method based on joint pre-training, which can be applied to various types of task processing application scenarios.

[0032] Based on this Figure 1 The flowchart illustrating the processing method based on joint pre-training is shown in the image. (Reference) Figure 1 As shown, the main steps include: Step S110: Extract features from the network topology graph to obtain graph data representation, and convert the text description of the task to be processed into a text embedding representation.

[0033] In this embodiment, the input multimodal data is first preprocessed. Feature extraction can be performed on the network topology graph in the multimodal data, converting it into a node feature matrix and an adjacency matrix to obtain a graph data representation. A network topology graph is a structured data representation that abstracts devices in a network as nodes and the physical or logical connections between devices as edges; it is a core modality of data. The network topology graph can correspond to a task to be processed; different tasks may have different network topologies.

[0034] Each node represents a network device or connection point, and the edges represent the connection relationships between devices.

[0035] For example, the network topology diagram and text description are standardized. After feature extraction, network devices in the network topology diagram are abstracted as nodes, and the connections between network devices are abstracted as edges, forming a structured graph data representation. Specifically, neighbor information for each node can be aggregated, and the feature representation of each node can be updated progressively to determine the graph data representation; where neighbor information includes one or more of device type, connection status, and configuration parameters. This encoding process does not require manual feature design and can automatically learn a deep representation of the network structure.

[0036] For text descriptions, the text description of the task to be processed can be segmented into words to obtain the segmentation results. These segmentation results are then embedded through an embedding layer to transform them into a dense vector representation, which serves as the text embedding representation of the text description. The text description can be a natural language description; for example, in the case of an intelligent question-answering task, the text description can be a text question.

[0037] In the data preprocessing stage, the network topology graph is converted into node feature matrices and adjacency matrices to determine the graph data representation; while the corresponding text description is transformed into a serialized vector representation through word segmentation, embedding, and other processes to determine the text embedding representation. This preprocessing stage ensures that data from these two different modalities is received and effectively processed by subsequent modules.

[0038] In step S120, the correlation between the graph data representation and the text embedding representation is calculated through the graph neural network encoder, text encoder and cross-modal attention fusion module in the pre-trained model to obtain the target graph data representation and the target text embedding representation, and the target graph data representation and the target text embedding representation are mapped to a unified representation space; the pre-trained model is obtained by training the graph neural network encoder, text encoder and cross-modal attention fusion module according to the multi-task objective.

[0039] In this embodiment of the disclosure, during the multimodal pre-training stage, a bimodal data encoder structure combined with a cross-modal attention fusion module can be used to process the graph data representation of the network topology and the text embedding representation of the text description, resulting in bidirectional enhanced representation learning. The bimodal data encoder structure includes a graph neural network encoder and a text encoder.

[0040] The graph neural network encoder is responsible for encoding the graph data representation of the network topology, extracting the feature representations of nodes and edges from the graph data representation as the topology representation. A graph neural network is a deep learning model specifically designed for processing graph-structured data. It learns effective feature representations of nodes and the entire graph by using a "message passing" mechanism that allows nodes to aggregate information from their neighbors. In this embodiment, the graph neural network is used as a dedicated encoder to encode the network topology, responsible for extracting complex connections and dependencies between network devices.

[0041] Simultaneously, a Transformer-based text encoder is used to process the text embedding representation of the text description, generating a semantic representation of the text. These two processes work in parallel, providing a foundation for subsequent cross-modal interaction.

[0042] For example, a graph neural network encoder captures the local connectivity features and global structural information of the topological representation corresponding to the network topology graph through a multi-layer message passing mechanism as the topological representation. A Transformer-based text encoder utilizes a self-attention mechanism to extract the semantic features and contextual relationships of the text embedding representation corresponding to the text description as the text semantic representation. The Transformer text encoder encodes the input text embedding representation through a self-attention mechanism. The text embedding representation is first converted into a word vector sequence through word segmentation and embedding layers, and then processed through multiple Transformer blocks. The self-attention mechanism in each layer can capture long-distance dependencies and contextual semantics between words, ultimately outputting a context-aware vector representation of each word unit, which serves as the text semantic representation. This text semantic representation is rich in semantic information.

[0043] It should be noted that the graph neural network encoder and the Transformer-based text encoder output feature representations of the same dimension, that is, the topological structure representation and the text semantic representation are feature representations of the same dimension, which lays the foundation for subsequent cross-modal fusion.

[0044] Cross-modal attention fusion mechanisms can be represented by cross-modal attention fusion modules, which constitute the cross-modal attention fusion model. Cross-modal attention fusion mechanisms allow data from one modality to focus on relevant information in data from another modality. For example, it allows network topology graphs to focus on relevant information in text descriptions, enabling bidirectional interaction and mutual enhancement between different modalities. Cross-modal attention mechanisms achieve bidirectional information interaction by calculating the attention weights between graph node representations and text representations.

[0045] Specifically, the cross-modal attention fusion mechanism can perform correlation calculations between the topological structure representation obtained by the graph neural network encoder and the text semantic representation obtained by the Transformer-based text encoder. Through the bidirectional attention mechanism, dynamic alignment and complementarity between the graph structure and the text semantics can be achieved.

[0046] For example, the cross-modal attention fusion module achieves dynamic interaction between the two modalities by calculating the relevance weights between the topological representation and the text semantic representation. Specifically, the cross-modal attention fusion module calculates the attention of the topological representation relative to the text semantic representation, and the attention of the text semantic representation relative to the topological representation, to obtain the target graph data representation and the target text embedding representation.

[0047] Figure 2 The flowchart for determining the correlation is illustrated in the diagram. (Refer to...) Figure 2 As shown, the main steps include: Step S210: Calculate the first attention distribution of each node in the topological structure representation with all text semantic representations, and perform weighted aggregation of the text semantic representations according to the first attention distribution to generate a context-enhanced text representation corresponding to each node as the target text embedding representation.

[0048] The cross-modal attention fusion module calculates the first attention distribution between each node in the topological structure representation and all text semantic representations. Then, it performs weighted aggregation of the text semantic representations based on this attention distribution to generate a context-enhanced text representation corresponding to each node.

[0049] Step S220: For each text semantic representation, calculate the attention score of all nodes of each text semantic representation and the topological structure representation to obtain the second attention distribution and aggregate it to generate the target graph data representation.

[0050] The cross-modal attention fusion module performs reverse attention calculations for each node in the text semantic representation to the topological structure representation, so as to obtain the attention score of all nodes as the second attention distribution. Based on the second attention distribution, the vectors of all nodes are weighted and summed to generate the node related to each text semantic representation, thereby realizing bidirectional information flow and semantic alignment.

[0051] After obtaining the context-enhanced text representation corresponding to each node based on the first attention distribution, and the node corresponding to each text semantic representation based on the second attention distribution, the node focuses on the text description fragment corresponding to the relevant text semantic representation, and the text description fragment corresponding to the text semantic representation also focuses on the node in the relevant network topology graph, forming a bidirectional enhanced representation learning, which can realize bidirectional interaction and mutual enhancement between node representation and text representation. For example, the target graph data representation and the target text embedding representation can be used as a joint representation. The joint representation simultaneously captures network topology and text semantic information, and has stronger representational capabilities. The joint representation generated in this embodiment not only contains structural information but also incorporates rich semantic features, which can better support various downstream tasks. The joint representation can be a data pair composed of the target graph data representation and the target text embedding representation.

[0052] The bidirectional cross-modal attention mechanism enables both attention computation from nodes to text descriptions and attention distribution from text descriptions to nodes. This bidirectional attention mechanism dynamically captures the complex correspondences between topological elements and text descriptions, achieving multimodal semantic alignment. This unified representation space brings semantically related topological nodes and text descriptions closer together in the vector space, providing a solid foundation for cross-modal retrieval, generation, and understanding tasks.

[0053] In this embodiment of the disclosure, to improve accuracy, the graph neural network encoder, text encoder, and cross-modal attention fusion module can be trained according to multi-task objectives to obtain a pre-trained model. Exemplarily, during pre-training, a multi-task learning strategy is adopted, simultaneously optimizing multiple objective functions to achieve pre-training. These multiple objective functions can be multi-task loss functions. The multi-task learning strategy may include multi-task objectives, which may include a contrastive learning task, a masked language modeling task, and a masked node modeling task. Specifically, the contrastive learning task reduces the representation distance between relevant topological nodes and text descriptions and increases the representation distance between irrelevant samples by constructing positive and negative sample pairs, thus promoting cross-modal semantic alignment. The masked language modeling task randomly masks some text words, requiring the model to predict the masked content based on contextual information, enhancing text understanding capabilities. The masked node modeling task randomly masks some node features, requiring the model to reconstruct the masked node features based on neighborhood information, improving topological structure understanding capabilities.

[0054] Building upon this foundation, several objective functions are employed, including: a node-text contrastive learning objective function to reduce the representation distance between relevant samples and increase the distance between irrelevant samples; a masked language modeling objective function to enhance language understanding by predicting masked text words; and a masked node modeling objective function to improve topological understanding by reconstructing the features of masked nodes. Through multi-task collaborative optimization, the model learns a unified cross-modal representation space.

[0055] After determining a multi-task loss function representing multiple objective functions, training is performed using a large number of unlabeled graph-text pairs during the pre-training phase. For example, the parameters of the graph neural network encoder, text encoder, and cross-modal fusion module are jointly optimized using the multi-task loss function until the multi-task loss function is minimized, resulting in a trained graph neural network encoder, text encoder, and cross-modal fusion module as the pre-trained model. In some embodiments, training samples are generated by processing the sample graph data representation and sample text embedding representation according to the multi-task objective. All multi-task objectives do not require manual annotation, and the model can learn from a large number of unlabeled graph-text pairs, significantly reducing data preparation costs. The graph neural network encoder, text encoder, and cross-modal attention fusion module are trained based on multiple objective functions and training samples to obtain the pre-trained model. The sample graph data representation and sample text embedding representation belong to the unlabeled network topology graph and text description. The sample graph data representation and sample text embedding representation constitute the graph-text pairs used for training.

[0056] The entire pre-training phase employs an end-to-end training approach, jointly optimizing the parameters of the graph neural network encoder, the text encoder, and the cross-modal attention fusion module through backpropagation. It supports large-scale distributed training and exhibits excellent scalability and practicality. The pre-trained model can be quickly adapted to specific downstream tasks through fine-tuning, significantly reducing application development costs.

[0057] Through the aforementioned multi-task pre-training phase, the target graph data representation and the target text embedding representation are mapped to a unified representation space, resulting in a cross-modal representation. This unified representation space is jointly optimized through multi-task pre-training objectives such as contrastive learning, masked language modeling, and masked node modeling. The graph neural network encoder and text encoder map the topological structure representation and text semantic representation to a vector space of the same dimension, respectively, and align them through a cross-modal attention mechanism. During training, positive sample pairs (related graph-text pairs) are brought closer together, while negative sample pairs are pushed further apart, thus bringing semantically similar topological nodes and text descriptions closer together in the vector space.

[0058] In this unified representation space, nodes and text descriptions of semantically related network topology graphs are mapped to nearby vector regions, while nodes and text descriptions of unrelated network topology graphs are mapped to more distant vector regions. By mapping the target graph data representation and the target text embedding representation to a unified semantic space, information from different modalities becomes comparable and operable, providing a consistent semantic foundation for downstream tasks and supporting various cross-modal application scenarios.

[0059] By using pre-training tasks, high-quality cross-modal representations can be learned from unlabeled or weakly labeled data, which reduces the difficulty of data acquisition and improves accuracy.

[0060] In step S130, the pre-trained model is adjusted according to the task to be processed. The target graph data representation and target text embedding representation of the task to be processed, mapped to a unified representation space, are input into the adjusted pre-trained model, and the application result is output.

[0061] In this embodiment of the disclosure, during the fine-tuning and application phase, the pre-trained model can be fine-tuned for specific downstream tasks. Downstream tasks can be of any type, such as network intelligent question answering, network topology report generation, network fault diagnosis, topology optimization suggestion generation, operation and maintenance training assistance, etc.

[0062] For example, in the network intelligent question answering task, the system receives natural language questions from users about network topology and quickly retrieves or generates corresponding answers using the learned joint representation; in the network topology report generation task, the model can automatically generate structured text descriptions based on the input graph structure, which can be used for automated generation of network documents or output of fault analysis reports.

[0063] Furthermore, the embodiments disclosed herein can also be applied to multiple practical scenarios such as network fault diagnosis, topology optimization suggestion generation, and operation and maintenance training assistance, significantly improving the level of automation and intelligence in network management.

[0064] In this embodiment of the disclosure, the pre-trained model can be fine-tuned according to the task to be processed. For example, a task output layer is added to the pre-trained model according to the type of the task to be processed, resulting in an intermediate model. The task output layer can be determined based on the type of the task to be processed, such as a classification task or a generation task. When the task to be processed is a classification task, a classifier can be added to the pre-trained model; when the task to be processed is a generation task, a generator can be added to the pre-trained model, thereby adjusting the pre-trained model into an intermediate model that can be applied to downstream applications.

[0065] In addition, the intermediate model can be fine-tuned using labeled data from the task at hand to obtain an adjusted pre-trained model. Specifically, the labeled data can be a small amount of labeled data related to the task at hand. The parameters of the intermediate model can be fine-tuned based on this small amount of labeled data to obtain an adjusted pre-trained model specific to the task at hand. Because the pre-trained model already possesses good representational capabilities, the fine-tuning process is fast, requires little data, is highly efficient, and can be applied to any type of task.

[0066] After obtaining the adjusted pre-trained model, the target graph data representation and target text embedding representation corresponding to the task to be processed can be input into the adjusted pre-trained model, thereby outputting the application results corresponding to the target text embedding representation.

[0067] In this embodiment, the unified pre-trained model can be widely applied to various practical scenarios such as intelligent network question answering, automatic topology generation, and fault diagnosis report generation, significantly improving the level of network management automation.

[0068] When the task to be processed is intelligent question answering, the input data can be a natural language question and a network topology graph. The user-input natural language question can be encoded into a text embedding representation, and the network topology graph can be encoded into a graph data representation. By calculating the semantic similarity between the text embedding representation of the natural language question and the node or subgraph representation in the graph data representation, the most relevant network elements are identified, and an answer or suggestion is generated as the application result. For example, key information can be extracted from the network elements, mapped, and then converted into an answer or suggestion that conforms to natural language usage.

[0069] This process requires no manual rules and is entirely based on the cross-modal semantic understanding capabilities learned by the pre-trained model.

[0070] The technical solution in this disclosure improves the model's ability to understand and represent multimodal data by constructing a unified joint pre-training framework to achieve deep integration of graph structure and text semantics. By introducing a self-supervised learning mechanism and combining it with multi-task pre-training objectives, the model can learn effective cross-modal representations from raw data in unsupervised or weakly supervised environments, significantly reducing the need for manually labeled data and improving the method's practicality and scalability. Through the design of a cross-modal attention mechanism, the model can dynamically capture the complex relationships between graph nodes and text fragments, achieving bidirectional enhancement and complementarity of information. This not only improves the representation quality of a single modality but also enhances the model's generalization ability in downstream tasks. The learned unified representation space can be widely applied to various task scenarios such as intelligent network question answering, automatic topology graph generation, and fault diagnosis and report generation, expanding its application scope.

[0071] Figure 3 The diagram illustrates the overall process of executing downstream tasks. (See reference) Figure 3 As shown, it mainly includes: In the data preprocessing stage, graph structure features are extracted from the network topology graph to obtain the node feature matrix as the graph data representation. The text description is cleaned and segmented to obtain the text embedding representation.

[0072] In the multimodal pre-training stage, the graph data representation is input into the graph neural network encoder to obtain the topological structure representation, and the text embedding representation is input into the Transformer text encoder to obtain the text semantic representation. The attention between the topological structure representation and the text semantic representation is calculated according to the cross-modal attention fusion mechanism, and the attention between the text semantic representation and the topological structure representation is also calculated to obtain the target graph data representation and the target text embedding representation.

[0073] Furthermore, a pre-trained model is obtained by training a model based on multi-task learning objectives. These multi-task objectives can include node-text contrastive learning, masked language modeling, and masked node modeling. Training samples can be generated based on these multi-task objectives. The parameters of the graph neural network encoder, the Transformer text encoder, and the cross-modal attention fusion mechanism are then adjusted based on these training samples and the multi-task loss function to obtain the pre-trained model. Through the pre-trained model, the target graph data representation and the target text embedding representation can be mapped to a unified representation space, resulting in a cross-modal representation.

[0074] Building upon this, cross-modal representations can be used in downstream application phases to generate application results through pending tasks in these phases. These pending tasks can include network topology generation, text description generation, intelligent question answering, network fault diagnosis, and automated configuration generation, among others.

[0075] The technical solution in this embodiment of the disclosure, through a dedicated dual encoder architecture, enables the graph neural network encoder and the Transformer text encoder to process different modal data in parallel and perform deep fusion at the representation level, thereby realizing the deep fusion of network topology and natural language, creating a unified representation space, and solving the problem of multimodal information fusion.

[0076] Through a bidirectional cross-modal attention mechanism, attention computation from nodes to text descriptions can be achieved, as well as attention distribution from text descriptions to nodes. This bidirectional attention mechanism can dynamically capture the complex correspondence between topological elements and text descriptions, realizing multimodal semantic alignment.

[0077] By organically combining contrastive learning, masked language modeling, and masked node modeling, a multi-task collaborative optimization strategy can simultaneously enhance the model's ability to understand topological structure and capture textual semantics, achieving knowledge transfer and collaborative optimization across different modalities. Through self-supervised learning, the model can learn effective cross-modal representations from a large number of unlabeled graph-text pairs, significantly reducing the cost and workload of manual annotation. The learned joint representation captures both network topology and textual semantic information, exhibiting stronger expressive power and better generalization performance, enabling it to adapt to various downstream task requirements.

[0078] By using pre-training tasks, high-quality cross-modal representations can be learned from unlabeled or weakly labeled data, significantly reducing the dependence on labeled data and achieving effective pre-training through self-supervised learning.

[0079] A unified cross-modal representation space was constructed, enabling topological representations and text representations to map to the same semantic space. This unified representation space was learned directly through end-to-end training, simplifying the adaptation complexity for downstream tasks.

[0080] In some embodiments of this disclosure, a processing apparatus based on joint pre-training is also provided, with reference to... Figure 4 As shown, the processing device 400 based on joint pre-training includes: The feature extraction module 401 is used to extract features from the network topology graph to obtain graph data representation and convert the text description of the task to be processed into a text embedding representation. The joint training module 402 is used to calculate the correlation between the graph data representation and the text embedding representation through the graph neural network encoder, text encoder and cross-modal attention fusion module in the pre-trained model, to obtain the target graph data representation and the target text embedding representation, and to map the target graph data representation and the target text embedding representation to a unified representation space; the pre-trained model is obtained by training the graph neural network encoder, text encoder and cross-modal attention fusion module according to the multi-task objective; The task processing module 403 is used to adjust the pre-trained model according to the task to be processed, input the target graph data representation and target text embedding representation of the task to be processed into the adjusted pre-trained model, and output the application result.

[0081] In one exemplary embodiment of this disclosure, the step of calculating the correlation between the graph data representation and the text embedding representation through a graph neural network encoder, a text encoder, and a cross-modal attention fusion module in a pre-trained model to obtain the target graph data representation and the target text embedding representation includes: The graph data representation is extracted using a graph neural network encoder; The text semantic representation of the text embedding representation is determined based on the text encoder; The attention of the topological representation relative to the text semantic representation is calculated using the cross-modal attention fusion module, and the attention of the text semantic representation relative to the topological representation is also calculated to obtain the target graph data representation and the target text embedding representation.

[0082] In one exemplary embodiment of this disclosure, the step of calculating the attention of the topological representation relative to the text semantic representation according to the cross-modal attention fusion module, and calculating the attention of the text semantic representation relative to the topological representation to obtain the target graph data representation and the target text embedding representation, includes: Calculate the first attention distribution between each node in the topological structure representation and all text semantic representations, and perform weighted aggregation of the text semantic representations according to the first attention distribution to generate a context-enhanced text representation corresponding to each node as the target text embedding representation; For each text semantic representation, the attention score of each text semantic representation and all nodes of the topological structure representation is calculated to obtain the second attention distribution, which is then aggregated to generate the target graph data representation.

[0083] In one exemplary embodiment of this disclosure, the method further includes: Training samples are generated by processing the sample graph data representation and sample text embedding representation according to the multi-task objectives. The graph neural network encoder, text encoder, and cross-modal attention fusion module are trained based on multiple objective functions and training samples to obtain a pre-trained model.

[0084] In one exemplary embodiment of this disclosure, the multi-task objectives include node-text contrastive learning, masked language modeling, and masked node modeling.

[0085] In one exemplary embodiment of this disclosure, adjusting the pre-trained model according to the task to be processed includes: According to the type of the task to be processed, a task output layer is added to the pre-trained model to obtain an intermediate model; The intermediate model is fine-tuned using the labeled data of the task to be processed to obtain the adjusted pre-trained model.

[0086] In one exemplary embodiment of this disclosure, the step of extracting features from the network topology graph to obtain graph data representation includes: Graph neural networks abstract each network device in a network topology graph as a node in the graph, and the connection relationships between network devices as edges; The neighbor information of each node is aggregated, and the feature representation of the node is updated step by step to determine the graph data representation; the neighbor information includes one or more of the following: device type, connection status, and configuration parameters.

[0087] It should be noted that the specific details of each part of the above-mentioned processing device based on joint pre-training have been described in detail in some implementations of the corresponding methods. For details that are not disclosed, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0088] Exemplary embodiments of this disclosure also provide an electronic device. This electronic device may be the aforementioned terminal device or server. Generally, the electronic device may include a processor and a memory, the memory storing executable instructions of the processor, and the processor configured to execute the aforementioned processing method based on joint pre-training by executing the executable instructions. Furthermore, the electronic device may also include a display for displaying an operating interface.

[0089] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0090] The following reference Figure 5 To describe an electronic device 500 according to such an embodiment of the present disclosure. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0091] like Figure 5 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), and a display unit 540.

[0092] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 510 can perform actions such as... Figure 1 The steps are shown.

[0093] Storage unit 520 may include readable media in the form of volatile storage units, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include read-only memory (ROM) 5203.

[0094] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0095] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0096] Electronic device 500 can also communicate with one or more external devices 600 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] It should be noted that some embodiments of this disclosure also provide a computer program product, which includes a computer program that implements the above-described method when executed by a processor.

[0098] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0099] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0100] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0101] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to be executed by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure.

[0102] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0103] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0104] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A processing method based on joint pre-training, characterized in that, include: Feature extraction is performed on the network topology graph to obtain graph data representation, and the text description of the task to be processed is converted into a text embedding representation. The correlation between the graph data representation and the text embedding representation is calculated by the graph neural network encoder, text encoder, and cross-modal attention fusion module in the pre-trained model to obtain the target graph data representation and the target text embedding representation, and the target graph data representation and the target text embedding representation are mapped to a unified representation space; the pre-trained model is obtained by training the graph neural network encoder, text encoder, and cross-modal attention fusion module according to the multi-task objective; The pre-trained model is adjusted according to the task to be processed. The target graph data representation and target text embedding representation of the task to be processed, mapped to a unified representation space, are input into the adjusted pre-trained model, and the application results are output.

2. The processing method based on joint pre-training according to claim 1, characterized in that, The step of calculating the correlation between the graph data representation and the text embedding representation through the graph neural network encoder, text encoder, and cross-modal attention fusion module in the pre-trained model to obtain the target graph data representation and the target text embedding representation includes: The graph data representation is extracted using a graph neural network encoder; The text semantic representation of the text embedding representation is determined based on the text encoder; The attention of the topological representation relative to the text semantic representation is calculated based on the cross-modal attention fusion module, and the attention of the text semantic representation relative to the topological representation is also calculated to obtain the target graph data representation and the target text embedding representation.

3. The processing method based on joint pre-training according to claim 2, characterized in that, The step of calculating the attention of the topological representation relative to the text semantic representation based on the cross-modal attention fusion module, and calculating the attention of the text semantic representation relative to the topological representation to obtain the target graph data representation and the target text embedding representation, includes: Calculate the first attention distribution between each node in the topological structure representation and all text semantic representations, and perform weighted aggregation of the text semantic representations according to the first attention distribution to generate a context-enhanced text representation corresponding to each node as the target text embedding representation; For each text semantic representation, calculate the attention score of each text semantic representation and all nodes of the topological structure representation to obtain the second attention distribution, and aggregate them to generate the target graph data representation.

4. The processing method based on joint pre-training according to claim 1, characterized in that, The method further includes: Training samples are generated by processing the sample graph data representation and sample text embedding representation according to the multi-task objectives. The graph neural network encoder, text encoder, and cross-modal attention fusion module are trained based on multiple objective functions and training samples to obtain a pre-trained model.

5. The processing method based on joint pre-training according to claim 1, characterized in that, The multi-task objectives include node-text contrastive learning, masked language modeling, and masked node modeling.

6. The processing method based on joint pre-training according to claim 1, characterized in that, The step of adjusting the pre-trained model according to the task to be processed includes: According to the type of the task to be processed, a task output layer is added to the pre-trained model to obtain an intermediate model; The intermediate model is fine-tuned using the labeled data of the task to be processed to obtain the adjusted pre-trained model.

7. The processing method based on joint pre-training according to claim 1, characterized in that, The process of extracting features from the network topology graph to obtain graph data representation includes: Graph neural networks abstract each network device in a network topology graph as a node in the graph, and the connection relationships between network devices as edges; The neighbor information of each node is aggregated, and the feature representation of the node is updated step by step to determine the graph data representation; the neighbor information includes one or more of the following: device type, connection status, and configuration parameters.

8. A processing device based on joint pre-training, characterized in that, include: The feature extraction module is used to extract features from the network topology graph corresponding to the task to be processed to obtain graph data representation, and to convert the text description of the task to be processed into a text embedding representation. The joint training module is used to calculate the correlation between the graph data representation and the text embedding representation through the graph neural network encoder, text encoder and cross-modal attention fusion module in the pre-trained model, to obtain the target graph data representation and the target text embedding representation, and to map the target graph data representation and the target text embedding representation to a unified representation space; the pre-trained model is obtained by training the graph neural network encoder, text encoder and cross-modal attention fusion module according to the multi-task objective; The task processing module is used to adjust the pre-trained model according to the task to be processed, input the target graph data representation and target text embedding representation of the task to be processed into the adjusted pre-trained model, and output the application results.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the processing method based on joint pre-training as described in any one of claims 1-7 by executing the executable instructions.

10. A computer program product, characterized in that, When the computer program is executed by the processor, it implements the processing method based on joint pre-training as described in any one of claims 1-7.

Images