Optical network multi-modal data fusion and alarm identification method and device

Abstract

The invention provides an optical network multi-modal data fusion and alarm identification method and device, and relates to the technical field of data processing, and the method comprises the steps: collecting multi-modal alarm data from an optical network, and inputting the multi-modal data into a multi-modal Transform model; the multi-modal Transform model fuses the data of different modals by using position embedding, cross-modal attention and self-attention mechanisms to obtain fused multi-modal data; and inputting the fused multi-modal data into a DNN model, and outputting an alarm identification result of the optical network.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for optical network multimodal data fusion and alarm identification. Background Art

[0002] In optical networks, alarm identification is critical for troubleshooting and network stability. However, modern network management systems are faced with massive and complex alarm data, including multimodal data from different devices, network layers, and management systems. These data are diverse in form and have certain correlations, which can complement each other.

[0003] Traditional alarm analysis methods usually rely on only a single type of data and are unable to effectively process multimodal data, resulting in low alarm recognition accuracy. Therefore, there is an urgent need for a method that can effectively integrate multimodal alarm data to achieve more comprehensive alarm recognition. Summary of the invention

[0004] The present invention provides an optical network multimodal data fusion and alarm identification method and device, which are used to solve the problem of how to effectively realize optical network multimodal data fusion and alarm identification in the prior art.

[0005] The present invention provides an optical network multimodal data fusion and alarm identification method, comprising: Collecting multimodal alarm data from an optical network and inputting the multimodal data into a multimodal Transformer model; The multimodal Transformer model uses position embedding, cross-modal attention, and self-attention mechanisms to fuse data from different modalities to obtain fused multimodal data. The fused multimodal data is input into a DNN model, and an alarm recognition result of the optical network is output.

[0006] According to an optical network multimodal data fusion and alarm identification method provided by the present invention, the multimodal alarm data is collected from the optical network, comprising: acquiring initial multimodal alarm data including performance data, alarm data and an alarm manual from the optical network; After data preprocessing, the initial multimodal alarm data is arranged row by row to obtain multimodal alarm data.

[0007] According to an optical network multimodal data fusion and alarm identification method provided by the present invention, the multimodal Transformer model uses position embedding, cross-modal attention and self-attention mechanisms to fuse data of different modalities to obtain fused multimodal data, including: Performing position embedding in the multimodal alarm data to obtain different modal data after position embedding; The different modal data after position embedding are fused through the cross-modal attention mechanism, and the outputs of the cross-modal Transformer sharing the same modality are connected through the self-attention mechanism to obtain the fused multi-modal data.

[0008] According to an optical network multimodal data fusion and alarm identification method provided by the present invention, position embedding is performed in the multimodal alarm data to obtain different modal data after position embedding, including: Obtaining the sequence length and embedding dimension of each modal alarm data to determine the position embedding vector of each modal alarm data; Based on the position embedding vector, the multimodal alarm data is position embedded to obtain different modal data after position embedding.

[0009] According to an optical network multimodal data fusion and alarm recognition method provided by the present invention, different modal data after position embedding are fused through a cross-modal attention mechanism, and the output of the cross-modal Transformer sharing the same modality is connected through a self-attention mechanism to obtain the fused multimodal data, including: The position-embedded data of different modalities are processed through a one-dimensional temporal convolutional layer. After extracting local features, the low-level signal of one modality is transformed into a set of different key-value pairs using a cross-modal attention module, and interacts with another modality to obtain the feature representation after cross-modal interaction. The feature representation after the cross-modal interaction is subjected to self-attention processing, and the outputs of the cross-modal Transformer sharing the same modality are connected to obtain fused multi-modal data.

[0010] The present invention also provides an optical network multimodal data fusion and alarm identification device, comprising: A collection module, used for collecting multimodal alarm data from the optical network and inputting the multimodal data into the multimodal Transformer model; The fusion module is used in the multimodal Transformer model to fuse data from different modalities using position embedding, cross-modal attention, and self-attention mechanisms to obtain fused multimodal data; The recognition module is used to input the fused multimodal data into the DNN model and output the alarm recognition result of the optical network.

[0011] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: acquiring initial multimodal alarm data including performance data, alarm data and an alarm manual from the optical network; After data preprocessing, the initial multimodal alarm data is arranged row by row to obtain multimodal alarm data.

[0012] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: Performing position embedding in the multimodal alarm data to obtain different modal data after position embedding; The different modal data after position embedding are fused through the cross-modal attention mechanism, and the outputs of the cross-modal Transformer sharing the same modality are connected through the self-attention mechanism to obtain the fused multi-modal data.

[0013] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: Obtaining the sequence length and embedding dimension of each modal alarm data to determine the position embedding vector of each modal alarm data; Based on the position embedding vector, the multimodal alarm data is position embedded to obtain different modal data after position embedding.

[0014] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: The position-embedded data of different modalities are processed through a one-dimensional temporal convolutional layer. After extracting local features, the low-level signal of one modality is transformed into a set of different key-value pairs using a cross-modal attention module, and interacts with another modality to obtain the feature representation after cross-modal interaction. The feature representation after the cross-modal interaction is subjected to self-attention processing, and the outputs of the cross-modal Transformer sharing the same modality are connected to obtain fused multi-modal data.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the optical network multimodal data fusion and alarm identification method as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the optical network multimodal data fusion and alarm identification method as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned optical network multimodal data fusion and alarm identification methods.

[0018] The optical network multimodal data fusion and alarm identification method and device provided by the present invention collect multimodal alarm data including performance data, alarm data and alarm manual from the optical network. The collected data is cleaned, deduplicated and integrated to ensure data quality. The preprocessed multimodal data is input into the multimodal Transformer model. The model uses position embedding, cross-modal attention and self-attention mechanisms to learn the interaction and correlation between different modal data and fuse them into a unified feature representation. The fused multimodal data is input into the DNN model, and after multi-layer calculation and feature extraction of the neural network, the alarm identification result is finally output. By fusing the optical network performance, alarm and manual data, the accuracy of alarm identification is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of a process flow of a method for optical network multimodal data fusion and alarm identification provided in an embodiment of the present application; Figure 2 A schematic diagram of a DNN network architecture provided in an embodiment of the present application; Figure 3 A schematic diagram of a multimodal data fusion module provided in an embodiment of the present application; Figure 4 A schematic diagram of a deep stack of cross-modal attention blocks provided in an embodiment of the present application; Figure 5 A schematic diagram of a comparison of multimodal data fusion algorithms provided in an embodiment of the present application; Figure 6 A schematic diagram of alarm identification comparison provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an optical network multimodal data fusion and alarm identification device provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Figure 1 A schematic diagram of a process flow of a multimodal data fusion and alarm identification method for an optical network provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, including: Step 110, collecting multimodal alarm data from the optical network, and inputting the multimodal data into a multimodal Transformer model; In the embodiment of the present application, alarm data in the optical network is collected, including performance indicators, alarm events and related documents, etc., and the data is cleaned and prepared to meet the requirements of model input. Useful features are extracted from the raw data to prepare for model training, and data of different modalities are integrated together to form a unified input format.

[0023] Optionally, collecting multimodal alarm data from the optical network includes: acquiring initial multimodal alarm data including performance data, alarm data and an alarm manual from the optical network; After data preprocessing, the initial multimodal alarm data is arranged row by row to obtain multimodal alarm data.

[0024] In the embodiment of the present application, performance data is obtained, which may include key performance indicators such as optical power, bit error rate, signal-to-noise ratio, etc. Alarm data is collected, which usually includes information such as alarm type, timestamp, location, severity, etc. Alarm manuals or documents are obtained, which usually contain alarm definitions, handling suggestions, and possible cause analysis. The acquisition of initial multimodal alarm data is completed.

[0025] Then, for the initial multimodal alarm data, identify and process missing values, outliers, or inconsistent data records. Standardize data formats, such as timestamps and units of measurement. Perform necessary numerical processing on performance data, such as normalization or scaling. Complete data preprocessing and use text processing techniques such as word segmentation, stop word removal, stemming, etc. to prepare text data. Convert data of different modalities into a unified format for processing by the multimodal Transformer model to obtain multimodal alarm data.

[0026] Step 120, the multimodal Transformer model uses position embedding, cross-modal attention and self-attention mechanisms to fuse data of different modalities to obtain fused multimodal data; In the embodiment of the present application, position embedding is used to provide the model with the position information of each element in the sequence, so that the model can understand the temporal or spatial order of the data.

[0027] In an embodiment of the present application, cross-modal attention is used to allow the model to exchange information between different modalities and capture the correlation between different data types.

[0028] In the embodiment of the present application, the self-attention mechanism is used to enable the elements within the sequence to influence each other, thereby capturing the long-distance dependencies within the sequence.

[0029] Then, the data processed by position encoding, cross-modal attention and self-attention are fused to form a unified feature representation. The fused multimodal data is obtained.

[0030] Step 130: input the fused multimodal data into a DNN model, and output an alarm recognition result of the optical network.

[0031] In an embodiment of the present application, the fused multimodal data is input into the DNN network to perform subsequent alarm recognition tasks.

[0032] Figure 2 A schematic diagram of the DNN network architecture provided in the embodiment of the present application is shown in FIG. Figure 2 As shown in Figure 1, the DNN network is a computational model that simulates the structure and function of the human brain neural network. Its basic unit is the neuron. Each neuron receives input from other neurons and changes the impact of the input on the neuron by adjusting the weight. The neural network can approximate complex functions through multiple layers of nonlinear hidden layers, achieving a universal approximation effect.

[0033] DNN can automatically extract useful features from raw data, greatly improving the generalization ability of the model. In addition, the highly nonlinear characteristics of DNN enable it to handle complex nonlinear relationships.

[0034] In DNN, multimodal data starts from the input layer, passes through the hidden layer layer by layer, and finally reaches the output layer. The output of each layer of neurons is used as the input of the next layer of neurons, and nonlinear transformation is achieved through activation functions. The training process of DNN relies on the back propagation algorithm and gradient descent algorithm. By calculating the error between the output layer and the true label, and back-propagating the error to each layer of neurons, the weights and bias items of the neurons are updated to minimize the error of alarm recognition.

[0035] The DNN model in the embodiment of the present application can effectively fuse the multi-modal data to output the alarm recognition results of the optical network, such as the alarm type, alarm severity, etc.

[0036] In an optional embodiment, the DNN model may be trained using a labeled alarm dataset, which includes input multimodal data and corresponding alarm labels.

[0037] Select a suitable loss function, such as cross entropy loss or mean square error, to measure the difference between the model prediction result and the true label. Use the gradient descent algorithm or other optimization algorithm to update the model parameters according to the gradient of the loss function to minimize the value of the loss function. The training of the DNN model has been implemented.

[0038] In an embodiment of the present application, multimodal alarm data including performance data, alarm data and alarm manuals are collected from the optical network. The collected data is cleaned, deduplicated and integrated to ensure data quality. The preprocessed multimodal data is input into the multimodal Transformer model. The model uses position embedding, cross-modal attention and self-attention mechanisms to learn the interactions and correlations between different modal data and fuse them into a unified feature representation. The fused multimodal data is input into the DNN model, and after multi-layer calculations and feature extraction of the neural network, the alarm recognition result is finally output. By fusing optical network performance, alarm and manual data, the accuracy of alarm recognition is significantly improved.

[0039] Optionally, the multimodal Transformer model utilizes position embedding, cross-modal attention, and self-attention mechanisms to fuse data of different modalities to obtain fused multimodal data, including: Performing position embedding in the multimodal alarm data to obtain different modal data after position embedding; The different modal data after position embedding are fused through the cross-modal attention mechanism, and the outputs of the cross-modal Transformer sharing the same modality are connected through the self-attention mechanism to obtain the fused multi-modal data.

[0040] In the embodiment of the present application, for each modality (such as performance data, alarm data, alarm manual) data sequence, a position code is generated, and the position code is added to the feature representation of the original data to form position-embedded data.

[0041] Specifically, the purpose of position embedding is to make the sequence carry temporal information, which is extended to :

[0042] in, Compute the embedding for each position index, are low-level location-aware features of different modalities.

[0043] Optionally, the positionally embedded data of different modalities are fused through a cross-modal attention mechanism, and the outputs of the cross-modal Transformer sharing the same modality are connected through a self-attention mechanism to obtain fused multi-modal data, including: The position-embedded data of different modalities are processed through a one-dimensional temporal convolutional layer. After extracting local features, the low-level signal of one modality is transformed into a set of different key-value pairs using a cross-modal attention module, and interacts with another modality to obtain the feature representation after cross-modal interaction. The feature representation after the cross-modal interaction is subjected to self-attention processing, and the outputs of the cross-modal Transformer sharing the same modality are connected to obtain fused multi-modal data.

[0044] Figure 3 A schematic diagram of a multimodal data fusion module provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, including position embedding, cross-modal attention and self-attention, in order to ensure that each element of the input sequence has sufficient awareness of its neighboring elements, the input sequence is passed through a one-dimensional temporal convolution layer:

[0045] where k {P, A, M} is the size of the convolution kernel of the modality {P, A, M} and d is a common dimension. The convolution sequence is expected to preserve the local structure of the sequence, which is crucial when collecting alarm data at different sampling rates.

[0046] The cross-modal transformer enables one modality to receive information from the other modality. Each modality continuously updates its sequence by incorporating low-level external information from a multi-head cross-modal attention module.

[0047] Specifically, the Multimodal Transformer uses a cross-modal attention block to integrate signals from different modalities. The cross-modal attention mechanism allows the model to directly focus on elements with strong signals or relevant information in other modalities without data alignment. Through the cross-modal attention block, the model can capture long-range cross-modal dependencies, which is difficult to achieve with traditional methods.

[0048] To simplify the calculation, Figure 4 A schematic diagram of a deep stack of cross-modal attention blocks provided in an embodiment of the present application is shown in FIG. Figure 4 As shown in the figure, all dimensions of each cross-modal attention block are fixed to d. Each cross-modal Transformer consists of D layers of cross-modal attention blocks, and the expression ability of the model is enhanced by stacking these layers. Taking the transmission of performance data to warning information as an example, the information of P mode will be transmitted to A mode through the cross-modal attention block, helping A mode to better understand the warning information.

[0049] The cross-modal Transformer performs feed-forward calculations according to the 𝑖=1,...,𝐷 layer:

[0050] Among them, 𝑓𝜃 is the positionwise feed-forward sublayer parameterized by 𝜃, represents the position at the 𝑖th layer In this process, each modality continuously updates its sequence through the underlying external information from the multi-head cross-modal attention module. At each level of the cross-modal attention block, the low-level signals from the source modality are transformed into a different set of key-value pairs to interact with the target modality.

[0051] Finally, self-attention is used to concatenate the outputs of cross-modal Transformers that share the same target modality to obtain , for example, ZA=[Z[D]P→A;Z[D]M→A]. Then, each of them passes through the sequence model to collect temporal information for prediction. Finally, the last element of the sequence model is extracted and output through the fully connected layer.

[0052] In an optional embodiment, the embodiment of the present application learns the alarm data of different modes of the optical network, and uses cross-attention and self-attention to capture the interaction between different modal data across different time steps without alignment. This method can mine the correlation between multimodal data to achieve the effect of information complementarity. The fused data can significantly improve the learning ability of subsequent machine learning algorithms during data training, thereby increasing the accuracy of alarm recognition. There is no need for expert experience and manual analysis of alarm manuals, which saves labor costs, achieves efficient multimodal information fusion, and improves the effectiveness of subsequent alarm-related tasks.

[0053] For example, based on the multimodal alarm data collected in a real OTN network operated by an operator, the performance data (P) includes optical power, FEC (forward error correction) and other parameters. The alarm data (A) includes the alarm time, alarm level and network element name. The alarm manual (M) contains the alarm name and possible causes.

[0054] Figure 5 The multimodal data fusion algorithm comparison diagram provided in the embodiment of the present application is as follows: Figure 5It is shown that the effects of three different multimodal data fusion algorithms, namely the multimodal Transformer model, Transformer, and Trans-Fusion, are compared through training time, cross-modal attention heat map, and feature correlation heat map. The multimodal Transformer model shows the shortest training time when reaching convergence, which verifies the superiority of the algorithm in multimodal data fusion.

[0055] Comparing the attention and correlation graphs between different modal data, we can see that the proposed multimodal Transformer model algorithm uses cross-modal attention to focus directly on elements with strong signals or other modality-related information. This ability enables the model to capture long-range cross-modal dependencies, which are difficult to reveal with traditional alignment methods, further confirming its effectiveness.

[0056] In the embodiment of the present application, the fused data is input into the DNN framework for alarm recognition, and the results are compared. The experimental results are evaluated using traditional alarm recognition indicators: accuracy (ACC) and F1 score (F1 score).

[0057] Figure 6 The alarm identification comparison diagram provided in the embodiment of the present application is as follows: Figure 6 The results show that, in a longitudinal comparison, when relying only on alarm data, the alarm recognition index remains stable. When fusing multiple modal data, the performance of the algorithm is significantly better than that of single modal data, indicating that data fusion significantly enhances the alarm recognition performance, affirms the positive impact of multimodal fusion, and reflects the effectiveness of multimodal data fusion.

[0058] In horizontal comparison, the multimodal Transformer model surpasses other alternatives in terms of ACC and F1 score. The data fused by this method significantly improves the performance of DNN-based alarm recognition, with an accuracy of up to 95.79% and an F1 score of up to 97.28%. This is because it integrates multimodal data using cross-modal attention, overcoming the alignment challenge that other methods cannot solve.

[0059] The optical network multimodal data fusion and alarm identification device provided by the present invention is described below. The optical network multimodal data fusion and alarm identification device described below and the optical network multimodal data fusion and alarm identification method described above can be referenced to each other.

[0060] Figure 7 A schematic diagram of the structure of the optical network multimodal data fusion and alarm identification device provided in the embodiment of the present application is shown in FIG. Figure 7 As shown, including: The collection module 710 collects multimodal alarm data in the optical network and inputs the multimodal data into the multimodal Transformer model; The fusion module 720 modal transformer uses position embedding, cross-modal attention and self-attention mechanisms to fuse data from different modalities to obtain fused multi-modal data; The identification module 730 inputs the fused multimodal data into the DNN model and outputs the alarm identification result of the optical network.

[0061] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: acquiring initial multimodal alarm data including performance data, alarm data and an alarm manual from the optical network; After data preprocessing, the initial multimodal alarm data is arranged row by row to obtain multimodal alarm data.

[0062] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: Performing position embedding in the multimodal alarm data to obtain different modal data after position embedding; The different modal data after position embedding are fused through the cross-modal attention mechanism, and the outputs of the cross-modal Transformer sharing the same modality are connected through the self-attention mechanism to obtain the fused multi-modal data.

[0063] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: Obtaining the sequence length and embedding dimension of each modal alarm data to determine the position embedding vector of each modal alarm data; Based on the position embedding vector, the multimodal alarm data is position embedded to obtain different modal data after position embedding.

[0064] According to an optical network multimodal data fusion and alarm identification device provided by the present invention, the device is also used for: The position-embedded data of different modalities are processed through a one-dimensional temporal convolutional layer. After extracting local features, the low-level signal of one modality is transformed into a set of different key-value pairs using a cross-modal attention module, and interacts with another modality to obtain the feature representation after cross-modal interaction. The feature representation after the cross-modal interaction is subjected to self-attention processing, and the outputs of the cross-modal Transformer sharing the same modality are connected to obtain fused multi-modal data.

[0065] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the optical network multimodal data fusion and alarm identification method as described in any one of the above is implemented.

[0066] In an embodiment of the present application, multimodal alarm data including performance data, alarm data and alarm manuals are collected from the optical network. The collected data is cleaned, deduplicated and integrated to ensure data quality. The preprocessed multimodal data is input into the multimodal Transformer model. The model uses position embedding, cross-modal attention and self-attention mechanisms to learn the interactions and correlations between different modal data and fuse them into a unified feature representation. The fused multimodal data is input into the DNN model, and after multi-layer calculations and feature extraction of the neural network, the alarm recognition result is finally output. By fusing optical network performance, alarm and manual data, the accuracy of alarm recognition is significantly improved.

[0067] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the optical network multimodal data fusion and alarm identification method, the method comprising: collecting multimodal alarm data from the optical network, and inputting the multimodal data into the multimodal Transformer model; The multimodal Transformer model uses position embedding, cross-modal attention, and self-attention mechanisms to fuse data from different modalities to obtain fused multimodal data. The fused multimodal data is input into a DNN model, and an alarm recognition result of the optical network is output.

[0068] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0069] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the optical network multimodal data fusion and alarm identification method provided by the above methods, the method comprising: collecting multimodal alarm data from the optical network, and inputting the multimodal data into a multimodal Transformer model; The multimodal Transformer model uses position embedding, cross-modal attention, and self-attention mechanisms to fuse data from different modalities to obtain fused multimodal data. The fused multimodal data is input into a DNN model, and an alarm recognition result of the optical network is output.

[0070] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the optical network multimodal data fusion and alarm identification method provided by the above methods, the method comprising: collecting multimodal alarm data from the optical network, and inputting the multimodal data into a multimodal Transformer model; The multimodal Transformer model uses position embedding, cross-modal attention, and self-attention mechanisms to fuse data from different modalities to obtain fused multimodal data. The fused multimodal data is input into a DNN model, and an alarm recognition result of the optical network is output.

[0071] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optical network multimodal data fusion and alarm identification method, characterized in that: include: Collecting multimodal alarm data from an optical network and inputting the multimodal data into a multimodal Transformer model; The multimodal Transformer model uses position embedding, cross-modal attention, and self-attention mechanisms to fuse data from different modalities to obtain fused multimodal data. The fused multimodal data is input into a DNN model, and an alarm recognition result of the optical network is output.

2. The optical network multimodal data fusion and alarm identification method according to claim 1, characterized in that: The collecting of multimodal alarm data from the optical network includes: acquiring initial multimodal alarm data including performance data, alarm data and an alarm manual from the optical network; After data preprocessing, the initial multimodal alarm data is arranged row by row to obtain multimodal alarm data.

3. The optical network multimodal data fusion and alarm identification method according to claim 1, characterized in that: The multimodal Transformer model uses position embedding, cross-modal attention and self-attention mechanisms to fuse data from different modalities to obtain fused multimodal data, including: Performing position embedding in the multimodal alarm data to obtain different modal data after position embedding; The different modal data after position embedding are fused through the cross-modal attention mechanism, and the outputs of the cross-modal Transformer sharing the same modality are connected through the self-attention mechanism to obtain the fused multi-modal data.

4. The optical network multimodal data fusion and alarm identification method according to claim 3 is characterized in that: Position embedding is performed in the multimodal alarm data to obtain different modal data after position embedding, including: Obtaining the sequence length and embedding dimension of each modal alarm data to determine the position embedding vector of each modal alarm data; Based on the position embedding vector, the multimodal alarm data is position embedded to obtain different modal data after position embedding.

5. According to the optical network multimodal data fusion and alarm identification method according to claim 3, it is characterized in that: The cross-modal attention mechanism is used to fuse the positionally embedded data of different modalities, and the self-attention mechanism is used to connect the outputs of the cross-modal Transformer that share the same modality to obtain the fused multi-modal data, including: The position-embedded data of different modalities are processed through a one-dimensional temporal convolutional layer. After extracting local features, the low-level signal of one modality is transformed into a set of different key-value pairs using a cross-modal attention module, and interacts with another modality to obtain the feature representation after cross-modal interaction. The feature representation after the cross-modal interaction is subjected to self-attention processing, and the outputs of the cross-modal Transformer sharing the same modality are connected to obtain fused multi-modal data.

6. An optical network multimodal data fusion and alarm identification device, characterized in that: include: A collection module, used for collecting multimodal alarm data from the optical network and inputting the multimodal data into the multimodal Transformer model; The fusion module is used in the multimodal Transformer model to fuse data from different modalities using position embedding, cross-modal attention, and self-attention mechanisms to obtain fused multimodal data; The recognition module is used to input the fused multimodal data into the DNN model and output the alarm recognition result of the optical network.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the optical network multimodal data fusion and alarm identification method as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the optical network multimodal data fusion and alarm identification method as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the optical network multimodal data fusion and alarm identification method as described in any one of claims 1 to 5 is implemented.

Images