Communication fault identification method and device based on cross-modal fusion, and electronic equipment

Abstract

The invention relates to the technical field of computers, and provides a communication fault identification method and device based on cross-modal fusion and electronic equipment, and the method comprises the steps: inputting multi-modal data in the operation process of a communication system into a feature extraction module in a fault identification model, and obtaining a feature vector of each modal data; inputting the feature vector of each modal data into a feature fusion module in a fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on an attention mechanism; inputting the cross-modal fusion feature vector into a fault prediction module in a fault recognition model to obtain a fault type and a corresponding fault position; the fault identification model is obtained by training based on the multi-modal sample data and the corresponding fault type label and the fault position label. According to the invention, cross-modal fusion is carried out on the feature vectors of each modal based on the attention mechanism to obtain the fused feature vector on the semantic level, so that the subsequent fault recognition efficiency and accuracy are improved during subsequent fault recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a communication fault identification method, device, and electronic equipment based on cross-modal fusion. Background Art

[0002] With the rapid development of communication technology and the increasing complexity of application scenarios, the types and manifestations of communication failures are becoming increasingly diverse. Traditional communication fault identification mainly faces the following two problems: First, the insufficient fusion of multimodal data (for example, device log text, video surveillance images, audio, and images, etc.) makes it impossible to effectively understand the intent of multimodal data; second, the lack of cross-modal association modeling mechanisms leads to fragmented communication fault characteristics and low fault identification efficiency and accuracy. Summary of the Invention

[0003] The present invention provides a communication fault identification method, device and electronic equipment based on cross-modal fusion, which are used to solve at least one of the above-mentioned technical problems existing in the prior art.

[0004] The present invention provides a communication fault identification method based on cross-modal fusion, comprising the following steps: Input the multimodal data during the operation of the communication system into the feature extraction module in the fault identification model to obtain the feature vector of each modal data; Inputting the feature vectors of each modal data into a feature fusion module in a fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on an attention mechanism; Inputting the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module; The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

[0005] According to a communication fault identification method based on cross-modal fusion provided by the present invention, the feature extraction module is implemented based on a multi-tower neural network architecture, including feature extraction towers corresponding to each modal data; The multimodal data during the operation of the communication system is input into the feature extraction module in the fault identification model to obtain the feature vectors of each modal data, including: Each modal data is input into the corresponding feature extraction tower respectively, and the feature vectors of each modal data output by each feature extraction tower are obtained.

[0006] According to a communication fault identification method based on cross-modal fusion provided by the present invention, the multimodal data includes: sensor data, device log data, video data, audio data and image data; the feature extraction tower includes: A sensor feature extraction tower, implemented based on a temporal convolutional network, is used to capture the temporal dependency of the sensor data and obtain a sensor feature vector; A log feature extraction tower, used to encode the context semantics in the device log data to obtain a log feature vector; A video feature extraction tower is used to extract temporal and spatial features of video frames in the video data based on a regional attention mechanism to obtain a video feature vector; An audio feature extraction tower is used to extract time-frequency features from the audio data and capture the dynamic pattern of the sound signal in the audio data changing over time to obtain an audio feature vector; The image feature extraction tower is used to extract the spatial features of the image based on the regional attention mechanism to obtain the image feature vector.

[0007] According to a communication fault identification method based on cross-modal fusion provided by the present invention, the feature fusion module includes: a spatiotemporal graph convolutional network and a gated graph attention module; Input the feature vectors of each modal data into the feature fusion module in the fault recognition model to obtain the cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism, including: The feature vector of each modal data is input into the spatiotemporal graph convolutional network, so that the spatiotemporal graph convolutional network constructs a four-dimensional feature tensor T={t,s,m,f i}, where f i represents the eigenvector of the i-th mode m, and t represents f i The corresponding time step, s represents f i The spatial location of the corresponding device; The four-dimensional feature tensor T is input into the gated graph attention module so that the gated graph attention module calculates the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T. i Based on the correlation, the gated graph attention mechanism is used to update the feature vector f i and the corresponding weights, and update the eigenvectors f i The corresponding weights are weighted to fuse into the cross-modal fusion feature vector.

[0008] According to a communication fault identification method based on cross-modal fusion provided by the present invention, the fault identification model also includes the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T. iThe learnable parameters corresponding to each item are used to characterize the contribution of spatiotemporal changes to different modalities, so that the gated graph attention module can be based on the learnable parameters to the feature vectors f of different modal data. i Perform dynamic weight allocation.

[0009] According to a communication fault identification method based on cross-modal fusion provided by the present invention, the fault prediction module includes: a neural network and a Transformer network based on a multi-task learning framework; The neural network based on the multi-task learning framework is used to extract features related to the fault type and features related to the fault location from the cross-modal fusion feature vector; The Transformer network is used to output the predicted fault type and fault location based on the spatiotemporal self-attention mechanism, features related to the fault type, and features related to the fault location.

[0010] According to a communication fault identification method based on cross-modal fusion provided by the present invention, the training method of the fault identification model is as follows: Input the multimodal sample data into the initial fault identification model to obtain the fault type prediction value and fault location prediction value output by the fault identification model; The fault type prediction value and the corresponding fault type label are substituted into the fault type loss function, and the fault location prediction value and the corresponding fault location label are substituted into the fault location loss function. When the weighted sum of the fault type loss function and the fault location loss function converges, the fault identification model training is completed.

[0011] According to a communication fault identification method based on cross-modal fusion provided by the present invention, after inputting the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module, the method further includes: Based on the fault type and the corresponding fault location, a corresponding fault decision is selected to handle the fault.

[0012] The present invention also provides a communication fault identification device based on cross-modal fusion, comprising the following modules: A multimodal feature extraction unit is used to input the multimodal data during the operation of the communication system into the feature extraction module in the fault identification model to obtain the feature vector of each modal data; A multimodal feature fusion unit, configured to input the feature vectors of each modal data into a feature fusion module in a fault recognition model, and obtain a cross-modal fusion feature vector output by the feature fusion module based on an attention mechanism; A prediction module execution unit, configured to input the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain a fault type and a corresponding fault location predicted by the fault prediction module; The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, it implements any of the above-described communication fault identification methods based on cross-modal fusion.

[0014] The communication fault identification method, device and electronic device based on cross-modal fusion provided by the present invention, through the communication fault identification method based on cross-modal fusion of this embodiment, extracts the feature vectors of multimodal data during device operation, and cross-modally fuses the feature vectors of each modality based on the attention mechanism to obtain a cross-modal fusion feature vector fused at the semantic level, so that the intention of the multimodal data can be effectively understood in subsequent fault identification. Moreover, the cross-modal fusion feature vector fused at the semantic level avoids the fragmentation of communication fault features, thereby improving the efficiency and accuracy of subsequent fault identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 It is a flow chart of the communication fault identification method based on cross-modal fusion provided by the present invention.

[0017] Figure 2 It is a structural diagram of the communication fault identification device based on cross-modal fusion provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The communication fault identification method based on cross-modal fusion in the embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps S110 to S130.

[0021] Step S110: Input the multimodal data generated during the operation of the communication system into the feature extraction module of the fault identification model to obtain feature vectors for each modal data. The multimodal data generated during the operation of each communication device in the communication system comprises various types of device operating status data, specifically including sensor data, device log data, video data, audio data, and image data. Sensor data includes data such as ambient temperature and humidity, network traffic, and device temperature, and can be collected by corresponding sensors (ambient temperature and humidity sensors, traffic sensors, and device temperature sensors). Video data and image data include video and images captured on-site of the device's exterior, as well as video and images of personnel operating the device. Audio data includes device sound signals monitored during operation, including alarm sounds and fan start / stop sounds. Device log data is internal logs recorded by the device during operation, including device IDs, operation logs, and alarm information.

[0022] In this step, the feature extraction module extracts features from each of the above modal data to obtain feature vectors for each modal data. Of course, before feature extraction, each modal data is also pre-processed (such as noise removal, standardization, and dimensionality reduction).

[0023] Step S120: Input the feature vectors of each modal data into the feature fusion module in the fault identification model to obtain a cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism. In this step, because the feature fusion module fuses the feature vectors of each modality based on the attention mechanism, the feature vectors of each modality are semantically associated during the fusion process, resulting in a semantically fused cross-modal fusion feature vector. This allows for effective understanding of the intent of multimodal data during subsequent fault identification. Furthermore, this semantically fused cross-modal fusion feature vector avoids fragmentation of communication fault features, improving the efficiency and accuracy of subsequent fault identification.

[0024] Step S130: Input the cross-modal fusion feature vector into a fault prediction module in a fault recognition model to obtain the fault type and corresponding fault location predicted by the fault prediction module. The fault recognition model is trained based on multimodal sample data and the fault type and fault location labels corresponding to the multimodal sample data.

[0025] The communication fault identification method based on cross-modal fusion of this embodiment extracts feature vectors of multimodal data during device operation and performs cross-modal fusion on the feature vectors of each modality based on an attention mechanism to obtain a cross-modal fusion feature vector fused at a semantic level. This enables effective understanding of the intent of the multimodal data during subsequent fault identification. Furthermore, the cross-modal fusion feature vector fused at a semantic level avoids fragmentation of communication fault features, thereby improving the efficiency and accuracy of subsequent fault identification.

[0026] In some embodiments, the feature extraction module is implemented based on a multi-tower neural network architecture, including feature extraction towers corresponding to each modal data.

[0027] Based on the feature extraction module of the aforementioned architecture, step S110 specifically includes inputting each modal data into its corresponding feature extraction tower, and obtaining feature vectors for each modal data output by each feature extraction tower. Specifically, feature extraction is performed on the multimodal data based on the MMSA-Space (Multimodal Semantic Alignment Space Model), and the multimodal data is mapped into a unified vector space using a multi-tower neural network architecture.

[0028] In this embodiment, a multi-tower neural network architecture is used to extract features from each modal data, which can extract features from each modal data in parallel, thereby improving the efficiency of feature extraction. Moreover, the multi-tower neural network architecture will fuse the semantic representations of each modal data at a specific layer of the neural network during feature extraction, that is, the feature vector extracted from a certain modal data also fuses the feature information of other modal data, which is more conducive to subsequent feature fusion.

[0029] In some embodiments, the multimodal data includes: sensor data, device log data, video data, audio data and image data; the feature extraction tower includes: sensor feature extraction tower, log feature extraction tower, video feature extraction tower, audio feature extraction tower and image feature extraction tower.

[0030] The sensor feature extraction tower, based on a temporal convolutional network (TCN), captures the temporal dependencies of sensor data and generates sensor feature vectors. Specifically, the preprocessed sensor data is fed into the tower, which uses the TCN to capture the temporal dependencies of the sensor data (such as ambient temperature and humidity, network traffic, and device temperature). This allows the tower to extract key features of the device's operating status and environmental changes. The tower ultimately outputs a sensor feature vector. For example, for device temperature data, the tower captures the temporal features of the device temperature.

[0031] The Log Feature Extraction Tower encodes the contextual semantics in the device log data to generate a log feature vector. Specifically, the Log Feature Extraction Tower can use BERT-wwm-ext (a pre-trained language model). The pre-processed device log data is input into the Log Feature Extraction Tower, which then encodes the contextual semantics of the device log data, capturing the deep semantic relationships within the logs and extracting key features, providing textual information for identifying fault type and location.

[0032] The video feature extraction tower is used to extract the temporal and spatial features of the video frames in the video data based on the regional attention mechanism to obtain a video feature vector. Specifically, the video feature extraction tower can be a VisionTransformer (ViT), which inputs the preprocessed video data into the video feature extraction tower, focuses on key areas in the video frame (such as red light warnings, personnel operation actions, equipment failure images, etc.) through the region attention mechanism, extracts the spatial and temporal features of the video frame, and identifies the time and location of the fault. In the video feature extraction tower, after the video frame is processed, for video frames containing faults, the video frame is converted into a video feature vector with specific fault features, which represents the fault information in the video frame.

[0033] The audio feature extraction tower is used to extract time-frequency features from the audio data and capture the dynamic patterns of the sound signals in the audio data over time, thereby obtaining an audio feature vector. Specifically, the audio feature extraction tower can be a Mel-CNN-GRU (Mel-CNN-Gated Recurrent Unit hybrid network). The preprocessed audio data is input into the audio feature extraction tower, which utilizes a convolutional neural network (CNN) and a gated recurrent unit (GRU) to extract time-frequency features from the audio data (such as fan start / stop sounds and alarm sounds) and capture the dynamic patterns of the sound signals over time. At the critical moment of identifying a fault (such as the silence after a fan stops or the change in the alarm sound), the output audio feature vector contains fault features, representing the key fault information in the audio data.

[0034] The image feature extraction tower is used to extract spatial features from the image data based on a regional attention mechanism, generating an image feature vector. Specifically, the image feature extraction tower can be a Vision Transformer (ViT). Image data is input into the image feature extraction tower, which uses a self-attention mechanism to focus on important areas in the image (such as damaged equipment parts and warning signs), extracting the spatial features of the image and generating an image feature vector. If a fault is detected in the image, the image feature vector can represent the fault information in the image.

[0035] In this embodiment, different feature extraction towers are used to perform feature extraction in parallel on sensor data, device log data, video data, audio data, and image data of the communication device, thereby improving feature extraction efficiency.

[0036] In some embodiments, the feature fusion module includes: a spatiotemporal graph convolutional network (ST-GCN) and a gated graph attention module. Based on the feature fusion module of this structure, step S120 specifically includes: The feature vector of each modal data is input into the spatiotemporal graph convolutional network, so that the spatiotemporal graph convolutional network constructs a four-dimensional feature tensor T={t,s,m,f i}, where f i represents the eigenvector of the i-th mode m, and t represents f i The corresponding time step, s represents f i The spatial location of the corresponding device.

[0037] The four-dimensional feature tensor T={t,s,m,f i} Input the gated graph attention module so that the gated graph attention module calculates the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T i Based on the correlation, the gated graph attention mechanism is used to update the feature vector f i and the corresponding weight b i , and update each feature vector f i According to the corresponding weight b i Weighted to fuse into the cross-modal fusion feature vector f fusion =∑f i *b i , b i is the feature vector f calculated by the gated graph attention mechanism i The dynamic weight of i ranges from 1 to N, where N represents the total number of modalities.

[0038] In this embodiment, by constructing a tensor, the spatiotemporal graph convolutional network can simultaneously process information of time, space, modality, and feature vectors, preserving the spatiotemporal correlation of the data and facilitating the understanding of the complex spatiotemporal dependencies of the data. By fusing the feature vectors of each modality through the gated graph attention mechanism, a cross-modal fusion feature vector with good fusion effect can be obtained, which is specifically manifested in the following technical effects: Efficient cross-modal information fusion: Through dynamic weighting and spatiotemporal dependency modeling, the accuracy of communication fault diagnosis is improved, ensuring the full utilization and effective fusion of multimodal data.

[0039] Enhanced adaptability and interpretability: By assigning dynamic weights to each mode, the model can adaptively adjust the contribution of each mode, improving adaptability in different fault scenarios.

[0040] Higher fault identification accuracy and positioning capability: The fused feature vector effectively supports rapid positioning and precise response to communication faults, significantly improving the accuracy of fault identification and the system's positioning capability.

[0041] The above technical effects enable this embodiment to provide an efficient and reliable communication fault identification solution, significantly improving the stability and fault response capability of the communication system.

[0042] In some embodiments, the fault identification model also includes the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T. i The corresponding learnable parameters a j , j=1,2,3,4, the learnable parameter a j It is used to characterize the contribution (i.e., importance) of spatiotemporal changes to different modalities so that the gated graph attention module can be based on the learnable parameter a j The eigenvector f of different modal data i Perform dynamic weight allocation. Learnable parameter a j During model training (backpropagation), learning and adjustments are made based on the actual characteristics of the data. Each modal's features in different dimensions (time, space, modality, and eigenvectors) are assigned a variable weight. The contribution of each modality is dynamically adjusted based on temporal and spatial variations to accommodate various communication fault scenarios. This optimizes the fusion of each modal's eigenvectors and improves the accuracy of communication fault identification and location. Specifically, for the learnable parameter a1 corresponding to time step t, time dimension adjustment (a1) automatically adjusts the data weight of each time step based on the characteristics of the time step (e.g., different fault moments have different sensitivity to data). For example, when a fault occurs in a specific time period, the model assigns a higher weight to that time step.

[0043] The learnable parameter a2 at the spatial position s indicates that the positions of different equipment components or camera perspectives may have different impacts on the prediction results under different fault scenarios. Through the learnable parameter a2, the model can automatically weight the data features of important areas according to the different spatial positions, especially when certain equipment components fail, focusing on the multimodal data of the area.

[0044] The learnable parameter a3 corresponding to mode m indicates that fault manifestations may vary depending on the data modalities (such as sensor data, device log data, image data, audio data, and video data). This learnable parameter a3 allows the model to dynamically adjust the weights of the feature vectors of each modality based on the characteristics of the current fault mode. For example, in the case of a device overload, sensor data may be more important than audio data.

[0045] For the eigenvector f i The corresponding learnable parameter a4 indicates that the feature vectors of each modality (such as sensor readings, device log features, image features, audio features, and video features) have different importance in fault diagnosis, that is, different weights. Through the learnable parameter a4, the model can weight the data according to the weight of the feature, thereby achieving dynamic adjustment of the feature importance.

[0046] In some embodiments, the fault prediction module includes: a neural network (DNN) and a Transformer network based on a multi-task learning framework (Multi-Task Learning, MTL).

[0047] The neural network based on the multi-task learning framework is used to extract features related to the fault type and features related to the fault location from the cross-modal fusion feature vector.

[0048] The Transformer network is used to output the predicted fault type and fault location based on the spatiotemporal self-attention mechanism, features related to the fault type, and features related to the fault location.

[0049] Specifically, input the cross-modal fusion feature vector f fusionIn the fault prediction module, a neural network within a multi-task learning framework extracts features related to the fault type and location from the cross-modal fusion feature vector. The spatiotemporal self-attention mechanism within the Transformer network architecture then captures the spatiotemporal dependencies between the features related to the fault type and the features related to the fault location, outputting the fault type and location. The spatiotemporal attention mechanism integrates temporal and spatial information on the basis of the traditional self-attention mechanism, allowing the model to automatically focus on important temporal and spatial features when processing multimodal data. Focusing on important temporal features refers to considering time series data (such as sensor data, audio data, and device log data) in multimodal data when training the fault recognition model. The spatiotemporal attention mechanism introduces the concept of temporal windows within the existing self-attention framework. By weightedly focusing on the relationship between the current time step and historical time steps, it enhances the model's ability to handle temporal dependencies in the temporal dimension. Among them, focusing on the important features of space means that when training the fault recognition model, for image data and video data, the spatiotemporal attention mechanism captures the features of key areas in the image and video frames through region attention (Region Attention) and spatial convolution. The model can adaptively weight in the spatial dimension, focusing on the most representative fault area in the image, and improving the contribution of image data and video data to the fault type.

[0050] The fault identification model outputs fault type and location by employing a multi-task learning (MTL) framework. This framework uses a single neural network (DNN) to extract potential fault features and patterns from a cross-modal fusion feature vector, simultaneously performing both fault type and location prediction tasks. Specifically, a shared DNN extracts common potential fault features, and two independent task heads (one for type prediction and one for location prediction) within the MTL framework output the fault type and location. Fault type prediction involves extracting fault-related features from the cross-modal fusion feature vector in the fully connected layers of the DNN. The softmax layer then outputs a probability distribution for each fault type (such as device overload, network outage, and hardware failure). Fault location prediction involves extracting information related to the fault location from the cross-modal fusion feature vector in the fully connected layers of the DNN. The output is one or more continuous values representing the fault location (such as the rack ID and the coordinates of a component).

[0051] In this embodiment, the MTL framework not only realizes the prediction of the fault type, but also realizes the prediction of the fault location, which helps to quickly locate the faulty device or the component of the faulty device during subsequent fault handling.

[0052] In some embodiments, the fault identification model is trained as follows: The multimodal sample data is input into the initial fault identification model to obtain the fault type prediction value and fault location prediction value output by the fault identification model.

[0053] The fault type prediction value and the corresponding fault type label are substituted into the fault type loss function, and the fault location prediction value and the corresponding fault location label are substituted into the fault location loss function. When the weighted sum of the fault type loss function and the fault location loss function converges, the fault identification model training is completed.

[0054] Specifically, the fault type loss function is the cross-entropy loss function, which is used to calculate the difference between the predicted type and the actual fault type label. The fault location loss function is the mean square error loss function (MSELoss), which is used to calculate the gap between the predicted fault location and the fault location label. total =L type + λL location , where L type is the fault type loss function, L location is the fault location loss function, λ is a hyperparameter used to control the weight balance between the two tasks, L total For L type and L location The total loss function of the two weights is total When converged, the fault identification model training is completed.

[0055] In this embodiment, through the weighted loss function, the fault identification model can optimize the prediction tasks of fault type and fault location during the training process, avoiding the high time cost and computational cost of training two models separately.

[0056] In some embodiments, after step S130 , the method further includes: selecting a corresponding fault decision to handle the fault based on the fault type and the corresponding fault location.

[0057] Specifically, based on the identified and located fault type and fault location cause, the fault cause can be determined by combining the real-time data of the fault and the relationship between the historical fault type and fault cause, and a fault decision can be selected to handle the fault based on the fault cause.

[0058] The communication fault identification device based on cross-modal fusion provided by the present invention is described below. The communication fault identification device based on cross-modal fusion described below and the communication fault identification method based on cross-modal fusion described above can refer to each other.

[0059] The communication fault identification device based on cross-modal fusion according to the embodiment of the present invention is as follows: Figure 2 As shown, it includes the following modules: The multimodal feature extraction unit 210 is configured to input the multimodal data generated during the operation of the communication system into the feature extraction module in the fault identification model to obtain a feature vector of each modal data.

[0060] The multimodal feature fusion unit 220 is used to input the feature vectors of each modal data into the feature fusion module in the fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism.

[0061] The prediction module execution unit 230 is configured to input the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module.

[0062] The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

[0063] The communication fault identification device based on cross-modal fusion of this embodiment extracts feature vectors of multimodal data during device operation and performs cross-modal fusion on the feature vectors of each modality based on an attention mechanism to obtain a cross-modal fusion feature vector fused at a semantic level. This allows the intent of the multimodal data to be effectively understood during subsequent fault identification. Furthermore, the cross-modal fusion feature vector fused at the semantic level avoids fragmentation of communication fault features, thereby improving the efficiency and accuracy of subsequent fault identification.

[0064] In some embodiments, the feature extraction module is implemented based on a multi-tower neural network architecture, including feature extraction towers corresponding to each modal data.

[0065] The multimodal feature extraction unit 210 is specifically configured to input each modal data into a corresponding feature extraction tower, and obtain a feature vector of each modal data output by each feature extraction tower.

[0066] In some embodiments, the multimodal data includes: sensor data, device log data, video data, audio data, and image data; the feature extraction tower includes: The sensor feature extraction tower is implemented based on a temporal convolutional network and is used to capture the temporal dependency of the sensor data and obtain a sensor feature vector.

[0067] The log feature extraction tower is used to encode the context semantics in the device log data to obtain a log feature vector.

[0068] The video feature extraction tower is used to extract the temporal and spatial features of the video frames in the video data based on the regional attention mechanism to obtain a video feature vector.

[0069] The audio feature extraction tower is used to extract the time-frequency features in the audio data and capture the dynamic pattern of the sound signal in the audio data changing over time to obtain an audio feature vector.

[0070] The image feature extraction tower is used to extract the spatial features of the image based on the regional attention mechanism to obtain the image feature vector.

[0071] In some embodiments, the feature fusion module includes: a spatiotemporal graph convolutional network and a gated graph attention module.

[0072] The multimodal feature fusion unit 220 is specifically used to input the feature vector of each modal data into the spatiotemporal graph convolutional network, so that the spatiotemporal graph convolutional network constructs a four-dimensional feature tensor T={t,s,m,f i}, where f i represents the eigenvector of the i-th mode m, and t represents f i The corresponding time step, s represents f i The spatial position of the corresponding device; the four-dimensional feature tensor T is input into the gated graph attention module so that the gated graph attention module calculates the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T i Based on the correlation, the gated graph attention mechanism is used to update the feature vector f i and the corresponding weights, and update the eigenvectors f i The corresponding weights are weighted to fuse into the cross-modal fusion feature vector.

[0073] In some embodiments, the fault identification model also includes the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T. i The learnable parameters corresponding to each item are used to characterize the contribution of spatiotemporal changes to different modalities, so that the gated graph attention module can be based on the learnable parameters to the feature vectors f of different modal data. i Perform dynamic weight allocation.

[0074] In some embodiments, the fault prediction module includes: a neural network and a Transformer network based on a multi-task learning framework.

[0075] The neural network based on the multi-task learning framework is used to extract features related to the fault type and features related to the fault location from the cross-modal fusion feature vector.

[0076] The Transformer network is used to output the predicted fault type and fault location based on the spatiotemporal self-attention mechanism, features related to the fault type, and features related to the fault location.

[0077] In some embodiments, the fault identification model is trained as follows: The multimodal sample data is input into the initial fault identification model to obtain the fault type prediction value and fault location prediction value output by the fault identification model.

[0078] The fault type prediction value and the corresponding fault type label are substituted into the fault type loss function, and the fault location prediction value and the corresponding fault location label are substituted into the fault location loss function. When the weighted sum of the fault type loss function and the fault location loss function converges, the fault identification model training is completed.

[0079] In some embodiments, the communication fault identification device based on cross-modal fusion also includes: a fault processing module, which is used to input the cross-modal fusion feature vector into the fault prediction module in the fault identification model, obtain the fault type and corresponding fault location predicted by the fault prediction module, and then select a corresponding fault decision to handle the fault based on the fault type and corresponding fault location.

[0080] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to execute a communication fault identification method based on cross-modal fusion, which includes: The multimodal data during the operation of the communication system is input into the feature extraction module in the fault identification model to obtain the feature vector of each modal data.

[0081] The feature vectors of each modal data are input into the feature fusion module in the fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism.

[0082] The cross-modal fusion feature vector is input into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module.

[0083] The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

[0084] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0085] On the other hand, the present invention further provides a computer program product, comprising a computer program, which may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the communication fault identification method based on cross-modal fusion provided by the above methods, which includes: The multimodal data during the operation of the communication system is input into the feature extraction module in the fault identification model to obtain the feature vector of each modal data.

[0086] The feature vectors of each modal data are input into the feature fusion module in the fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism.

[0087] The cross-modal fusion feature vector is input into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module.

[0088] The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

[0089] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the communication fault identification method based on cross-modal fusion provided by the above methods is implemented, and the method includes: The multimodal data during the operation of the communication system is input into the feature extraction module in the fault identification model to obtain the feature vector of each modal data.

[0090] The feature vectors of each modal data are input into the feature fusion module in the fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism.

[0091] The cross-modal fusion feature vector is input into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module.

[0092] The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0094] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A communication fault identification method based on cross-modal fusion, characterized in that: include: Input the multimodal data during the operation of the communication system into the feature extraction module in the fault identification model to obtain the feature vector of each modal data; Inputting the feature vectors of each modal data into a feature fusion module in a fault recognition model to obtain a cross-modal fusion feature vector output by the feature fusion module based on an attention mechanism; Inputting the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module; The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

2. The communication fault identification method based on cross-modal fusion according to claim 1 is characterized in that: The feature extraction module is implemented based on a multi-tower neural network architecture, including feature extraction towers corresponding to each modal data; The multimodal data during the operation of the communication system is input into the feature extraction module in the fault identification model to obtain the feature vectors of each modal data, including: Each modal data is input into the corresponding feature extraction tower respectively, and the feature vectors of each modal data output by each feature extraction tower are obtained.

3. The communication fault identification method based on cross-modal fusion according to claim 2 is characterized in that: The multimodal data includes: sensor data, device log data, video data, audio data and image data; the feature extraction tower includes: A sensor feature extraction tower, implemented based on a temporal convolutional network, is used to capture the temporal dependency of the sensor data and obtain a sensor feature vector; A log feature extraction tower, used to encode the context semantics in the device log data to obtain a log feature vector; A video feature extraction tower is used to extract temporal and spatial features of video frames in the video data based on a regional attention mechanism to obtain a video feature vector; An audio feature extraction tower is used to extract time-frequency features from the audio data and capture the dynamic pattern of the sound signal in the audio data changing over time to obtain an audio feature vector; The image feature extraction tower is used to extract the spatial features of the image based on the regional attention mechanism to obtain the image feature vector.

4. The communication fault identification method based on cross-modal fusion according to claim 1 is characterized in that: The feature fusion module includes: a spatiotemporal graph convolutional network and a gated graph attention module; Input the feature vectors of each modal data into the feature fusion module in the fault recognition model to obtain the cross-modal fusion feature vector output by the feature fusion module based on the attention mechanism, including: The feature vector of each modal data is input into the spatiotemporal graph convolutional network, so that the spatiotemporal graph convolutional network constructs a four-dimensional feature tensor T={t,s,m,f i }, where f i represents the eigenvector of the i-th mode m, and t represents f i The corresponding time step, s represents f i The spatial location of the corresponding device; The four-dimensional feature tensor T is input into the gated graph attention module so that the gated graph attention module calculates the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T. i Based on the correlation, the gated graph attention mechanism is used to update the feature vector f i and the corresponding weights, and update the eigenvectors f i The corresponding weights are weighted to fuse into the cross-modal fusion feature vector.

5. The communication fault identification method based on cross-modal fusion according to claim 4 is characterized in that: The fault identification model also includes the mode m, time step t, spatial position s and feature vector f in the four-dimensional feature tensor T. i The learnable parameters corresponding to each item are used to characterize the contribution of spatiotemporal changes to different modalities, so that the gated graph attention module can be based on the learnable parameters to the feature vectors f of different modal data. i Perform dynamic weight allocation.

6. The communication fault identification method based on cross-modal fusion according to claim 1 is characterized in that: The fault prediction module includes: a neural network and a Transformer network based on a multi-task learning framework; The neural network based on the multi-task learning framework is used to extract features related to the fault type and features related to the fault location from the cross-modal fusion feature vector; The Transformer network is used to output the predicted fault type and fault location based on the spatiotemporal self-attention mechanism, features related to the fault type, and features related to the fault location.

7. The communication fault identification method based on cross-modal fusion according to claim 1 is characterized in that: The fault identification model is trained as follows: Input the multimodal sample data into the initial fault identification model to obtain the fault type prediction value and fault location prediction value output by the fault identification model; The fault type prediction value and the corresponding fault type label are substituted into the fault type loss function, and the fault location prediction value and the corresponding fault location label are substituted into the fault location loss function. When the weighted sum of the fault type loss function and the fault location loss function converges, the fault identification model training is completed.

8. The communication fault identification method based on cross-modal fusion according to any one of claims 1 to 7, characterized in that: After inputting the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain the fault type and corresponding fault location predicted by the fault prediction module, the method further includes: Based on the fault type and the corresponding fault location, a corresponding fault decision is selected to handle the fault.

9. A communication fault identification device based on cross-modal fusion, characterized in that: include: A multimodal feature extraction unit is used to input the multimodal data during the operation of the communication system into the feature extraction module in the fault identification model to obtain the feature vector of each modal data; A multimodal feature fusion unit, configured to input the feature vectors of each modal data into a feature fusion module in a fault recognition model, and obtain a cross-modal fusion feature vector output by the feature fusion module based on an attention mechanism; A prediction module execution unit, configured to input the cross-modal fusion feature vector into a fault prediction module in a fault identification model to obtain a fault type and a corresponding fault location predicted by the fault prediction module; The fault identification model is obtained by training based on multimodal sample data and fault type labels and fault location labels corresponding to the multimodal sample data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the communication fault identification method based on cross-modal fusion according to any one of claims 1 to 8 is implemented.

Images