A method and device for predicting transmission failure based on multi-view network

Abstract

The present invention discloses a method and device for predicting transmission faults based on a multi-perspective network, belonging to the technical field of fault prediction. The method comprises: collecting real-time signal data of a transmission device, and converting the real-time signal data into a multi-perspective image set; the multi-perspective image set comprises multiple two-dimensional images under multiple perspective types; using a trained multi-perspective model to perform multi-perspective feature extraction and fusion on the multi-perspective image set to obtain a fusion feature map dominated by multiple different perspective types; fusing the fusion feature maps dominated by multiple different perspective types to obtain a total fusion feature map; using a fault prediction model, predicting the fault of the transmission device based on the total fusion feature map, thereby realizing a transmission fault prediction method with high real-time performance and more accurate prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of fault prediction, and in particular relates to a transmission fault prediction method and device based on a multi-view network. Background Art

[0002] The launcher is a crucial component of any aircraft system, responsible for accurately and reliably delivering the aircraft to its intended target. Ensuring stable launcher operation is essential for ensuring the aircraft's effective performance at critical moments. A launcher typically consists of multiple subsystems, including electrical, hydraulic, mechanical, guidance, and propulsion systems. These subsystems are interconnected, and a failure in any one can impact the operation of the entire launcher. For example, an electrical failure can disrupt signal transmission, a hydraulic failure can prevent the launcher from operating properly, and a mechanical failure can cause the aircraft to become stuck.

[0003] In order to ensure the stable operation of the launch device, domestic and foreign researchers have proposed various fault prediction methods based on the structural characteristics and working characteristics of the launch device. Traditional fault prediction methods include regular inspections, simple parameter monitoring, regular preventive maintenance, etc. Most methods are diagnostic and remedial work performed after the fault occurs, and the fault information has a certain lag.

[0004] Therefore, how to provide a launch failure prediction method with high real-time performance and more accurate prediction becomes an important issue. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a transmission fault prediction method and device based on a multi-view network.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a transmission fault prediction method based on a multi-view network, the transmission fault prediction method comprising:

[0008] Collecting real-time signal data of the transmitting device during operation, and converting the real-time signal data into a multi-view image set; the multi-view image set includes multiple two-dimensional images under multiple view types;

[0009] Using the trained multi-view model to perform multi-view feature extraction and fusion on the multi-view image set to obtain a plurality of fusion feature maps dominated by different view types;

[0010] Fusing the multiple fusion feature maps dominated by different view types to obtain an overall fusion feature map;

[0011] A fault prediction model is used to predict transmission faults based on the total fusion feature map.

[0012] Optionally, the multi-view feature extraction and fusion of the multi-view image set is performed using the trained multi-view model to obtain a plurality of fusion feature maps dominated by different view types, including:

[0013] Using the trained multi-view model, under each view type, feature extraction is performed on the multi-view image set to obtain a single-view feature map under each view type;

[0014] Through the multi-head self-attention mechanism, different weights are assigned to the single-view feature maps under each view type, and after fusion, multiple fused feature maps dominated by different view types are obtained.

[0015] Optionally, the viewing angle type includes temperature, voltage or vibration.

[0016] Optionally, using a fault prediction model to perform transmission fault prediction based on the total fusion feature map includes:

[0017] Performing feature sampling at different levels on the total fusion feature map to obtain a multi-scale feature map;

[0018] Performing feature fusion on the multi-scale feature map using a sampling feature pyramid and a path aggregation network to obtain a multi-scale fused feature map;

[0019] The fault prediction model is used to predict transmission faults based on the multi-scale fusion feature map.

[0020] Optionally, the fault prediction model is a YOLO-X model that uses Darknet53 as the backbone network.

[0021] Optionally, the multi-view model training method includes:

[0022] Constructing a multi-view sample image set; the multi-view sample image set includes sample images under multiple view types and a true fault label corresponding to each sample image; the true fault label corresponding to each sample image is used to mark the view type and location of the true fault signal data in the sample image;

[0023] Using the initial multi-view model to perform multi-view feature extraction and fusion on the multi-view sample image set to obtain a plurality of fused sample feature maps dominated by different view types;

[0024] Fusing the plurality of fused sample feature maps dominated by different view types to obtain an overall fused sample feature map;

[0025] Using the fault prediction model, performing transmission fault prediction based on the total fusion sample feature map to obtain predicted sample fault information; the predicted sample fault information includes the viewing angle type and location of the predicted fault signal data in the total fusion sample feature map;

[0026] Based on the difference between the predicted sample fault information and the true fault label, the initial multi-view model is iteratively trained using the squared error loss function and the cross entropy loss function until the loss converges, thereby obtaining a trained multi-view model.

[0027] Optionally, converting the real-time signal data into a multi-view image set includes:

[0028] sampling a plurality of signal segments from the real-time signal data;

[0029] Each signal segment is converted into a two-dimensional image using a signal-to-image conversion formula to obtain the multi-view image set.

[0030] In a second aspect, the present invention provides a transmission failure prediction device based on a multi-view network, the transmission failure prediction device comprising:

[0031] a conversion module, configured to collect real-time signal data of the transmitting device during operation and convert the real-time signal data into a multi-view image set; the multi-view image set includes a plurality of two-dimensional images under a plurality of viewing angle types;

[0032] An extraction module is used to perform multi-view feature extraction and fusion on the multi-view image set using the trained multi-view model to obtain a plurality of fusion feature maps dominated by different view types;

[0033] A fusion module, configured to fuse the plurality of fusion feature maps dominated by different view types to obtain an overall fusion feature map;

[0034] The prediction module is used to use the fault prediction model to predict the transmission fault based on the total fusion feature map.

[0035] The present invention provides a method for predicting transmission faults based on a multi-perspective network. The method collects real-time signal data of the transmission device during operation, converts the real-time signal data into a multi-perspective image set including multiple two-dimensional images under multiple perspective types, and uses a multi-perspective model to perform multi-perspective feature extraction and fusion on the multi-perspective image set, thereby constructing a fusion feature map with multi-perspective representation. The fusion feature map can more comprehensively and intuitively display the real-time state changes of the transmission device, making the prediction of transmission device faults more flexible, accurate and efficient.

[0036] In addition, a fault prediction model is used to predict transmission faults based on the total fusion feature map, which improves the prediction accuracy of targets of various scales, especially minor faults.

[0037] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a method for predicting transmission failures based on a multi-view network provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic structural diagram of a multi-view model provided by an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of a portion of the structure of a fault prediction model provided by an embodiment of the present invention;

[0041] Figure 4 This is a structural diagram of a transmission fault prediction device based on a multi-view network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0043] In order to solve the problems of hysteresis and low prediction accuracy in existing transmission fault prediction methods, the present invention provides a transmission fault prediction method based on a multi-view network. Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for predicting transmission failures based on a multi-view network provided by an embodiment of the present invention, which specifically includes the following steps:

[0044] Step S101 : collecting real-time signal data of the transmitting device during operation, and converting the real-time signal data into a multi-view image set; the multi-view image set includes a plurality of two-dimensional images under a plurality of view types.

[0045] In an embodiment of the present invention, the launch device will collect and transmit various key parameter data such as temperature, voltage, vibration, etc. in real time through sensors during the launch of the aircraft and during operation. These data are called real-time signal data.

[0046] In an embodiment of the present invention, one-dimensional real-time signal data is converted into a two-dimensional image in the data processing dimension to provide a richer perspective understanding.

[0047] In one implementation, converting real-time signal data into a multi-view image set includes:

[0048] sampling a plurality of signal segments from real-time signal data;

[0049] Using the signal-image conversion formula, each signal segment is converted into a two-dimensional image to obtain a multi-view image set.

[0050] In the embodiment of the present invention, firstly, real-time data at each viewing angle is collected, that is, a one-dimensional signal, and multiple lengths are obtained from the one-dimensional signal. signal segment;

[0051] A two-dimensional image is converted for each signal segment. Specifically, the signal segment is mapped to image coordinates to convert it into a two-dimensional image. In a two-dimensional image, each pixel represents the signal value at a point in time.

[0052] Specifically, a signal segment of real-time signal data can be randomly intercepted and then divided into multiple sub-segments. Each sub-segment contains multiple signal points. The sub-segments are then used as the horizontal coordinates and the signal points as the vertical coordinates. The pixel value of the corresponding pixel in the two-dimensional image is then obtained based on the signal value of each signal point, ultimately obtaining the converted two-dimensional image. In other words, the pixel value of each pixel in the two-dimensional image is obtained by converting the signal value of the signal point at each time point.

[0053] For example, a signal segment of a real-time data signal can be randomly intercepted and divided into sub-signal segments, each of which contains signal points, and then The sub-signal segments are used as the horizontal axis, The signal points are used as the vertical coordinates, and then the pixel value of the pixel corresponding to the signal point in the two-dimensional image is obtained according to the signal value of each signal point. For example, according to the first signal segment The signal value of the signal point corresponds to the horizontal coordinate of 1 and the vertical coordinate of The pixel value of the pixel point, finally after obtaining the pixel value of each pixel point in the two-dimensional image, the size is obtained. 2D image.

[0054] In an embodiment of the present invention, mapping the signal segments to image coordinates to obtain a two-dimensional image can be achieved using a signal-to-image conversion formula. The signal-to-image conversion formula includes:

[0055] ;

[0056] in, Indicates the first Row, No. The pixel value of the pixel point in the column is The first signal segment The signal value of each signal point is converted; ; ; represents the rounding function; Indicates the The first signal segment The signal value of a signal point. middle , Right now .

[0057] It is understood that if the collected real-time signal data includes temperature data and voltage data, the multi-view image set includes a two-dimensional image under the temperature perspective type and a two-dimensional image under the voltage perspective type. Similarly, if the collected real-time signal data includes temperature data, voltage data, and vibration data, the multi-view image set includes a two-dimensional image under the temperature perspective type, a two-dimensional image under the voltage perspective type, and a two-dimensional image under the vibration perspective type.

[0058] In step S102 , the trained multi-view model is used to perform multi-view feature extraction and fusion on the multi-view image set to obtain a plurality of fusion feature maps dominated by different view types.

[0059] In an embodiment of the present invention, if the multi-perspective image set includes two-dimensional images under a temperature perspective type, two-dimensional images under a voltage perspective type, and two-dimensional images under a vibration perspective type, then the fusion feature maps dominated by multiple different perspective types include a fusion feature map dominated by a temperature perspective type, a fusion feature map dominated by a voltage perspective type, and a fusion feature map dominated by a vibration perspective type.

[0060] In an embodiment of the present invention, before using the trained multi-view model to perform multi-view feature extraction and fusion on a multi-view image set, a preprocessing operation can be performed to optimize the two-dimensional image. Specific operations may include image scaling and standardization, data enhancement and image augmentation, etc.

[0061] Because different real-time signal data samples at varying speeds, the resulting 2D images vary in size. Therefore, image scaling and normalization can be used to adjust the 2D images to a fixed size to accommodate the input requirements of the multi-view model. The 2D images are then normalized, scaling the pixel values ​​to the range [0, 1]. Specifically, each grayscale image (i.e., the pixel value of the 2D image) is divided by 255 to achieve pixel value scaling. This operation eliminates dimensional differences and improves the numerical stability of the multi-view model.

[0062] Data enhancement specifically involves using random flipping and random rotation to increase data diversity and enhance the generalization ability of the network.

[0063] Image augmentation specifically involves applying pixel-level perturbations, such as noise and blur, to increase data diversity and improve the robustness of multi-view models. In this embodiment of the present invention, by pre-optimizing two-dimensional image data, it can be made more compatible with the input requirements of the multi-view model, thereby enhancing subsequent image processing, accelerating the convergence of the model network, and improving the accuracy of feature acquisition.

[0064] In one implementation, a trained multi-view model is used to perform multi-view feature extraction and fusion on a multi-view image set to obtain multiple fusion feature maps dominated by different view types, including:

[0065] Using the trained multi-view model, feature extraction is performed on the multi-view image set under each view type to obtain a single-view feature map under each view type;

[0066] Through the multi-head self-attention mechanism, different weights are assigned to the single-view feature maps under each view type, and after fusion, multiple fused feature maps dominated by different view types are obtained.

[0067] See also Figure 2 , Figure 2 It is a structural diagram of a multi-perspective model provided by an embodiment of the present invention. The multi-perspective model includes multiple sub-perspective branch models, and each sub-perspective branch model processes operations under a corresponding perspective type. For example, the multi-perspective model includes a sub-perspective branch model corresponding to perspective 1, a sub-perspective branch model corresponding to perspective 2, and a sub-perspective branch model corresponding to perspective 3; wherein perspective 1, perspective 2, and perspective 3 can be temperature, voltage, and vibration, respectively. In an embodiment of the present invention, each sub-perspective branch model includes a single feature extraction module, a first residual connection and a layer normalization layer, a dominant fusion module, a second residual connection and a layer normalization layer, a feedforward network, and a third residual connection and a layer normalization layer;

[0068] A single feature extraction module is used to extract features from a multi-view image set under each view type to obtain a single view feature map under each view type;

[0069] When the sub-perspective branch model is a temperature perspective branch model, the single feature extraction module in the sub-perspective branch model is used to extract the temperature feature map; when the sub-perspective branch model is a voltage perspective branch model, the single feature extraction module in the sub-perspective branch model is used to extract the voltage feature map; when the sub-perspective branch model is a vibration perspective branch model, the single feature extraction module in the sub-perspective branch model is used to extract the vibration feature map.

[0070] In an embodiment of the present invention, one input end of the first residual connection and the layer normalization layer is connected to the input end of the single feature extraction module, and the other input end is connected to the output end of the single feature extraction module. The residual connection avoids the problems of gradient vanishing and gradient exploding, and the layer normalization stabilizes the data distribution, accelerates the calculation convergence, and is more suitable for dynamic computing scenarios.

[0071] The output of the first residual connection and the layer normalization layer is connected to a dominant fusion module. This module uses a multi-head self-attention mechanism to assign different weights to the single-view feature maps under different view types. After fusing, it obtains multiple fused feature maps dominated by different view types.

[0072] In the embodiment of the present invention, after obtaining the temperature characteristic diagram , voltage characteristic diagram and vibration characteristic diagram After that, all kinds of feature maps are normalized to the same dimension, such as mapping them to dimensions of vector.

[0073] The single-view feature maps from different view types are merged through the feature fusion layer to obtain the merged features :

[0074] ;

[0075] In this embodiment of the present invention, the leading fusion module employs a multi-head self-attention mechanism, using multiple independent "attention heads" to capture the complex relationships between features from different viewpoints in parallel. Each head independently calculates attention weights, and the results are then concatenated and fed into a feedforward network, avoiding the information limitations of a single head. Each attention head corresponds to one viewpoint type.

[0076] In an embodiment of the present invention, the dominant fusion module may be a temperature dominant fusion module, a voltage dominant fusion module, or a vibration dominant fusion module.

[0077] In one implementation, different weights are assigned to the temperature feature map, the voltage feature map, and the vibration feature map through a multi-head self-attention mechanism, and after fusion, a fused feature map dominated by the temperature perspective type, a fused feature map dominated by the voltage perspective type, and a fused feature map dominated by the vibration perspective type are obtained, including: using the multi-head self-attention mechanism, the temperature feature map, the voltage feature map, and the vibration feature map are fused according to a first weight assignment to obtain a fused feature map dominated by the temperature perspective type;

[0078] Through the multi-head self-attention mechanism, the temperature feature map, voltage feature map and vibration feature map are fused according to the second weight distribution to obtain a fused feature map dominated by the voltage perspective type;

[0079] Through the multi-head self-attention mechanism, the temperature feature map, voltage feature map and vibration feature map are fused according to the third weight distribution to obtain a fused feature map dominated by the vibration perspective type.

[0080] In an embodiment of the present invention, the temperature characteristic graph, the voltage characteristic graph, and the vibration characteristic graph are fused according to the first weight distribution to obtain a fused characteristic graph dominated by the temperature perspective type, that is, the temperature characteristic graph is dominant, and the voltage characteristic graph and the vibration characteristic graph are fused in collaboration. Specifically, a higher weight is assigned to the temperature characteristic graph, and relatively lower weights are assigned to the voltage characteristic graph and the vibration characteristic graph for fusion to obtain a fused characteristic graph dominated by the temperature perspective type.

[0081] In an embodiment of the present invention, the temperature characteristic graph, the voltage characteristic graph, and the vibration characteristic graph are fused according to the second weight distribution to obtain a fused characteristic graph dominated by the voltage perspective type, that is, the voltage characteristic graph is dominant, and the temperature characteristic graph and the vibration characteristic graph are fused in collaboration. Specifically, a higher weight is assigned to the voltage characteristic graph, and relatively lower weights are assigned to the temperature characteristic graph and the vibration characteristic graph for fusion, to obtain a fused characteristic graph dominated by the voltage perspective type.

[0082] In an embodiment of the present invention, the temperature characteristic graph, the voltage characteristic graph, and the vibration characteristic graph are fused according to the third weight distribution to obtain a fused characteristic graph dominated by the vibration perspective type, that is, the vibration characteristic graph is dominant, and the temperature characteristic graph and the voltage characteristic graph are fused in collaboration. Specifically, a higher weight is assigned to the vibration characteristic graph, and relatively lower weights are assigned to the temperature characteristic graph and the voltage characteristic graph for fusion, to obtain a fused characteristic graph dominated by the vibration perspective type.

[0083] In the embodiment of the present invention, the specific value of the weight distribution is determined during the iterative training of the multi-view model.

[0084] In an embodiment of the present invention, the acquisition of fused feature maps dominated by different view types can be achieved by using a multi-head self-attention mechanism. The specific algorithm is as follows:

[0085] First, let's explain the multi-head self-attention mechanism. In this mechanism, the query vector is used to find relevant information in the input sequence. It represents the information that the current model should focus on for a specific output. The key vector is used to match the query vector. Each element in the input sequence has a key vector that represents the characteristics or information of that element. The value vector corresponds to the key vector and is used to generate the final output.

[0086] In the multi-head self-attention mechanism, the query vector is mapped to the matching key vector, and finally the value vector corresponding to the most matching key vector is output.

[0087] In this embodiment of the present invention, for each attention head, that is, each view type, the query is calculated by scaling the dot product and key The similarity is normalized by the softmax function to obtain the attention matrix :

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] in, Indicates the The query vector (Query) in the attention head is used to query the potential connection between single-view feature maps under different view types during self-attention calculation; Indicates the The first weight matrix in the attention head; Indicates the The value vector (Value) in the attention head contains the actual information content of the input single-view feature map; Indicates the The second weight matrix in the attention head; Indicates the The key vector (Key) in the attention head is the same as Cooperate; Indicates the The third weight matrix in the attention head; Represents the dimension of the key; Indicates the The attention matrix corresponding to the attention head; superscript Represents the transpose operation of a matrix.

[0093] Use the attention matrix to weight the value vector and get the output of a single attention head :

[0094] ;

[0095] Among them, when When the attention head represents the temperature perspective type, Indicates the fusion feature map dominated by temperature perspective type; when When the attention head represents the voltage perspective type, Indicates the fusion feature map dominated by voltage perspective type; when When the attention head represents the vibration view type angle, A fused feature map indicating the dominance of vibration view types.

[0096] In an embodiment of the present invention, one input end of the second residual connection and the layer normalization layer is connected to the input end of the dominant fusion module, and the other input end is connected to the output end of the dominant fusion module. The residual connection avoids the problems of gradient vanishing and gradient exploding, and the layer normalization stabilizes the data distribution, accelerates the calculation convergence, and is more suitable for dynamic computing scenarios.

[0097] In the embodiment of the present invention, the input end of the feedforward network is connected to the second residual connection and the output end of the layer normalization layer, which can avoid the problem of information limitation of a single head.

[0098] In this embodiment of the present invention, one input of the third residual connection and layer normalization layer is connected to the input of the feedforward network, and the other input is connected to the output of the feedforward network. Ultimately, the third residual connection and layer normalization layer output a fused feature map dominated by different view types. Residual connections avoid the problems of vanishing and exploding gradients, and layer normalization stabilizes data distribution, accelerating computational convergence, making it more suitable for dynamic computing scenarios.

[0099] Step S103: fuse multiple fusion feature maps dominated by different view types to obtain an overall fusion feature map.

[0100] In an embodiment of the present invention, a feature splicing operation is performed on the aforementioned fusion feature maps dominated by multiple different perspective types, such as the fusion feature map dominated by the temperature perspective type, the fusion feature map dominated by the voltage perspective type, and the fusion feature map dominated by the vibration perspective type, to obtain a total fusion feature map.

[0101] Specifically, The outputs of the heads are concatenated and linearly transformed to obtain the total fusion feature map :

[0102] ;

[0103] in, Indicates feature concatenation operation; subscript Represents the total number of attention heads, which can be understood as the total number of perspective types in multiple perspectives; Represents a learnable matrix used to adapt the feature dimension.

[0104] Step S104: using the fault prediction model, perform transmission fault prediction based on the total fusion feature map.

[0105] In this embodiment of the present invention, before using the fault prediction model to predict transmission faults based on the total fused feature map, the total fused feature map can be further processed and optimized using an MLP (Multi-Layer Perceptron) module. This step aims to further enhance the expressive power of the total fused feature map through nonlinear transformations. The optimized fused feature map serves as input for subsequent transmission fault prediction operations.

[0106] In an embodiment of the present invention, a fault prediction model is used to predict transmission faults based on the total fusion feature map, including:

[0107] Perform feature sampling at different levels on the total fusion feature map to obtain a multi-scale feature map;

[0108] The multi-scale feature map is fused using the sampling feature pyramid and path aggregation network to obtain a multi-scale fused feature map;

[0109] The fault prediction model is used to predict transmission faults based on multi-scale fusion feature maps.

[0110] The fault prediction model is the YOLO-X model that uses Darknet53 as the backbone network. The fault prediction model is used to predict transmission faults. The specific operations are as follows:

[0111] See also Figure 3 , Figure 3 This is a partial structural diagram of the fault prediction model provided by an embodiment of the present invention. The fault prediction model uses Darknet53 as the backbone network and enhances feature extraction capabilities and reduces computational complexity by connecting cross-stage partial CSP (Cross Stage Partial) networks. Figure 3 Specifically, the structure of the YOLO-X network decoupling detection head is shown in FIG.

[0112] Specifically, in the backbone network of the YOLO-X model, the input total fusion features are first processed by multi-layer convolution, batch normalization (BatchNorm) and LeakyReLU activation function, and the basic features of different levels are gradually extracted to obtain the features including 、 and The multi-scale feature map of Figure 3 middle ;in Represents the height of the multi-scale feature map, represents the width of the multi-scale feature map, is the total number of target fault categories. P3, P4, and P5 are the three levels of features in the feature pyramid. The shallow network captures detailed information such as edges and textures to pinpoint the fault location; the deep network extracts semantic features to identify fault types, such as temperature anomalies and voltage fluctuations.

[0113] In an embodiment of the present invention, a cascade structure of Feature Pyramid Networks (FPN) and Path Aggregation Networks (PAN) is adopted to perform bidirectional fusion on the multi-scale feature map output by the backbone network to obtain a multi-scale fused feature map.

[0114] The top-down path in the feature pyramid network: upsampling the deep high-semantic features and laterally connecting them with the shallow high-resolution features to enhance the semantic information of the small fault targets;

[0115] Path aggregation network bottom-up path: downsamples shallow detail features and further integrates them with deep semantic features to improve the localization accuracy of large-scale faults.

[0116] In an embodiment of the present invention, the multi-scale fusion feature map includes three different scales, corresponding to different receptive fields of the two-dimensional image, to ensure the multi-scale detection capability of minor faults, such as early vibration anomalies, and significant faults, such as voltage drops.

[0117] For multi-scale feature maps, the YOLO-X network's decoupling head is used for fault prediction. The YOLO-X network's decoupling head consists of a classification branch, a regression branch, and an IoU branch.

[0118] The classification branch (Cls.) is used to predict the probability of each grid point in the multi-scale fusion feature map belonging to each type of fault. The fault types can be temperature anomalies, voltage fluctuations, excessive vibration, etc.

[0119] The regression branch (Reg.) is used to predict the center point coordinate offset of the fault target and width and height factors , thereby determining the location and size of the predicted fault area;

[0120] The IoU branch is used to estimate the intersection-over-union (IoU) ratio between the predicted box and the true box, and to assist in screening high-confidence prediction results.

[0121] In the embodiment of the present invention, the prediction results output by the detection heads of each scale are screened and optimized. First, the low confidence prediction frames are filtered according to a preset threshold, such as 0.3, and the confidence The calculation formula is:

[0122] ;

[0123] ;

[0124] in, Represents the prediction box, including predicted fault information; Indicates the real frame, including the real fault information; is the probability of failure, is the intersection-over-union (IoU) of the predicted box and the true box.

[0125] Then, non-maximum suppression is performed, using the highest confidence box as the reference box. The intersection-over-union (IoU) of the remaining predicted boxes with the reference box is calculated. The confidence level of predicted boxes with overlap exceeding a threshold, such as 0.6, is gradually reduced, ultimately retaining the optimal prediction box without redundancy. Finally, based on the scale mapping relationship between the multi-scale fusion feature map and the 2D image, the predicted center point offset and width and height parameters are restored to actual image coordinates to accurately locate the fault area.

[0126] In this embodiment of the present invention, the YOLO-X model's predictions are processed through non-maximum suppression (NMS) and confidence filtering to produce the final detection result. The fault signal region is the actual image location obtained by restoring the coordinates of the predicted box based on the mapping relationship between the multi-scale fusion feature map and the original two-dimensional image. The class label is the most likely fault category, such as temperature fault, voltage fault, or vibration fault, determined by the probability predicted by the classification branch.

[0127] In embodiments of the present invention, a fault prediction model can be used to identify and locate fault signal data. The fault prediction model generates a multi-scale fusion feature map containing the predicted fault signal data. The prediction map represents the coordinate location of the fault signal data within the multi-scale fusion feature map and its fault type. By tracing the multi-scale fusion feature map containing the fault signal data, it is possible to deduce the perspective performance parameters that constitute the real-time signal data causing the fault.

[0128] For example, if the prediction box indicates that the perspective type of the fault signal data is temperature, it can be determined that the temperature parameter constituting the real-time signal data is faulty. If the prediction box indicates that the perspective type of the fault signal data is vibration, it can be determined that the vibration parameter constituting the real-time signal data is faulty.

[0129] In an embodiment of the present invention, real-time signal data of the transmitting device is collected, the real-time signal data is converted into a multi-perspective image set including multiple two-dimensional images under multiple perspective types, and a multi-perspective model is used to perform multi-perspective feature extraction and fusion on the multi-perspective image set, and a fusion feature map with multi-perspective representation is constructed. The fusion feature map can more comprehensively and intuitively display the real-time state changes of the transmitting device, making the prediction of transmitting device failure more flexible, accurate and efficient.

[0130] In addition, a fault prediction model is used to predict faults during launch based on the total fusion feature map, which improves the prediction accuracy of targets of various scales, especially minor faults.

[0131] The embodiment of the present invention further provides a method for training a multi-view model, which specifically includes the following steps:

[0132] a) Construct a multi-view sample image set; the multi-view sample image set includes sample images under multiple view types and the true fault label corresponding to each sample image; the true fault label corresponding to each sample image is used to mark the view type and location of the true fault signal data in the sample image.

[0133] Taking temperature, voltage, and vibration as examples, a large number of multi-view image samples containing different fault types were collected and manually classified and labeled to obtain a multi-view sample image set. This multi-view sample image set, including multiple temperature, voltage, and vibration sample images, along with the corresponding real-world fault labels for each sample image, served as training data. During training, the network learned the pixel-level features of different fault types in the images, such as the color, texture, and shape of the fault area.

[0134] Combining the multi-view information of the image enables the network to accurately identify and mark the fault area from different angles, effectively avoiding the labeling errors caused by a single perspective and ensuring the accuracy and reliability of the labeling.

[0135] b) Using the initial multi-view model, multi-view feature extraction and fusion are performed on the multi-view sample image set to obtain fused sample feature maps dominated by multiple different view types.

[0136] In an embodiment of the present invention, multiple fused sample feature maps dominated by different perspective types include multiple fused sample feature maps dominated by temperature perspective types, multiple fused sample feature maps dominated by voltage perspective types, and multiple fused sample feature maps dominated by vibration perspective types.

[0137] The specific feature extraction and fusion process is as described above and will not be repeated here.

[0138] c) Fuse multiple fusion sample feature maps dominated by different view types to obtain the total fusion sample feature map.

[0139] Specifically, the temperature perspective type-dominated fusion sample feature map, the voltage perspective type-dominated fusion sample feature map, and the vibration perspective type-dominated fusion sample feature map are fused to obtain the total fusion sample feature map.

[0140] The specific feature fusion process is as described above and will not be repeated here.

[0141] d) Using the fault prediction model, transmission fault prediction is performed based on the total fusion sample feature map to obtain predicted sample fault information; the predicted sample fault information includes the perspective type and existence location of the predicted fault signal data in the total fusion sample feature map.

[0142] In the embodiment of the present invention, the fault prediction model is specifically a YOLO-X model. The specific process of using the YOLO-X model for fault prediction is as described above and will not be repeated here. The perspective type and location of the fault signal information are predicted in the final total fusion feature map.

[0143] e) Based on the difference between the predicted sample fault information and the true fault label, the squared error loss function and the cross entropy loss function are used to iteratively train the initial multi-view model until the loss converges, obtaining a trained multi-view model.

[0144] In the embodiment of the present invention, the difference between the predicted sample fault information and the real fault label specifically includes a difference in view angle type and a difference in existence position between the real fault signal data and the predicted fault signal data.

[0145] In the embodiment of the present invention, the position difference between the real fault signal data and the predicted fault signal data can be calculated using the mean square error (MSE) function. The mean square error (MSE) function is used to measure the prediction error of the center coordinates and width and height of the prediction area. The calculation formula is as follows:

[0146] ;

[0147] in, , Indicates the total number of sample images under multiple view types; Indicates the The location of the real fault signal data of each sample image; Indicates the The predicted location of the fault signal data for each sample image.

[0148] In the embodiment of the present invention, the difference in perspective type between the real fault signal data and the predicted fault signal data is used to measure whether the fault category of the predicted fault signal point is accurate, and the cross-entropy loss function can be used to calculate it. For each predicted fault signal data, the cross-entropy loss between its predicted fault perspective type and the perspective type of the real fault signal data is calculated. :

[0149] ;

[0150] in, , Indicates the total number of perspective types; Indicates the The predicted fault information of the sample image belongs to The true probability of each perspective type; Indicates the The predicted fault information of the sample image belongs to The predicted probability of each perspective type.

[0151] The multi-view model and fault prediction model are updated in a direction that minimizes the discrepancy between the predicted sample fault information and the true fault labels until the loss converges or the number of iterations reaches a preset number. This results in a trained multi-view model and fault prediction model. This model can then be used to perform transmission fault prediction based on the multi-view network.

[0152] Based on the same inventive concept, the embodiment of the present invention also provides a launch failure prediction device based on a multi-view network, see Figure 4 , Figure 4 : is a schematic structural diagram of a transmission failure prediction device based on a multi-view network provided by an embodiment of the present invention, the transmission failure prediction device comprising:

[0153] The conversion module 401 is used to collect real-time signal data when the transmitting device is running and convert the real-time signal data into a multi-view image set; the multi-view image set includes multiple two-dimensional images under multiple view types;

[0154] An extraction module 402 is configured to perform multi-view feature extraction and fusion on a multi-view image set using the trained multi-view model to obtain a plurality of fused feature maps dominated by different view types;

[0155] A fusion module 403 is configured to fuse multiple fusion feature maps dominated by different view types to obtain an overall fusion feature map;

[0156] The prediction module 404 is configured to use the fault prediction model to perform transmission fault prediction based on the total fusion feature map.

[0157] In an embodiment of the present invention, real-time signal data of the transmitting device during operation is collected, the real-time signal data is converted into a multi-perspective image set including multiple two-dimensional images under multiple perspective types, and a multi-perspective model is used to perform multi-perspective feature extraction and fusion on the multi-perspective image set, and a fusion feature map with multi-perspective representation is constructed. The fusion feature map can more comprehensively and intuitively display the real-time state changes of the transmitting device, making the prediction of transmitting device failure more flexible, accurate and efficient.

[0158] In addition, a fault prediction model is used to predict transmission faults based on the total fusion feature map, which improves the prediction accuracy of targets of various scales, especially minor faults.

[0159] Optionally, the extraction module 402 is specifically configured to:

[0160] Using the trained multi-view model, feature extraction is performed on the multi-view image set under each view type to obtain a single-view feature map under each view type; different weights are assigned to the single-view feature maps under each view type through a multi-head self-attention mechanism, and after fusion, a fusion feature map dominated by multiple different view types is obtained.

[0161] Optionally, the viewing angle type includes temperature, voltage, or vibration.

[0162] Optionally, the prediction module 404 is specifically configured to:

[0163] The total fusion feature map is sampled at different levels to obtain a multi-scale feature map. The multi-scale feature map is fused using a sampling feature pyramid and a path aggregation network to obtain a multi-scale fusion feature map. The launch fault prediction is performed based on the multi-scale fusion feature map.

[0164] Optionally, the fault prediction model is a YOLO-X model that uses Darknet53 as the backbone network.

[0165] Optional multi-view model training methods include:

[0166] Construct a multi-view sample image set; the multi-view sample image set includes sample images under multiple view types and the true fault label corresponding to each sample image; the true fault label corresponding to each sample image is used to mark the view type and location of the true fault signal data in the sample image;

[0167] The initial multi-view model is used to extract and fuse multi-view features from a set of multi-view sample images to obtain fused sample feature maps dominated by multiple different view types.

[0168] The total fusion sample feature map is obtained by fusing multiple fusion sample feature maps dominated by different view types;

[0169] Using the fault prediction model, transmission fault prediction is performed based on the total fusion sample feature map to obtain predicted sample fault information; the predicted sample fault information includes the view type and location of the predicted fault signal data in the total fusion feature map;

[0170] Based on the difference between the predicted sample fault information and the true fault label, the squared difference loss function and the cross entropy loss function are used to iteratively train the initial multi-view model until the loss converges, and the trained multi-view model is obtained.

[0171] Optionally, the conversion module 401 converts the real-time signal data into a multi-view image set, including:

[0172] Multiple signal segments are sampled from real-time signal data; each signal segment is converted into a two-dimensional image using a signal-to-image conversion formula to obtain a multi-view image set.

[0173] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0174] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0175] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "a" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0176] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] It should be noted that the device of an embodiment of the present invention is a device that applies the above-mentioned method for predicting transmission faults based on a multi-perspective network. All embodiments of the above-mentioned method for predicting transmission faults based on a multi-perspective network are applicable to the device and can achieve the same or similar beneficial effects.

[0178] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for predicting transmission failures based on a multi-view network, characterized in that: The launch failure prediction method comprises: Collecting real-time signal data of the transmitting device during operation, and converting the real-time signal data into a multi-view image set; the multi-view image set includes multiple two-dimensional images under multiple view types; Using the trained multi-view model to perform multi-view feature extraction and fusion on the multi-view image set to obtain a plurality of fusion feature maps dominated by different view types; Fusing the multiple fusion feature maps dominated by different view types to obtain an overall fusion feature map; Using a fault prediction model, a transmission fault prediction is performed based on the total fusion feature map; The trained multi-view model is used to perform multi-view feature extraction and fusion on the multi-view image set to obtain a plurality of fusion feature maps dominated by different view types, including: Using the trained multi-view model, under each view type, feature extraction is performed on the multi-view image set to obtain a single-view feature map under each view type; The multi-head self-attention mechanism is used to assign different weights to the single-view feature maps under different view types, and after fusion, multiple fused feature maps dominated by different view types are obtained; The fault prediction model is a YOLO-X model using Darknet53 as the backbone network; The training method of the multi-view model includes: Constructing a multi-view sample image set; the multi-view sample image set includes sample images under multiple view types and a true fault label corresponding to each sample image; the true fault label corresponding to each sample image is used to mark the view type and location of the true fault signal data in the sample image; Using the initial multi-view model to perform multi-view feature extraction and fusion on the multi-view sample image set to obtain a plurality of fused sample feature maps dominated by different view types; Fusing the plurality of fused sample feature maps dominated by different view types to obtain an overall fused sample feature map; Using the fault prediction model, performing transmission fault prediction based on the total fusion sample feature map to obtain predicted sample fault information; the predicted sample fault information includes the viewing angle type and location of the predicted fault signal data in the total fusion sample feature map; Based on the difference between the predicted sample fault information and the true fault label, the initial multi-view model is iteratively trained using the squared error loss function and the cross entropy loss function until the loss converges, thereby obtaining a trained multi-view model.

2. The method for predicting transmission failure according to claim 1, wherein: The viewing angle type includes temperature, voltage or vibration.

3. The method for predicting transmission failure according to claim 1, wherein: Using a fault prediction model, transmission fault prediction is performed based on the total fusion feature map, including: Performing feature sampling at different levels on the total fusion feature map to obtain a multi-scale feature map; Performing feature fusion on the multi-scale feature map using a sampling feature pyramid and a path aggregation network to obtain a multi-scale fused feature map; The fault prediction model is used to predict transmission faults based on the multi-scale fusion feature map.

4. The method for predicting transmission failure according to claim 1, wherein: Converting the real-time signal data into a multi-view image set includes: sampling a plurality of signal segments from the real-time signal data; Each signal segment is converted into a two-dimensional image using a signal-to-image conversion formula to obtain the multi-view image set.

5. A transmission failure prediction device based on a multi-view network, characterized in that: The launch failure prediction device includes: a conversion module, configured to collect real-time signal data of the transmitting device during operation and convert the real-time signal data into a multi-view image set; the multi-view image set includes a plurality of two-dimensional images under a plurality of viewing angle types; An extraction module is used to perform multi-view feature extraction and fusion on the multi-view image set using the trained multi-view model to obtain a plurality of fusion feature maps dominated by different view types; A fusion module, configured to fuse the plurality of fusion feature maps dominated by different view types to obtain an overall fusion feature map; A prediction module, configured to use a fault prediction model to predict transmission faults based on the total fusion feature map; The extraction module is specifically used to: Using the trained multi-view model, feature extraction is performed on the multi-view image set under each view type to obtain a single-view feature map under each view type; different weights are assigned to the single-view feature maps under each view type through a multi-head self-attention mechanism, and after fusion, multiple fused feature maps dominated by different view types are obtained; The fault prediction model is a YOLO-X model using Darknet53 as the backbone network; The training method of the multi-view model includes: Constructing a multi-view sample image set; the multi-view sample image set includes sample images under multiple view types and a true fault label corresponding to each sample image; the true fault label corresponding to each sample image is used to mark the view type and location of the true fault signal data in the sample image; Using the initial multi-view model to perform multi-view feature extraction and fusion on the multi-view sample image set to obtain a plurality of fused sample feature maps dominated by different view types; Fusing the plurality of fused sample feature maps dominated by different view types to obtain an overall fused sample feature map; Using the fault prediction model, performing transmission fault prediction based on the total fusion sample feature map to obtain predicted sample fault information; the predicted sample fault information includes the viewing angle type and location of the predicted fault signal data in the total fusion sample feature map; Based on the difference between the predicted sample fault information and the true fault label, the initial multi-view model is iteratively trained using the squared error loss function and the cross entropy loss function until the loss converges, thereby obtaining a trained multi-view model.

6. The launch failure prediction device according to claim 5, characterized in that: The viewing angle type includes temperature, voltage or vibration.

Images