Mechanical fault diagnosis method and system based on deep fusion of multi-modal monitoring data

Abstract

The invention discloses a mechanical fault diagnosis method and system based on deep fusion of multi-modal monitoring data. The method comprises the following steps: randomly selecting two kinds of modal target monitoring data from multi-modal target monitoring data; a corresponding target mechanical fault diagnosis model is determined according to the two kinds of modal target monitoring data, the target mechanical fault diagnosis model is obtained by training a sample monitoring data set with the same modal as the two kinds of modal target monitoring data, and the sample monitoring data set is acquired multi-modal monitoring data in the operation process of multiple kinds of mechanical equipment; the target mechanical fault diagnosis model comprises a symmetric fusion module and a fault classifier, the symmetric fusion module is used for carrying out bidirectional symmetric interaction on two kinds of modal target monitoring data, and the fault classifier is used for classifying mechanical faults; and inputting the two kinds of modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result. According to the invention, the accuracy of mechanical fault diagnosis can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of mechanical fault diagnosis, and in particular to a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data. Background Art

[0002] Mechanical equipment is widely used in modern industry. It works under a series of complex and harsh conditions such as heavy load, high speed and high temperature for a long time. Its key components (such as gears and bearings, etc.) will inevitably fail. Once a key component fails, it will bring huge operating risks to the mechanical equipment and even cause major safety accidents. Therefore, it is of great value to carry out fault diagnosis of key components of equipment.

[0003] Considering that vibration signals are sensitive to mechanical faults, mechanical fault diagnosis methods based on vibration signal analysis have always been the mainstream research direction in this field. With the development of artificial intelligence technology, especially various neural network models, intelligent fault diagnosis methods based on neural networks have been widely and deeply studied. Existing intelligent diagnosis methods have been able to achieve efficient processing of vibration data and fault feature extraction. However, the physical structure of large-scale mechanical equipment is complex, and the types of component failures are diverse. Different fault locations or types will cause abnormal vibration signals of equipment, making it difficult for vibration monitoring data to fully characterize the complex fault types of equipment. However, existing intelligent diagnosis methods are better at processing single-mode monitoring data, and it is difficult to achieve full integration and synchronous processing of multiple-mode monitoring data. But it is difficult to achieve accurate diagnosis of complex mechanical equipment faults based only on vibration data. Therefore, the existing intelligent diagnosis methods are not very accurate for mechanical fault diagnosis. Summary of the invention

[0004] This application aims to propose a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data, which can improve the accuracy of mechanical fault diagnosis.

[0005] In a first aspect, an embodiment of the present application provides a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data, the method comprising:

[0006] Collect multi-modal target monitoring data during the operation of target mechanical equipment;

[0007] Selecting any two modal target monitoring data from the multimodal target monitoring data;

[0008] Determine a corresponding target mechanical fault diagnosis model according to the two modal target monitoring data, wherein the target mechanical fault diagnosis model is obtained by training a sample monitoring data set of the same modality as the two modal target monitoring data, the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment, and the target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults;

[0009] The two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0010] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0011] The method collects multimodal target monitoring data during the operation of target mechanical equipment; selects two modal target monitoring data from the multimodal target monitoring data at random; determines the corresponding target mechanical fault diagnosis model according to the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained by a sample monitoring data set with the same modality as the two modal target monitoring data, the sample monitoring data set is the multimodal monitoring data collected during the operation of multiple mechanical equipment, and the target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults; the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result. In this way, the two modal target monitoring data are bidirectionally symmetric interacted by the symmetric fusion module, and the deep extraction of feature information of multiple modal data can be realized, and then the multiple modal data are deeply fused, so that the deeply fused data has a stronger representation ability for the fault state of the mechanical equipment, thereby improving the accuracy of mechanical fault diagnosis.

[0012] In some implementations, the target mechanical fault diagnosis model is trained by the following steps:

[0013] Acquire a sample monitoring data set of the same modality as the two modal target monitoring data;

[0014] Normalizing the data in the sample monitoring data set to obtain a normalized data set;

[0015] The normalized data set is selected and segmented, and the segmented data is labeled with fault labels to obtain a processed data set;

[0016] Dividing the processed data set into a sample training set and a sample test set;

[0017] Using the sample training set to train a mechanical fault diagnosis model to obtain a trained mechanical fault diagnosis model;

[0018] The trained mechanical fault diagnosis model is tested using the sample test set to obtain a target mechanical fault diagnosis model.

[0019] In some embodiments, the target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes a plurality of first convolutional layers and the second branch includes a plurality of second convolutional layers, the number of the symmetric fusion modules, the number of the first convolutional layers, and the number of the second convolutional layers are equal, and the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result, including:

[0020] Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain the convolution result of each first convolution layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain the convolution result of each second convolution layer;

[0021] The symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch;

[0022] Using the interaction result of the j-th first convolutional layer as the input of the j+1-th first convolutional layer, and using the interaction result of the j-th second convolutional layer as the input of the j+1-th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, and obtain the depth interaction result of the first branch and the depth interaction result of the second branch;

[0023] Concatenating and fusing the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a depth fusion feature;

[0024] The deep fusion features are input into the fault classifier to obtain a mechanical fault diagnosis result.

[0025] In some embodiments, the symmetric fusion module includes a first cross attention mechanism and a second cross attention mechanism, and the symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the jth first convolution layer in the first branch and the convolution result of the jth second convolution layer in the second branch to obtain the interaction result of the jth first convolution layer in the first branch and the interaction result of the jth second convolution layer in the second branch, including:

[0026] For the convolution result of the jth first convolutional layer in the first branch, the first cross attention mechanism is used to interact the convolution result of the jth second convolutional layer in the second branch with the convolution result of the jth first convolutional layer in the first branch to obtain the interaction result of the jth first convolutional layer in the first branch;

[0027] For the convolution result of the j-th second convolutional layer in the second branch, the second cross-attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch.

[0028] In some implementations, the using the first cross attention mechanism to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch includes:

[0029]

[0030] in, represents the interaction result of the j-th first convolutional layer in the first branch corresponding to the first modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the jth second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1, i，j The transformation matrix, express The transformation matrix, Indicates F1 i，j The weight matrix of Indicates F1 i，j The total number of dimensions.

[0031] In some implementations, the using the second cross attention mechanism to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch includes:

[0032]

[0033] in, represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the jth second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, express The weight matrix of express The total number of dimensions.

[0034] In some implementations, the step of concatenating and fusing the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a depth fusion feature includes:

[0035]

[0036] Among them, F i represents the deep fusion features corresponding to the two modal target monitoring data i, concat represents the data splicing operation, axis=-1 means that the splicing process starts from the last feature dimension, represents the deep interaction result of the first branch, Represents the depth interaction result of the second branch.

[0037] In a second aspect, the embodiment of the present application further provides a mechanical fault diagnosis system based on deep fusion of multimodal monitoring data, the system comprising:

[0038] A monitoring data acquisition unit, used to collect multi-modal target monitoring data during the operation of target mechanical equipment;

[0039] A target data selection unit, used for selecting any two modal target monitoring data from the multi-modal target monitoring data;

[0040] A diagnostic model determination unit, used for determining a corresponding target mechanical fault diagnostic model according to the two modal target monitoring data, wherein the target mechanical fault diagnostic model is obtained by training a sample monitoring data set of the same modality as the two modal target monitoring data, the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment, the target mechanical fault diagnostic model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used for bidirectional symmetric interaction of the two modal target monitoring data, and the fault classifier is used for classifying mechanical faults;

[0041] The mechanical fault diagnosis unit is used to input the two modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0042] In the third aspect, an embodiment of the present application also provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described above.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described above.

[0044] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0046] Figure 1 It is a flow chart of an embodiment of a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data provided by the present application;

[0047] Figure 2 It is a structural schematic diagram of a mechanical fault diagnosis model in the best embodiment of the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data provided by the present application;

[0048] Figure 3It is a structural schematic diagram of an embodiment of a mechanical fault diagnosis system based on deep fusion of multimodal monitoring data provided by the present application;

[0049] Figure 4 It is a schematic diagram of the structure of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and should not be construed as limitations on the present application. In the following description, specific details such as specific system structures, technologies, etc. are proposed for illustration rather than for limitation, so as to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that embodiments of the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the embodiments of the present application.

[0051] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0052] It should also be understood that the references to "one embodiment" or "some embodiments" described in the specification of the embodiments of the present application mean that one or more embodiments of the embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0053] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0054] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0055] Mechanical equipment is widely used in modern industry. It works under a series of complex and harsh conditions such as heavy load, high speed and high temperature for a long time. Its key components (such as gears and bearings, etc.) will inevitably fail. Once a key component fails, it will bring huge operating risks to the mechanical equipment and even cause major safety accidents. Therefore, it is of great value to carry out fault diagnosis of key components of equipment.

[0056] Considering that vibration signals are sensitive to mechanical faults, mechanical fault diagnosis methods based on vibration signal analysis have always been the mainstream research direction in this field. With the development of artificial intelligence technology, especially various neural network models, intelligent fault diagnosis methods based on neural networks have been widely and deeply studied. Existing intelligent diagnosis methods have been able to achieve efficient processing of vibration data and fault feature extraction. However, the physical structure of large-scale mechanical equipment is complex, and the types of component failures are diverse. Different fault locations or types will cause abnormal vibration signals of equipment, making it difficult for vibration monitoring data to fully characterize the complex fault types of equipment. However, existing intelligent diagnosis methods are better at processing single-mode monitoring data, and it is difficult to achieve full integration and synchronous processing of multiple-mode monitoring data. But it is difficult to achieve accurate diagnosis of complex mechanical equipment faults based only on vibration data. Therefore, the existing intelligent diagnosis methods are not very accurate for mechanical fault diagnosis.

[0057] In order to solve the problem that the above-mentioned existing intelligent diagnosis methods are not very accurate in diagnosing mechanical faults, the present application proposes a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data.

[0058] Reference Figure 1 The present application provides a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data, the method comprising the following steps:

[0059] Step S100, collecting multimodal target monitoring data during the operation of target mechanical equipment;

[0060] Step S200, arbitrarily selecting two modal target monitoring data from the multi-modal target monitoring data;

[0061] Step S300, determining a corresponding target mechanical fault diagnosis model according to the two modal target monitoring data, wherein the target mechanical fault diagnosis model is obtained by training a sample monitoring data set having the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected during the operation of various mechanical equipment, the target mechanical fault diagnosis model comprising a symmetric fusion module and a fault classifier, the symmetric fusion module being used for bidirectional symmetric interaction of the two modal target monitoring data, and the fault classifier being used for classifying mechanical faults;

[0062] Step S400: input two modal target monitoring data into a target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0063] In this embodiment, by collecting multimodal target monitoring data during the operation of target mechanical equipment; arbitrarily selecting two modal target monitoring data from the multimodal target monitoring data; determining the corresponding target mechanical fault diagnosis model according to the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained by a sample monitoring data set with the same modality as the two modal target monitoring data, the sample monitoring data set is the multimodal monitoring data collected during the operation of multiple mechanical equipment, the target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults; the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result. In this way, by using the symmetric fusion module to make the two modal target monitoring data perform bidirectional symmetric interaction, the deep extraction of feature information of multiple modal data can be realized, and then the multiple modal data are deeply fused, so that the deeply fused data has a stronger characterization ability for the fault state of the mechanical equipment, thereby improving the accuracy of mechanical fault diagnosis.

[0064] The above-mentioned multimodal target monitoring data collected during the operation of the target mechanical equipment may be the collection of operating parameters such as vibration, sound, current, etc. during the operation of the target mechanical equipment as multimodal target monitoring data, for example, vibration is represented as one mode of monitoring data, and sound is also represented as one mode of monitoring data.

[0065] The target mechanical equipment mentioned above may be the mechanical equipment that needs to be monitored at the current moment.

[0066] The above-mentioned determination of the corresponding target mechanical fault diagnosis model based on the two modal target monitoring data may be a target mechanical fault diagnosis model obtained by training with monitoring data of the same modality as the two modal target monitoring data, selected based on the two modal target monitoring data.

[0067] The symmetric fusion module may be a module constructed using a cross attention mechanism for bidirectional symmetric interaction of two modal target monitoring data, or may be a module constructed using a technology known to those skilled in the art for bidirectional symmetric interaction of two modal target monitoring data.

[0068] The above cross attention mechanism can be to use one input as a query, and then calculate the attention weight associated with it based on another input. This mechanism allows the model to dynamically pay attention to different inputs and decide which parts are most important. The main function of cross attention is to capture the dependency between two inputs.

[0069] The above-mentioned fault classifier may be a fault classifier constructed using a Softmax layer, and is used to classify mechanical faults, including classification results of types such as mechanical faults and mechanical no faults.

[0070] The above-mentioned two-way symmetrical interaction may be the interaction between two modal target monitoring data.

[0071] In some implementations, the target mechanical fault diagnosis model is trained by the following steps:

[0072] Acquire a sample monitoring data set having the same modality as the two modal target monitoring data;

[0073] Normalizing the data in the sample monitoring data set to obtain a normalized data set;

[0074] The normalized data set is selected and segmented, and the segmented data is labeled with fault labels to obtain a processed data set;

[0075] Divide the processed data set into a sample training set and a sample test set;

[0076] The sample training set is used to train the mechanical fault diagnosis model to obtain a trained mechanical fault diagnosis model;

[0077] The trained mechanical fault diagnosis model is tested using a sample test set to obtain the target mechanical fault diagnosis model.

[0078] In this embodiment, a processed data set is obtained by normalizing the data in the sample monitoring data set, selecting and segmenting the normalized data set, and annotating the segmented data with fault labels; the processed data set is divided into a sample training set and a sample test set; with this sample training set and sample test set, a relatively accurate target mechanical fault diagnosis model can be obtained, thereby laying a good model foundation for subsequent mechanical fault diagnosis.

[0079] In some embodiments, the target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes a plurality of first convolutional layers and the second branch includes a plurality of second convolutional layers, the number of symmetric fusion modules, the number of first convolutional layers, and the number of second convolutional layers are equal, and the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result, including:

[0080] Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain the convolution result of each first convolution layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain the convolution result of each second convolution layer;

[0081] The convolution result of the jth first convolution layer in the first branch and the convolution result of the jth second convolution layer in the second branch are bidirectionally and symmetrically interacted by using a symmetric fusion module to obtain the interaction result of the jth first convolution layer in the first branch and the interaction result of the jth second convolution layer in the second branch;

[0082] The interaction result of the jth first convolutional layer is used as the input of the j+1th first convolutional layer, and the interaction result of the jth second convolutional layer is used as the input of the j+1th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, and the deep interaction results of the first branch and the deep interaction results of the second branch are obtained;

[0083] Concatenate and fuse the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a deep fusion feature;

[0084] The deep fusion features are input into the fault classifier to obtain the mechanical fault diagnosis results.

[0085] In this embodiment, a symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the jth first convolutional layer in the first branch and the convolution result of the jth second convolutional layer in the second branch, so as to obtain the interaction result of the jth first convolutional layer in the first branch and the interaction result of the jth second convolutional layer in the second branch; the interaction result of the jth first convolutional layer is used as the input of the j+1th first convolutional layer, and the interaction result of the jth second convolutional layer is used as the input of the j+1th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetric interaction, so as to obtain the deep interaction result of the first branch and the deep interaction result of the second branch; the deep interaction result of the first branch and the deep interaction result of the second branch are spliced ​​and fused to obtain deep fusion features; the deep fusion features are input into the fault classifier to obtain the mechanical fault diagnosis result. In this way, by symmetrically interacting each convolutional layer in the two branches, we can achieve full-process symmetrical interaction of different modal feature information in the feature extraction process, give full play to the collaborative characterization of different modal monitoring data on the fault status of mechanical equipment, and then input the deep fusion features into the fault classifier to improve the accuracy of mechanical fault diagnosis.

[0086] In some embodiments, the symmetric fusion module includes a first cross attention mechanism and a second cross attention mechanism, and the symmetric fusion module is used to perform a bidirectional symmetric interaction between the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch, including:

[0087] For the convolution result of the jth first convolutional layer in the first branch, the first cross attention mechanism is used to interact the convolution result of the jth second convolutional layer in the second branch with the convolution result of the jth first convolutional layer in the first branch to obtain the interaction result of the jth first convolutional layer in the first branch;

[0088] For the convolution result of the j-th second convolutional layer in the second branch, the second cross attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch.

[0089] In this embodiment, the convolution result of the j-th second convolutional layer in the second branch is interacted with the convolution result of the j-th first convolutional layer in the first branch through the first cross-attention mechanism, and the convolution result of the j-th first convolutional layer in the first branch is interacted with the convolution result of the j-th second convolutional layer in the second branch through the second cross-attention mechanism. This can achieve full-process symmetrical interaction of different modal feature information, laying a good data foundation for subsequent mechanical fault diagnosis.

[0090] In some embodiments, a first cross attention mechanism is used to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch, including:

[0091]

[0092] in, represents the interaction result of the j-th first convolutional layer in the first branch corresponding to the first modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the jth second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, Indicates F1 i，j The weight matrix of express The total number of dimensions.

[0093] In some embodiments, a second cross attention mechanism is used to interact the convolution result of the jth first convolutional layer in the first branch with the convolution result of the jth second convolutional layer in the second branch to obtain the interaction result of the jth second convolutional layer in the second branch, including:

[0094]

[0095] in, represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the jth second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, express The weight matrix of express The total number of dimensions.

[0096] In some implementations, the depth interaction result of the first branch and the depth interaction result of the second branch are concatenated and fused to obtain a depth fusion feature, including:

[0097]

[0098] Among them, F i represents the deep fusion features corresponding to the two modal target monitoring data i, concat represents the data splicing operation, axis=-1 means that the splicing process starts from the last feature dimension, represents the deep interaction result of the first branch, Represents the deep interaction result of the second branch.

[0099] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0100] Mechanical equipment is widely used in modern industry. It works under a series of complex and harsh conditions such as heavy load, high speed and high temperature for a long time. Its key components (gears and bearings, etc.) will inevitably fail. Once a key component fails, it will bring huge operating risks to the mechanical equipment and even cause major safety accidents. Therefore, it is of great value to carry out fault diagnosis of key components of equipment.

[0101] Considering that vibration signals are sensitive to mechanical faults, mechanical fault diagnosis methods based on vibration signal analysis have always been the mainstream research direction in this field. With the development of artificial intelligence technology, especially various neural network models, intelligent fault diagnosis methods based on neural networks have been widely and deeply studied. Existing intelligent diagnosis methods have been able to achieve efficient processing of vibration data and fault feature extraction. However, the physical structure of large-scale mechanical equipment is complex, and the types of component failures are diverse. Different fault locations or types will cause abnormal vibration signals of mechanical equipment, making it difficult for vibration monitoring data to fully characterize the complex fault types of mechanical equipment. Therefore, it is difficult to achieve accurate diagnosis of complex mechanical equipment faults based only on vibration data. In addition to the monitoring data of the vibration mode, mechanical equipment usually monitors parameters of multiple modes such as sound and current during operation. When mechanical equipment fails, operating parameters such as vibration, sound and current will change. However, existing intelligent diagnosis methods are better at processing monitoring data of a single mode, and it is difficult to achieve full fusion and synchronous processing of monitoring data of multiple modes. Therefore, studying the fusion processing method of monitoring data of multiple modes is of great value to achieve accurate diagnosis of faults of complex mechanical equipment.

[0102] In order to solve the above-mentioned problem of difficulty in accurately diagnosing faults of complex mechanical equipment, this embodiment proposes a mechanical fault intelligent diagnosis method driven by deep fusion of multimodal monitoring data, including the following steps:

[0103] Step 1: Use various types of fast-changing monitoring signals (including vibration signals, sound signals, and current signals, etc.) collected during the operation of mechanical equipment as multimodal mechanical monitoring data (i.e., sample monitoring data). Various types of fast-changing monitoring signals are fast-changing signals with non-stationary characteristics, and collect multimodal target monitoring data during the operation of target mechanical equipment. Multimodal target monitoring data is the fast-changing monitoring signal collected when the mechanical equipment to be monitored is in motion. First, perform data normalization preprocessing on the multimodal mechanical monitoring data set, and its calculation formula can be expressed as:

[0104]

[0105] Among them, X I is the mechanical monitoring data or target monitoring data of the Ith mode, where I=1,2; |X I |To obtain monitoring data X I The absolute value of ; max(·) is the maximum value of the data; is the normalized monitoring data.

[0106] Then, the normalized multimodal mechanical monitoring data is selected and segmented. First, the starting time point t0 of data selection needs to be determined, and then the multimodal mechanical monitoring data point corresponding to this time point is determined. Data selection and segmentation are performed starting from this data point. Any set of multimodal machinery monitoring data samples obtained by segmentation can be expressed as:

[0107]

[0108] in, and Represents the i-th sample of the first mode and the second mode respectively, and N is the dimension of a single sample. A fault label is labeled for each sample obtained by segmentation, thereby completing the data preprocessing process. The multimodal machinery monitoring dataset after segmentation and labeling is divided into a training set and a test set.

[0109] Step 2: Construct a multimodal data feature deep fusion model (i.e., mechanical fault diagnosis model) based on full-process interactive symmetric fusion, such as Figure 2 As shown in the figure, the mechanical fault diagnosis model mainly includes two network branches, each of which is composed of an independent deep convolutional neural network composed of multiple convolutional layers. The input of each network branch is a data sample of any modality, and the output is the data feature corresponding to the modality data sample. Once the data modality processed by each network branch is determined, it will not change. For any sample , the convolution result obtained after the jth convolution layer operation (i.e., feature ) can be expressed as:

[0110]

[0111] Among them, Conv j (·) represents the convolution operation process of the j-th convolutional layer.

[0112] A symmetric fusion module based on twice symmetric cross attention mechanism is used to realize the symmetric interaction of data features between different modalities. For the data feature F1 under modality I=1, i，j , the weighted value after a cross-attention operation (For multimodal target monitoring data, It also represents the interaction result of the jth first convolutional layer in the first branch corresponding to the first modal target monitoring data i) can be expressed as:

[0113]

[0114] Among them, Softmax(·) represents the Softmax operation function, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, Indicates F1i，j The weight matrix of Indicates F1 i，j The total number of dimensions.

[0115] Symmetrically, for the data characteristics under mode I = 2 , the value after another cross-attention operation is weighted (For multimodal target monitoring data, It also represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i) can be expressed as:

[0116]

[0117] in, express The weight matrix of express The total number of dimensions.

[0118] The number of the above-mentioned symmetric fusion modules is exactly the same as the number of convolutional layers in the network branches in the constructed mechanical fault diagnosis model, thereby realizing the full-process interaction of multimodal data features.

[0119] Select any two monitoring data samples of different modalities from the training set established in step 1, use the constructed model to perform deep feature extraction and fusion on the data samples, and obtain the deep interaction features of the multimodal monitoring data of the mechanical equipment (i.e., the deep interaction results of the first branch and the deep interaction results of the second branch).

[0120] Step 3: Splice the acquired multimodal data deep interaction features. The feature splicing process can be expressed as:

[0121]

[0122] Among them, F i For data samples and The multimodal data fusion features (i.e. deep fusion features) are finally extracted. and The samples are and At the output of the last convolutional layer, concat(·) indicates the data concatenation operation, and axis=-1 indicates that the concatenation process starts from the last feature dimension.

[0123] The deep fusion features after splicing are input into the Softmax fault classifier to realize the classification of mechanical faults, and the training is completed by updating the model parameters.

[0124] Step 4: Select two monitoring data with the same modality as in the training process from the test set, and use the selected test set data to test the trained model. The model after the test (i.e., the target mechanical fault diagnosis model) can be used for fault classification of multi-modal target monitoring data, thereby realizing intelligent diagnosis of mechanical equipment faults.

[0125] In order to better illustrate the technical effect of the technical solution of this embodiment, this embodiment conducted the following experiments:

[0126] This embodiment uses a specific implementation scenario to experimentally verify the technical solution of this embodiment. The data set used in this experiment is the rolling bearing data set of the University of Ottawa, which is collected on a vibration and acoustic fault test bench. The bearing under test is located at the motor drive end of the test bench, and its model is NSK6203ZZ. The sensors used include a vibration accelerometer and a microphone, which are used to collect monitoring signals of two different modes of vibration and sound of the bearing under test. The model of the accelerometer is 623C01, and the sensitivity is 100mV / g. The model of the microphone is 130F20, and the sensitivity is 45mV / Pa. A data acquisition instrument is used for signal acquisition, and the sampling frequency is 42kHz.

[0127] In the experiment of this embodiment, damages of different severity were processed in the inner ring, outer ring, ball and cage of the bearing in turn to simulate different bearing faults, and a total of seven bearing vibration and sound data in different health states were obtained, namely 1) healthy state, 2) inner ring development fault, 3) inner ring fault, 4) outer ring fault, 5) ball fault, 6) cage development fault, and 7) cage fault. The monitoring data in each health state contains 800 samples, including 400 vibration data samples and 400 sound data samples. Each sample consists of 2048 consecutive data points.

[0128] In order to verify the effectiveness and superiority of the technical solution of this embodiment, three related methods were constructed as comparison methods according to the technical solution of this embodiment, namely:

[0129] 1) Deep convolutional neural network for vibration data (DCNN-V): The network structure is completely consistent with the network branch structure of the technical solution of this embodiment, and only processes single-mode vibration data;

[0130] 2) Deep convolutional neural network for sound data (DCNN-A): The network structure is completely consistent with the network branch structure of the technical solution of this embodiment, and only processes single-modal sound data;

[0131] 3) Deep convolutional neural network for simple fusion of vibration and sound (DCNN-SF): The network structure is exactly the same as that of the technical solution of this embodiment, but no symmetric fusion module is set, and only simple fusion of different modal data can be achieved;

[0132] From the constructed data set, 1, 5, 10, 15 and 20 samples were randomly selected as training samples, and the remaining samples were used as test samples to test and compare the above three comparison methods and the proposed method. The evaluation standard used for the comparison is the diagnostic accuracy. Each group of experiments was repeated 10 times, and the average results were taken for analysis. The experimental results can be seen in Table 1 below:

[0133] Table 1 shows the experimental results of the diagnostic accuracy of various methods.

[0134]

[0135] The above results show that compared with the diagnostic model that only uses single-modal monitoring data of vibration or sound, using multiple modal monitoring data to diagnose bearing faults can achieve better results, especially when there is only one training sample. In addition, compared with DCNN-SF, the technical solution of this embodiment shows a higher fault recognition accuracy under the same conditions, which shows that the multi-modal data feature full-process interaction solution based on the symmetric fusion module proposed in this embodiment has a significant gain effect on improving the fault recognition accuracy.

[0136] Compared with the prior art, this embodiment has the following advantages:

[0137] 1) The multimodal data feature deep fusion model based on full-process interactive symmetric fusion constructed in this embodiment can realize the deep extraction of feature information of multiple modal data, and at the same time realize the full-process symmetric interaction of different modal feature information in the feature extraction process. The experimental results of this embodiment also prove that the acquired multimodal data deep fusion features have a stronger characterization ability for the fault status of mechanical equipment.

[0138] 2) Based on the above-mentioned multimodal data feature deep fusion model, this embodiment proposes a mechanical fault intelligent diagnosis method driven by the deep fusion of multimodal monitoring data. It fully utilizes the collaborative characterization capability of multimodal monitoring data for the fault status of mechanical equipment, significantly improves the accuracy of intelligent diagnosis of equipment faults, and has the potential for practical engineering application.

[0139] Reference Figure 3 The embodiment of the present application also provides a mechanical fault diagnosis system based on deep fusion of multimodal monitoring data, the system comprising a monitoring data acquisition unit 100, a target data selection unit 200, a diagnosis model determination unit 300 and a mechanical fault diagnosis unit 400, wherein:

[0140] The monitoring data collection unit 100 is used to collect multi-modal target monitoring data during the operation of the target mechanical equipment;

[0141] The target data selection unit 200 is used to select any two modal target monitoring data from the multi-modal target monitoring data;

[0142] The diagnostic model determination unit 300 is used to determine the corresponding target mechanical fault diagnosis model according to the two modal target monitoring data, wherein the target mechanical fault diagnosis model is obtained by training a sample monitoring data set of the same modality as the two modal target monitoring data, the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment, and the target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults;

[0143] The mechanical fault diagnosis unit 400 is used to input two modal target monitoring data into a target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0144] It should be noted that since the mechanical fault diagnosis system based on deep fusion of multimodal monitoring data in this embodiment and the above-mentioned mechanical fault diagnosis method based on deep fusion of multimodal monitoring data are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0145] Reference Figure 4 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0146] at least one memory;

[0147] at least one processor;

[0148] at least one program;

[0149] The program is stored in the memory, and the processor executes at least one program to implement the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described above in the present disclosure.

[0150] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0151] The electronic device according to the embodiment of the present application is described in detail below.

[0152] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0153] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data in the embodiment of the present disclosure.

[0154] Input / output interface 1800, used to implement information input and output;

[0155] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0156] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0157] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0158] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned mechanical fault diagnosis method based on deep fusion of multimodal monitoring data.

[0159] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0161] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0162] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0164] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0165] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0166] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

[0170] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A mechanical fault diagnosis method based on deep fusion of multimodal monitoring data, characterized in that: The method comprises: Collect multi-modal target monitoring data during the operation of target mechanical equipment; Selecting any two modal target monitoring data from the multimodal target monitoring data; Determine a corresponding target mechanical fault diagnosis model according to the two modal target monitoring data, wherein the target mechanical fault diagnosis model is obtained by training a sample monitoring data set of the same modality as the two modal target monitoring data, the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment, and the target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults; The two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

2. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 1 is characterized in that: The target mechanical fault diagnosis model is trained by the following steps: Acquire a sample monitoring data set of the same modality as the two modal target monitoring data; Normalizing the data in the sample monitoring data set to obtain a normalized data set; The normalized data set is selected and segmented, and the segmented data is labeled with fault labels to obtain a processed data set; Dividing the processed data set into a sample training set and a sample test set; Using the sample training set to train a mechanical fault diagnosis model to obtain a trained mechanical fault diagnosis model; The trained mechanical fault diagnosis model is tested using the sample test set to obtain a target mechanical fault diagnosis model.

3. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 1 is characterized in that: The target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes a plurality of first convolutional layers and the second branch includes a plurality of second convolutional layers, the number of the symmetric fusion modules, the number of the first convolutional layers and the number of the second convolutional layers are equal, and the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result, including: Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain the convolution result of each first convolution layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain the convolution result of each second convolution layer; The symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch; Using the interaction result of the j-th first convolutional layer as the input of the j+1-th first convolutional layer, and using the interaction result of the j-th second convolutional layer as the input of the j+1-th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, and obtain the depth interaction result of the first branch and the depth interaction result of the second branch; Concatenating and fusing the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a depth fusion feature; The deep fusion features are input into the fault classifier to obtain a mechanical fault diagnosis result.

4. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 3 is characterized in that: The symmetric fusion module includes a first cross attention mechanism and a second cross attention mechanism, and the symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the jth first convolution layer in the first branch and the convolution result of the jth second convolution layer in the second branch to obtain the interaction result of the jth first convolution layer in the first branch and the interaction result of the jth second convolution layer in the second branch, including: For the convolution result of the jth first convolutional layer in the first branch, the first cross attention mechanism is used to interact the convolution result of the jth second convolutional layer in the second branch with the convolution result of the jth first convolutional layer in the first branch to obtain the interaction result of the jth first convolutional layer in the first branch; For the convolution result of the j-th second convolutional layer in the second branch, the second cross-attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch.

5. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 4 is characterized in that: The using the first cross attention mechanism to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch, includes: in, represents the interaction result of the j-th first convolutional layer in the first branch corresponding to the first modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, F1i,j represents the convolution result of the j-th first convolutional layer in the first branch, and W 1i,j express The transformation matrix, express The transformation matrix, express The weight matrix of express The total number of dimensions.

6. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 4 is characterized in that: The using the second cross attention mechanism to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch, includes: in, represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, F1i,j represents the convolution result of the j-th first convolutional layer in the first branch, and W 1i,j express The transformation matrix, express The transformation matrix, express The weight matrix of express The total number of dimensions.

7. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 3 is characterized in that: The step of splicing and fusing the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a depth fusion feature includes: Among them, F i represents the deep fusion features corresponding to the two modal target monitoring data i, concat represents the data splicing operation, axis=-1 means that the splicing process starts from the last feature dimension, represents the deep interaction result of the first branch, Represents the deep interaction result of the second branch.

8. A mechanical fault diagnosis system based on deep fusion of multimodal monitoring data, characterized in that: The system comprises: A monitoring data acquisition unit, used to collect multi-modal target monitoring data during the operation of target mechanical equipment; A target data selection unit, used for selecting any two modal target monitoring data from the multi-modal target monitoring data; A diagnostic model determination unit, used for determining a corresponding target mechanical fault diagnostic model according to the two modal target monitoring data, wherein the target mechanical fault diagnostic model is obtained by training a sample monitoring data set of the same modality as the two modal target monitoring data, the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment, the target mechanical fault diagnostic model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used for bidirectional symmetric interaction of the two modal target monitoring data, and the fault classifier is used for classifying mechanical faults; The mechanical fault diagnosis unit is used to input the two modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described in any one of claims 1 to 7.

Images