Mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data

Abstract

The present application discloses a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data. The method randomly selects two modal target monitoring data from the multimodal target monitoring data; determines the corresponding target mechanical fault diagnosis model based on the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained by a sample monitoring data set with the same modality as the two modal target monitoring data, and the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment. The target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier. The symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults; the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result. The present application can improve the accuracy of mechanical fault diagnosis.

Description

Technical Field

[0001] The present application relates to the technical field of mechanical fault diagnosis, and in particular to a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data. Background Art

[0002] Mechanical equipment is widely used in modern industry. Operating under complex and harsh conditions such as heavy loads, high speeds, and high temperatures, critical components (such as gears and bearings) inevitably fail. Failure of these components can pose significant operational risks and even lead to serious safety accidents. Therefore, fault diagnosis of critical components is crucial.

[0003] Given the sensitivity of vibration signals to mechanical faults, mechanical fault diagnosis methods based on vibration signal analysis have long been a mainstream research direction in this field. With the advancement of artificial intelligence (AI), particularly various neural network models, intelligent fault diagnosis methods based on neural networks have been extensively and intensively studied. Existing intelligent diagnostic methods are capable of efficiently processing vibration data and extracting fault signatures. However, large-scale mechanical equipment has complex physical structures and diverse component failure types. Each fault location or type can cause abnormal vibration signals in the equipment, making it difficult to fully characterize the complex fault types of the equipment using vibration monitoring data alone. However, existing intelligent diagnostic methods are more adept at processing monitoring data from a single modality and struggle to fully integrate and synchronize monitoring data from multiple modalities. Accurately diagnosing complex mechanical equipment faults based solely on vibration data is difficult. Therefore, existing intelligent diagnostic methods are not very accurate for mechanical fault diagnosis. Summary of the Invention

[0004] This application aims to propose a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data, which can improve the accuracy of mechanical fault diagnosis.

[0005] In a first aspect, an embodiment of the present application provides a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data, the method comprising:

[0006] Collect multi-modal target monitoring data during the operation of target mechanical equipment;

[0007] arbitrarily selecting two modal target monitoring data from the multimodal target monitoring data;

[0008] Determining a corresponding target mechanical fault diagnosis model based on the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained using a sample monitoring data set having the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected during the operation of various mechanical equipment, the target mechanical fault diagnosis model including a symmetric fusion module and a fault classifier, the symmetric fusion module being used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier being used to classify mechanical faults;

[0009] The two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0010] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0011] This method collects multimodal target monitoring data from the operation of target mechanical equipment; arbitrarily selects two modal target monitoring data from the multimodal target monitoring data; and determines a corresponding target mechanical fault diagnosis model based on the two modal target monitoring data. The target mechanical fault diagnosis model is trained using a sample monitoring data set with the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected from multiple mechanical equipment during operation. The target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier. The symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults. The two modal target monitoring data are then input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result. In this way, by enabling the bidirectional symmetric interaction of the two modal target monitoring data through the symmetric fusion module, deep extraction of feature information from the multimodal data can be achieved. The multimodal data is then deeply fused, resulting in a more robust representation of the mechanical equipment fault state after the deep fusion, thereby improving the accuracy of mechanical fault diagnosis.

[0012] In some embodiments, the target mechanical fault diagnosis model is trained by the following steps:

[0013] Acquire a sample monitoring data set of the same modality as the two modal target monitoring data;

[0014] Normalizing the data in the sample monitoring data set to obtain a normalized data set;

[0015] Select and segment the normalized data set, and annotate the segmented data with fault labels to obtain the processed data set;

[0016] Dividing the processed data set into a sample training set and a sample test set;

[0017] Using the sample training set to train a mechanical fault diagnosis model to obtain a trained mechanical fault diagnosis model;

[0018] The trained mechanical fault diagnosis model is tested using the sample test set to obtain a target mechanical fault diagnosis model.

[0019] In some embodiments, the target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes a plurality of first convolutional layers and the second branch includes a plurality of second convolutional layers, the number of the symmetric fusion modules, the number of the first convolutional layers, and the number of the second convolutional layers are equal, and inputting the two modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result includes:

[0020] Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain a convolution result of each first convolutional layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain a convolution result of each second convolutional layer;

[0021] The symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, so as to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch;

[0022] The interaction result of the j-th first convolutional layer is used as the input of the j+1-th first convolutional layer, and the interaction result of the j-th second convolutional layer is used as the input of the j+1-th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, thereby obtaining the depth interaction result of the first branch and the depth interaction result of the second branch;

[0023] Concatenating and fusing the depth interaction results of the first branch and the depth interaction results of the second branch to obtain a depth fusion feature;

[0024] The deep fusion features are input into the fault classifier to obtain a mechanical fault diagnosis result.

[0025] In some embodiments, the symmetric fusion module includes a first cross attention mechanism and a second cross attention mechanism, and the symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch, including:

[0026] For the convolution result of the j-th first convolutional layer in the first branch, the first cross attention mechanism is used to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch;

[0027] For the convolution result of the j-th second convolutional layer in the second branch, the second cross-attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch.

[0028] In some embodiments, using the first cross attention mechanism to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch includes:

[0029]

[0030] in, represents the interaction result of the j-th first convolutional layer in the first branch corresponding to the first modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1, i，j The transformation matrix, express The transformation matrix, Indicates F1 i，j The weight matrix, Indicates F1 i，j The total number of dimensions.

[0031] In some embodiments, using the second cross attention mechanism to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch includes:

[0032]

[0033] in, represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, express The weight matrix, express The total number of dimensions.

[0034] In some embodiments, the step of concatenating and fusing the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a depth fusion feature includes:

[0035]

[0036] Among them, F i Represents the deep fusion features corresponding to the two modal target monitoring data i, concat represents the data splicing operation, axis=-1 means that the splicing process starts from the last feature dimension, represents the depth interaction result of the first branch, Indicates the depth interaction result of the second branch.

[0037] In a second aspect, an embodiment of the present application further provides a mechanical fault diagnosis system based on deep fusion of multimodal monitoring data, the system comprising:

[0038] A monitoring data acquisition unit, used to collect multimodal target monitoring data during the operation of target mechanical equipment;

[0039] A target data selection unit, configured to select any two modal target monitoring data from the multimodal target monitoring data;

[0040] a diagnostic model determination unit, configured to determine a corresponding target mechanical fault diagnostic model based on the two modal target monitoring data, wherein the target mechanical fault diagnostic model is trained using a sample monitoring data set having the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected during the operation of various mechanical equipment, the target mechanical fault diagnostic model comprising a symmetric fusion module and a fault classifier, the symmetric fusion module being configured to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier being configured to classify mechanical faults;

[0041] The mechanical fault diagnosis unit is used to input the two modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0042] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described above.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described above.

[0044] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0046] Figure 1 This is a flow chart of an embodiment of a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data provided by the present application;

[0047] Figure 2 This is a schematic diagram of the structure of a mechanical fault diagnosis model in the best embodiment of the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data provided by the present application;

[0048] Figure 3This is a schematic structural diagram of an embodiment of a mechanical fault diagnosis system based on deep fusion of multimodal monitoring data provided by the present application;

[0049] Figure 4 It is a structural diagram of an embodiment of the electronic device provided by this application. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the embodiments of the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the embodiments of the present application.

[0051] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0052] It should also be understood that references to "one embodiment" or "some embodiments" described in the description of the embodiments of the present application mean that one or more embodiments of the embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the phrases "in one embodiment," "in some embodiments," "in some other embodiments," "in some other embodiments," etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0053] In the description of this application, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0054] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technical personnel in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0055] Mechanical equipment is widely used in modern industry. Operating under complex and harsh conditions such as heavy loads, high speeds, and high temperatures, critical components (such as gears and bearings) inevitably fail. Failure of these components can pose significant operational risks and even lead to serious safety accidents. Therefore, fault diagnosis of critical components is crucial.

[0056] Given the sensitivity of vibration signals to mechanical faults, mechanical fault diagnosis methods based on vibration signal analysis have long been a mainstream research direction in this field. With the advancement of artificial intelligence (AI), particularly various neural network models, intelligent fault diagnosis methods based on neural networks have been extensively and intensively studied. Existing intelligent diagnostic methods are capable of efficiently processing vibration data and extracting fault signatures. However, large-scale mechanical equipment has complex physical structures and diverse component failure types. Each fault location or type can cause abnormal vibration signals in the equipment, making it difficult to fully characterize the complex fault types of the equipment using vibration monitoring data alone. However, existing intelligent diagnostic methods are more adept at processing monitoring data from a single modality and struggle to fully integrate and synchronize monitoring data from multiple modalities. Accurately diagnosing complex mechanical equipment faults based solely on vibration data is difficult. Therefore, existing intelligent diagnostic methods are not very accurate for mechanical fault diagnosis.

[0057] In order to solve the problem that the above-mentioned existing intelligent diagnosis methods are not very accurate in diagnosing mechanical faults, this application proposes a mechanical fault diagnosis method and system based on deep fusion of multimodal monitoring data.

[0058] Reference Figure 1 , an embodiment of the present application provides a mechanical fault diagnosis method based on deep fusion of multimodal monitoring data, the method comprising the following steps:

[0059] Step S100: collecting multimodal target monitoring data during the operation of target mechanical equipment;

[0060] Step S200: arbitrarily selecting two modal target monitoring data from the multimodal target monitoring data;

[0061] Step S300: Determine a corresponding target mechanical fault diagnosis model based on the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained using a sample monitoring data set having the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected during the operation of various mechanical equipment. The target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module being used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier being used to classify mechanical faults.

[0062] Step S400: inputting two modal target monitoring data into a target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0063] In this embodiment, multimodal target monitoring data is collected during the operation of target mechanical equipment; two modal target monitoring data are arbitrarily selected from the multimodal target monitoring data; a corresponding target mechanical fault diagnosis model is determined based on the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained by a sample monitoring data set with the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected during the operation of multiple mechanical equipment, the target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults; the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result. In this way, by enabling the two modal target monitoring data to perform bidirectional symmetric interaction through the symmetric fusion module, deep extraction of feature information of multiple modal data can be achieved, and then deep fusion of multiple modal data is performed, so that the deeply fused data has a stronger characterization capability for the fault state of the mechanical equipment, thereby improving the accuracy of mechanical fault diagnosis.

[0064] The above-mentioned multimodal target monitoring data collected during the operation of the target mechanical equipment can be the collection of operating parameters such as vibration, sound, current, etc. during the operation of the target mechanical equipment as multimodal target monitoring data. For example, vibration is represented as monitoring data of one mode, and sound is also represented as monitoring data of one mode.

[0065] The target mechanical equipment may be the mechanical equipment that needs to be monitored at the current moment.

[0066] The above-mentioned determination of the corresponding target mechanical fault diagnosis model based on the two modal target monitoring data can be a target mechanical fault diagnosis model obtained by training with monitoring data of the same modality as the two modal target monitoring data based on the two modal target monitoring data.

[0067] The symmetric fusion module can be a module constructed using a cross-attention mechanism for bidirectional symmetric interaction of two modal target monitoring data. Alternatively, it can be a module constructed using techniques well known to those skilled in the art for bidirectional symmetric interaction of two modal target monitoring data.

[0068] The crisscross attention mechanism described above uses one input as a query and then calculates the attention weight associated with it based on the other input. This mechanism allows the model to dynamically focus on different inputs and determine which parts are most important. The main function of crisscross attention is to capture the dependency between two inputs.

[0069] The above-mentioned fault classifier may be a fault classifier constructed using a Softmax layer, and is used to classify mechanical faults, including classification results of types such as mechanical faults and mechanical no faults.

[0070] The above-mentioned two-way symmetrical interaction can be the interaction between two modal target monitoring data.

[0071] In some embodiments, the target mechanical fault diagnosis model is trained by the following steps:

[0072] Acquire a sample monitoring data set having the same modality as the two modal target monitoring data;

[0073] Normalizing the data in the sample monitoring data set to obtain a normalized data set;

[0074] Select and segment the normalized data set, and annotate the segmented data with fault labels to obtain the processed data set;

[0075] Divide the processed data set into a sample training set and a sample test set;

[0076] The sample training set is used to train the mechanical fault diagnosis model to obtain a trained mechanical fault diagnosis model;

[0077] The trained mechanical fault diagnosis model is tested using a sample test set to obtain the target mechanical fault diagnosis model.

[0078] In this embodiment, a processed data set is obtained by normalizing the data in the sample monitoring data set, selecting and segmenting the normalized data set, and annotating the segmented data with fault labels; the processed data set is divided into a sample training set and a sample test set; with this sample training set and sample test set, a relatively accurate target mechanical fault diagnosis model can be obtained, thereby laying a good model foundation for subsequent mechanical fault diagnosis.

[0079] In some embodiments, the target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes multiple first convolutional layers and the second branch includes multiple second convolutional layers, the number of symmetric fusion modules, the number of first convolutional layers, and the number of second convolutional layers are equal, and two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result, including:

[0080] Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain the convolution result of each first convolution layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain the convolution result of each second convolution layer;

[0081] The symmetric fusion module is used to perform bidirectional symmetric interaction between the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, and the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch are obtained;

[0082] The interaction result of the jth first convolutional layer is used as the input of the j+1th first convolutional layer, and the interaction result of the jth second convolutional layer is used as the input of the j+1th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, and the depth interaction results of the first branch and the depth interaction results of the second branch are obtained;

[0083] The depth interaction results of the first branch and the depth interaction results of the second branch are spliced ​​and fused to obtain the depth fusion feature;

[0084] The deep fusion features are input into the fault classifier to obtain the mechanical fault diagnosis results.

[0085] In this embodiment, a symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch; the interaction result of the j-th first convolutional layer is used as the input of the j+1-th first convolutional layer, and the interaction result of the j-th second convolutional layer is used as the input of the j+1-th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetric interaction, to obtain the deep interaction result of the first branch and the deep interaction result of the second branch; the deep interaction results of the first branch and the deep interaction results of the second branch are spliced ​​and fused to obtain deep fusion features; the deep fusion features are input into the fault classifier to obtain the mechanical fault diagnosis result. In this way, by symmetrically interacting each convolutional layer in the two branches, the full-process symmetrical interaction of different modal feature information in the feature extraction process is realized, and the collaborative characterization of the fault status of mechanical equipment by different modal monitoring data is fully utilized. Then, the deep fusion features are input into the fault classifier, which can improve the accuracy of mechanical fault diagnosis.

[0086] In some embodiments, the symmetric fusion module includes a first cross attention mechanism and a second cross attention mechanism, and the symmetric fusion module is used to perform a bidirectional symmetric interaction between the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch, including:

[0087] For the convolution result of the jth first convolutional layer in the first branch, the first cross attention mechanism is used to interact the convolution result of the jth second convolutional layer in the second branch with the convolution result of the jth first convolutional layer in the first branch to obtain the interaction result of the jth first convolutional layer in the first branch;

[0088] For the convolution result of the j-th second convolutional layer in the second branch, the second cross attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch.

[0089] In this embodiment, the convolution result of the j-th second convolutional layer in the second branch is interacted with the convolution result of the j-th first convolutional layer in the first branch through the first cross-attention mechanism, and the convolution result of the j-th first convolutional layer in the first branch is interacted with the convolution result of the j-th second convolutional layer in the second branch through the second cross-attention mechanism. This can achieve full-process symmetrical interaction of different modal feature information, laying a good data foundation for subsequent mechanical fault diagnosis.

[0090] In some embodiments, a first cross attention mechanism is used to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch, including:

[0091]

[0092] in, represents the interaction result of the j-th first convolutional layer in the first branch corresponding to the first modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, Indicates F1 i，j The weight matrix, express The total number of dimensions.

[0093] In some embodiments, a second cross-attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch, including:

[0094]

[0095] in, represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i, Softmax(·) represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, F1 i，j represents the convolution result of the jth first convolutional layer in the first branch, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, express The weight matrix, express The total number of dimensions.

[0096] In some embodiments, the depth interaction result of the first branch and the depth interaction result of the second branch are spliced ​​and fused to obtain a depth fusion feature, including:

[0097]

[0098] Among them, F i Represents the deep fusion features corresponding to the two modal target monitoring data i, concat represents the data splicing operation, axis=-1 means that the splicing process starts from the last feature dimension, represents the deep interaction result of the first branch, Represents the deep interaction result of the second branch.

[0099] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0100] Mechanical equipment is widely used in modern industry. Operating under complex and harsh conditions such as heavy loads, high speeds, and high temperatures, critical components (such as gears and bearings) inevitably fail. Failure of these components can pose significant operational risks and even lead to serious safety accidents. Therefore, fault diagnosis of critical components is crucial.

[0101] Given the sensitivity of vibration signals to mechanical faults, mechanical fault diagnosis methods based on vibration signal analysis have long been a mainstream research direction in this field. With the development of artificial intelligence (AI) technology, particularly various neural network models, intelligent fault diagnosis methods based on neural networks have been extensively and intensively studied. Existing intelligent diagnostic methods have enabled efficient processing of vibration data and fault feature extraction. However, large-scale mechanical equipment has complex physical structures and diverse component failure types. Different fault locations or types can cause abnormal vibration signals in the mechanical equipment, making it difficult for vibration monitoring data to fully characterize the complex fault types of the mechanical equipment. Therefore, accurate diagnosis of complex mechanical equipment faults based solely on vibration data is difficult. In addition to monitoring data from the vibration modality, mechanical equipment typically monitors parameters in multiple modalities, such as sound and current, during operation. When mechanical equipment fails, operating parameters such as vibration, sound, and current will all change. However, existing intelligent diagnostic methods are more adept at processing monitoring data from a single modality and struggle to fully integrate and synchronously process monitoring data from multiple modalities. Therefore, research on methods for integrating and processing multi-modal monitoring data is crucial for accurate fault diagnosis of complex mechanical equipment.

[0102] To address the aforementioned difficulty in accurately diagnosing faults of complex mechanical equipment, this embodiment proposes an intelligent mechanical fault diagnosis method driven by deep fusion of multimodal monitoring data, comprising the following steps:

[0103] Step 1: Use various types of fast-changing monitoring signals (including vibration signals, sound signals, and current signals, etc.) collected during the operation of mechanical equipment as multimodal mechanical monitoring data (i.e., sample monitoring data). Various types of fast-changing monitoring signals are fast-changing signals with non-stationary characteristics, and collect multimodal target monitoring data during the operation of the target mechanical equipment. Multimodal target monitoring data is the fast-changing monitoring signal collected when the mechanical equipment to be monitored is in motion. First, perform data normalization preprocessing on the multimodal mechanical monitoring data set, and its calculation formula can be expressed as:

[0104]

[0105] Among them, X I is the mechanical monitoring data or target monitoring data of the Ith mode, where I=1,2; |X I To obtain monitoring data X I The absolute value of ; max(·) is the maximum value of the data; is the normalized monitoring data.

[0106] Then, the normalized multimodal mechanical monitoring data is selected and segmented. First, the starting time point t0 of data selection needs to be determined, and then the multimodal mechanical monitoring data point corresponding to this time point is determined. Data selection and segmentation are performed starting from this data point. Any set of multimodal machinery monitoring data samples obtained by segmentation can be expressed as:

[0107]

[0108] in, and represents the i-th sample of the first and second modes, respectively, and N is the dimension of a single sample. Each segmented sample is labeled with a fault label, completing the data preprocessing process. The segmented and labeled multimodal machinery monitoring dataset is divided into a training set and a test set.

[0109] Step 2: Construct a multimodal data feature deep fusion model based on full-process interactive symmetric fusion (i.e., mechanical fault diagnosis model), such as Figure 2 As shown in the figure, the mechanical fault diagnosis model mainly consists of two network branches, each of which is composed of an independent deep convolutional neural network composed of multiple convolutional layers. The input of each network branch is a data sample of any modality, and the output is the data feature corresponding to the modality data sample. Once the data modality processed by each network branch is determined, it will not change. For any sample , the convolution result obtained after the j-th convolution layer operation (i.e., feature ) can be expressed as:

[0110]

[0111] Among them, Conv j (·) represents the convolution operation process of the j-th convolutional layer.

[0112] A symmetric fusion module based on a two-step symmetric cross attention mechanism is used to achieve symmetric interaction of data features between different modalities. For the data feature F1 under modality I=1, i，j , the value after a weighted operation of the cross attention mechanism (For multimodal target monitoring data, It also represents the interaction result of the j-th first convolutional layer in the first branch corresponding to the first modal target monitoring data i) can be expressed as:

[0113]

[0114] Among them, Softmax(·) represents the Softmax operation function, W1 i，j Indicates F1 i，j The transformation matrix, express The transformation matrix, Indicates F1i，j The weight matrix, Indicates F1 i，j The total number of dimensions.

[0115] Symmetrically, for the data characteristics under mode I = 2 , the value after another cross-attention operation is weighted (For multimodal target monitoring data, It also represents the interaction result of the j-th second convolutional layer in the second branch corresponding to the second modal target monitoring data i) can be expressed as:

[0116]

[0117] in, express The weight matrix, express The total number of dimensions.

[0118] The number of the above-mentioned symmetric fusion modules is exactly the same as the number of convolutional layers in the network branches in the constructed mechanical fault diagnosis model, thereby realizing the full-process interaction of multimodal data features.

[0119] Select any two monitoring data samples of different modalities from the training set established in step 1, use the constructed model to perform deep feature extraction and fusion on the data samples, and obtain the deep interaction features of the multimodal monitoring data of the mechanical equipment (that is, the deep interaction results of the first branch and the deep interaction results of the second branch).

[0120] Step 3: Splice the acquired multimodal data deep interaction features. The feature splicing process can be expressed as:

[0121]

[0122] Among them, F i For data samples and The multimodal data fusion features (i.e. deep fusion features) are finally extracted. and The samples and At the output of the last convolutional layer, concat(·) indicates the data concatenation operation, and axis=-1 indicates that the concatenation process starts from the last feature dimension.

[0123] After splicing, the deep fusion features are input into the Softmax fault classifier to realize the classification of mechanical faults, and the training is completed by updating the model parameters.

[0124] Step 4: Select two monitoring data sets with the same modalities as those used during training from the test set and test the trained model using the selected test set data. The tested model (i.e., the target mechanical fault diagnosis model) can be used to classify faults in multimodal target monitoring data, thereby enabling intelligent diagnosis of mechanical equipment faults.

[0125] In order to better illustrate the technical effect of the technical solution of this embodiment, this embodiment conducted the following experiments:

[0126] This embodiment uses a specific implementation scenario to experimentally verify the technical solution of this embodiment. The data set used in this experiment is the rolling bearing data set of the University of Ottawa, which is collected on a vibration and acoustic fault test bench. The bearing under test is located at the motor drive end of the test bench, and its model is NSK6203ZZ. The sensors used include a vibration accelerometer and a microphone, which are used to collect monitoring signals of two different modes of vibration and sound of the bearing under test. The model of the accelerometer is 623C01, and its sensitivity is 100mV / g. The model of the microphone is 130F20, and its sensitivity is 45mV / Pa. A data acquisition instrument is used for signal acquisition, and the sampling frequency is 42kHz.

[0127] In this example, the experiments involved inflicting damage of varying severity on the inner ring, outer ring, ball bearing, and cage of the bearing to simulate various bearing failures. This yielded bearing vibration and acoustic data for seven different health states: 1) healthy, 2) inner ring developing failure, 3) inner ring failure, 4) outer ring failure, 5) ball bearing failure, 6) cage developing failure, and 7) cage failure. The monitoring data for each health state consisted of 800 samples, including 400 vibration data samples and 400 acoustic data samples. Each sample consisted of 2048 consecutive data points.

[0128] In order to verify the effectiveness and superiority of the technical solution of this embodiment, three related methods were constructed as comparative methods based on the technical solution of this embodiment, namely:

[0129] 1) Deep Convolutional Neural Network for Vibration Data (DCNN-V): The network structure is exactly the same as the network branch structure of the technical solution of this embodiment, and only processes single-modal vibration data;

[0130] 2) Deep Convolutional Neural Network for Sound Data (DCNN-A): The network structure is exactly the same as the network branch structure of the technical solution of this embodiment, and only processes single-modal sound data;

[0131] 3) Deep Convolutional Neural Network for Simple Vibration-Sound Fusion (DCNN-SF): The network structure is exactly the same as that of the technical solution of this embodiment, but without the symmetric fusion module, only simple fusion of different modal data can be achieved;

[0132] From the constructed dataset, 1, 5, 10, 15, and 20 samples were randomly selected as training samples, and the remaining samples were used as test samples. The three comparison methods and the proposed method were tested and compared. The evaluation criterion used for the comparison was diagnostic accuracy. Each group of experiments was repeated 10 times, and the average results were taken for analysis. The experimental results can be seen in Table 1 below:

[0133] Table 1 shows the experimental results of the diagnostic accuracy of various methods

[0134]

[0135] The above results demonstrate that, compared to diagnostic models that rely solely on single-modal monitoring data from vibration or sound, using multiple modal monitoring data simultaneously to diagnose bearing faults can achieve superior results, especially when only one training sample is available. Furthermore, compared to DCNN-SF, the technical solution of this embodiment demonstrates higher fault identification accuracy under the same conditions, demonstrating that the proposed full-process interactive solution for multimodal data features based on a symmetric fusion module significantly improves fault identification accuracy.

[0136] Compared with the prior art, this embodiment has the following advantages:

[0137] 1) The multimodal data feature deep fusion model constructed in this embodiment, based on full-process interactive symmetric fusion, enables in-depth extraction of feature information from multiple modal data, while also enabling full-process symmetric interaction of different modal feature information during the feature extraction process. Experimental results from this embodiment also demonstrate that the acquired multimodal data deep fusion features have a stronger ability to characterize mechanical equipment fault conditions.

[0138] 2) Based on the above-mentioned multimodal data feature deep fusion model, this embodiment proposes a mechanical fault intelligent diagnosis method driven by the deep fusion of multimodal monitoring data. It fully utilizes the collaborative characterization capability of multimodal monitoring data on the fault status of mechanical equipment, significantly improves the accuracy of intelligent diagnosis of equipment faults, and has the potential for practical engineering application.

[0139] Reference Figure 3 The embodiment of the present application further provides a mechanical fault diagnosis system based on deep fusion of multimodal monitoring data, the system comprising a monitoring data acquisition unit 100, a target data selection unit 200, a diagnostic model determination unit 300, and a mechanical fault diagnosis unit 400, wherein:

[0140] The monitoring data acquisition unit 100 is used to collect multimodal target monitoring data during the operation of the target mechanical equipment;

[0141] The target data selection unit 200 is used to select any two modal target monitoring data from the multimodal target monitoring data;

[0142] a diagnostic model determination unit 300 for determining a corresponding target mechanical fault diagnostic model based on the two modal target monitoring data, wherein the target mechanical fault diagnostic model is trained using a sample monitoring data set having the same modality as the two modal target monitoring data, the sample monitoring data set being multimodal monitoring data collected during the operation of various mechanical equipment, and the target mechanical fault diagnostic model includes a symmetric fusion module and a fault classifier, the symmetric fusion module being configured to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier being configured to classify mechanical faults;

[0143] The mechanical fault diagnosis unit 400 is used to input the two modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

[0144] It should be noted that since the mechanical fault diagnosis system based on deep fusion of multimodal monitoring data in this embodiment and the above-mentioned mechanical fault diagnosis method based on deep fusion of multimodal monitoring data are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0145] Reference Figure 4 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0146] at least one memory;

[0147] at least one processor;

[0148] at least one program;

[0149] The program is stored in the memory, and the processor executes at least one program to implement the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data implemented in the present disclosure.

[0150] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0151] The electronic device according to the embodiment of the present application is described in detail below.

[0152] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0153] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called by the processor 1600 to execute the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to the embodiment of the present disclosure.

[0154] Input / output interface 1800, used for information input and output;

[0155] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0156] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0157] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0158] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned mechanical fault diagnosis method based on deep fusion of multimodal monitoring data.

[0159] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0161] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0163] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0164] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0165] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

[0170] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A mechanical fault diagnosis method based on deep fusion of multimodal monitoring data, characterized in that: The method comprises: Collect multi-modal target monitoring data during the operation of target mechanical equipment; arbitrarily selecting two modal target monitoring data from the multimodal target monitoring data; A corresponding target mechanical fault diagnosis model is determined based on the two modal target monitoring data, wherein the target mechanical fault diagnosis model is trained by a sample monitoring data set with the same modality as the two modal target monitoring data, and the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment. The target mechanical fault diagnosis model includes a symmetric fusion module and a fault classifier, wherein the symmetric fusion module is used to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is used to classify mechanical faults. Specifically: The target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes a plurality of first convolutional layers and the second branch includes a plurality of second convolutional layers, the number of the symmetric fusion modules, the number of the first convolutional layers, and the number of the second convolutional layers are equal, and the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result, including: Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain a convolution result of each first convolutional layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain a convolution result of each second convolutional layer; The symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, so as to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch; The interaction result of the j-th first convolutional layer is used as the input of the j+1-th first convolutional layer, and the interaction result of the j-th second convolutional layer is used as the input of the j+1-th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, thereby obtaining the depth interaction result of the first branch and the depth interaction result of the second branch; Concatenating and fusing the depth interaction results of the first branch and the depth interaction results of the second branch to obtain a depth fusion feature; Inputting the deep fusion features into the fault classifier to obtain a mechanical fault diagnosis result; The two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

2. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 1 is characterized in that: The target mechanical fault diagnosis model is trained by the following steps: Acquire a sample monitoring data set of the same modality as the two modal target monitoring data; Normalizing the data in the sample monitoring data set to obtain a normalized data set; Select and segment the normalized data set, and annotate the segmented data with fault labels to obtain the processed data set; Dividing the processed data set into a sample training set and a sample test set; Using the sample training set to train a mechanical fault diagnosis model to obtain a trained mechanical fault diagnosis model; The trained mechanical fault diagnosis model is tested using the sample test set to obtain a target mechanical fault diagnosis model.

3. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 1 is characterized in that: The symmetric fusion module includes a first cross attention mechanism and a second cross attention mechanism, and the symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch, including: For the convolution result of the j-th first convolutional layer in the first branch, the first cross attention mechanism is used to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch; For the convolution result of the j-th second convolutional layer in the second branch, the second cross-attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch.

4. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 3 is characterized in that: The method of using the first cross attention mechanism to interact the convolution result of the j-th second convolutional layer in the second branch with the convolution result of the j-th first convolutional layer in the first branch to obtain the interaction result of the j-th first convolutional layer in the first branch includes: in, Represents the first modal target monitoring data The corresponding interaction result of the j-th first convolutional layer in the first branch, Represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, represents the convolution result of the j-th first convolutional layer in the first branch, express The transformation matrix, express The transformation matrix, express The weight matrix, express The total number of dimensions.

5. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 3 is characterized in that: The second cross attention mechanism is used to interact the convolution result of the j-th first convolutional layer in the first branch with the convolution result of the j-th second convolutional layer in the second branch to obtain the interaction result of the j-th second convolutional layer in the second branch, including: in, Represents the second modal target monitoring data The corresponding interaction result of the j-th second convolutional layer in the second branch, Represents the Softmax operation function, represents the convolution result of the j-th second convolutional layer in the second branch, represents the convolution result of the j-th first convolutional layer in the first branch, express The transformation matrix, express The transformation matrix, express The weight matrix, express The total number of dimensions.

6. The mechanical fault diagnosis method based on deep fusion of multimodal monitoring data according to claim 1 is characterized in that: The step of splicing and fusing the depth interaction result of the first branch and the depth interaction result of the second branch to obtain a depth fusion feature includes: in, Represents two modal target monitoring data The corresponding deep fusion features, Indicates data splicing operation, Indicates that the splicing process starts from the last feature dimension, represents the depth interaction result of the first branch, Indicates the depth interaction result of the second branch.

7. A mechanical fault diagnosis system based on deep fusion of multimodal monitoring data, characterized in that: The system comprises: A monitoring data acquisition unit, used to collect multimodal target monitoring data during the operation of target mechanical equipment; A target data selection unit, configured to select any two modal target monitoring data from the multimodal target monitoring data; A diagnostic model determination unit is configured to determine a corresponding target mechanical fault diagnostic model based on the two modal target monitoring data, wherein the target mechanical fault diagnostic model is trained using a sample monitoring data set having the same modality as the two modal target monitoring data, wherein the sample monitoring data set is multimodal monitoring data collected during the operation of various mechanical equipment. The target mechanical fault diagnostic model includes a symmetric fusion module and a fault classifier, wherein the symmetric fusion module is configured to perform bidirectional symmetric interaction on the two modal target monitoring data, and the fault classifier is configured to classify mechanical faults. Specifically: The target mechanical fault diagnosis model includes a first branch and a second branch, the first branch includes a plurality of first convolutional layers and the second branch includes a plurality of second convolutional layers, the number of the symmetric fusion modules, the number of the first convolutional layers, and the number of the second convolutional layers are equal, and the two modal target monitoring data are input into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result, including: Inputting the first modality target monitoring data of the two modal target monitoring data into the first branch to obtain a convolution result of each first convolutional layer, and inputting the second modality target monitoring data of the two modal target monitoring data into the second branch to obtain a convolution result of each second convolutional layer; The symmetric fusion module is used to perform bidirectional symmetric interaction on the convolution result of the j-th first convolutional layer in the first branch and the convolution result of the j-th second convolutional layer in the second branch, so as to obtain the interaction result of the j-th first convolutional layer in the first branch and the interaction result of the j-th second convolutional layer in the second branch; The interaction result of the j-th first convolutional layer is used as the input of the j+1-th first convolutional layer, and the interaction result of the j-th second convolutional layer is used as the input of the j+1-th second convolutional layer, until the convolution results of all first convolutional layers and the convolution results of all second convolutional layers complete bidirectional symmetrical interaction, thereby obtaining the depth interaction result of the first branch and the depth interaction result of the second branch; Concatenating and fusing the depth interaction results of the first branch and the depth interaction results of the second branch to obtain a depth fusion feature; Inputting the deep fusion features into the fault classifier to obtain a mechanical fault diagnosis result; The mechanical fault diagnosis unit is used to input the two modal target monitoring data into the target mechanical fault diagnosis model to obtain a mechanical fault diagnosis result.

8. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the mechanical fault diagnosis method based on deep fusion of multimodal monitoring data as described in any one of claims 1 to 6.

Images