Small sample fault detection method and system based on fusion network

Abstract

The invention discloses a small sample fault detection method and system based on a fusion network, and relates to the technical field of signal processing. The method comprises the following steps: acquiring multi-modal data in operation of mechanical equipment in real time, and preprocessing the multi-modal data; and constructing a small sample fault detection model, dividing the preprocessed multi-modal data into a training set and a verification set, performing fault detection training on the small sample fault detection model by using the training set, and verifying the trained small sample fault detection model by using the verification set. After verification is passed, a trained small sample fault detection model is obtained; and performing real-time fault detection on the mechanical equipment in operation by using the trained small sample fault detection model, and outputting a detection result. Through the method provided by the invention, the fault detection accuracy and the service life of the physical model can be effectively improved during fault detection of the mechanical equipment in operation, so that manpower and material resources for fault detection of the mechanical equipment are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing, and in particular to a small sample fault detection method and system based on a fusion network. Background Art

[0002] With the development of sensors and artificial intelligence, a large amount of equipment operation data can be collected, and an equipment health monitoring model can be established based on artificial intelligence digital drive to obtain information features hidden in the physical model from the huge data, thereby reducing human labor costs and the uncertainty of diagnostic results.

[0003] At present, my country is carrying out a comprehensive layout of production automation. The occurrence of mechanical failure may cause the production line to stop, and serious safety accidents may occur. Therefore, mechanical fault detection is a focal issue in the current environment.

[0004] The current system used for fault detection has low mobility and is only applicable to specific environments. Existing mechanical fault detection methods mainly rely on traditional signal processing technology and expert systems. However, traditional signal processing methods often perform poorly when dealing with complex and diverse mechanical faults, and are difficult to deal with high-dimensional, nonlinear and noisy data in practical applications. As a result, the physical model fault mechanism analysis of mechanical equipment is also restricted by external interference, making it difficult to establish a system to monitor the degree of faults, which in turn leads to low accuracy in fault detection of operating mechanical equipment or a short service life of the physical model, increasing the manpower and material resources for mechanical equipment fault detection. Summary of the invention

[0005] In order to overcome the problem that the existing signal processing technology has low accuracy in fault detection of operating mechanical equipment or the service life of the physical model is short, resulting in large manpower and material resources for mechanical equipment fault detection, the present invention proposes a small sample fault detection method and system based on a fusion network, which can effectively improve the accuracy of fault detection and the service life of the network model in the process of fault detection of operating mechanical equipment, thereby effectively reducing the manpower and material resources for mechanical equipment fault detection.

[0006] To achieve the purpose of the present invention, the present invention adopts the following technical solutions:

[0007] A small sample fault detection method based on a fusion network, the method comprising the following steps:

[0008] Collect multimodal data of mechanical equipment in operation in real time and pre-process the multimodal data;

[0009] Constructing a small sample fault detection model, and dividing the preprocessed multimodal data into a training set and a validation set, using the training set to perform fault detection training on the small sample fault detection model, and using the validation set to verify the trained small sample fault detection model, and obtaining a trained small sample fault detection model after verification;

[0010] The trained small sample fault detection model is used to perform real-time fault detection on running mechanical equipment and output the detection results.

[0011] In the above technical scheme, preprocessing the collected multimodal data can improve the practicality and reliability of the data, while reducing the computational requirements for data processing; the constructed small sample fault detection model can obtain a feature of the changes in multimodal data in mechanical equipment after training with a training set and verification with a verification set, so as to use the trained small sample fault detection model to perform fault detection on the running mechanical equipment according to the multimodal data collected in real time, and output accurate and reliable detection results, so as to improve the accuracy of fault detection and the service life of the network model, thereby effectively reducing the manpower and material resources for mechanical equipment fault detection.

[0012] Furthermore, the process of preprocessing the multimodal data includes:

[0013] Perform data cleaning and data slicing on the collected multimodal data;

[0014] Constructing a conditional adversarial network, and using the conditional adversarial network to generate a new set of multimodal data based on the multimodal data;

[0015] Combining the multimodal data after data cleaning and data slicing with the generated multimodal data to obtain an enhanced multimodal data set, and normalizing the enhanced multimodal data set;

[0016] Among them, the collected multimodal data includes audio data and vibration data of mechanical equipment.

[0017] Furthermore, the process of generating a new set of multimodal data based on the multimodal data using the conditional adversarial network includes:

[0018] The constructed conditional adversarial network includes a generator, a discriminator, and a loss function optimizer;

[0019] Set up several rounds of optimization training to iteratively optimize the conditional adversarial network;

[0020] The generator generates a multimodal signal based on the Gaussian noise in the multimodal data;

[0021] The discriminator determines the probability value of the multimodal signal being true data or false data based on the multimodal signal, and outputs the judgment result;

[0022] The loss function optimizer calculates the loss of the discriminator on real data and fake data and updates the discriminator parameters. It also calculates the loss of the generator and updates the generator parameters.

[0023] When the iterative optimization round ends or the loss function converges, a trained conditional adversarial network is obtained, and a new set of multimodal data is generated using the trained conditional adversarial network.

[0024] Furthermore, the loss function optimizer adopts the binary cross entropy loss function, which is expressed as:

[0025]

[0026] The Adam optimizer is used to iteratively optimize the conditional adversarial network during the optimization training process;

[0027] Among them, y represents the true label, Represents the predicted probability.

[0028] Furthermore, the expression for normalizing the enhanced multimodal dataset is:

[0029]

[0030] Among them, X represents the original sample data, X min represents the minimum value of the sample data set, X max Indicates the maximum value of the sample data set, X ′ Represents normalized data.

[0031] In the above technical scheme, data cleaning is performed on the collected multimodal data, which can effectively eliminate invalid data in the original collected data, thereby improving the practicality of the data; data slicing is performed on the collected multimodal data, which can divide the data into several small sample data, thereby making it more convenient to operate the data and improving the efficiency of data processing; a new set of multimodal data is generated according to the multimodal data using the conditional adversarial network; and the multimodal data after data cleaning and data slicing are combined with the generated multimodal data to obtain an enhanced multimodal data set, which can effectively solve the problem that there are fewer fault samples or a relatively single fault type in the original sampled multimodal data, thereby improving the model's ability to recognize and classify fault sample data features during model training, so as to improve the model's ability to capture faults, and at the same time, the generated data increases the robustness of the network; normalization is performed on the enhanced multimodal data set, which can effectively reduce the requirements for computing performance in the data calculation process, thereby improving the efficiency of data processing.

[0032] Furthermore, the process of performing fault detection training on the small sample fault detection model using the training set includes:

[0033] The constructed small sample fault detection model includes data signal conversion layer, important time segment capture layer, LSTM network layer, CNN network layer, attention residual network layer and fault detection result classification layer;

[0034] Set up several rounds of optimization training to optimize the parameters of the small sample fault detection model;

[0035] The data signal conversion layer is used to convert the one-dimensional data of the enhanced multimodal data set in the training set into two-dimensional data;

[0036] The important time segment capture layer is used to capture the important features of the sample data in the training set, and the captured features are weighted to obtain the attention weight features;

[0037] Use the LSTM network layer to extract the time series features of the one-dimensional data in the training set;

[0038] Use CNN network layers to extract deep two-dimensional data features from two-dimensional data;

[0039] The attention residual network layer is used to extract the fault features of the samples in the training set based on the extracted time series features and deep two-dimensional data features, and the residual block and attention mechanism are used;

[0040] The fault detection result classification layer identifies the fault type of the samples in the training set based on the attention weight features and fault features, and outputs the identification results;

[0041] The validation set is used to validate the small sample fault detection model after each round of training, and the loss function is set to optimize the parameters of the small sample fault detection model after each round of training. When the round of iterative training ends or the loss function converges, the trained small sample fault detection model is obtained;

[0042] Among them, the fault types include normal operation, minor fault, moderate fault and severe fault.

[0043] Furthermore, the process of converting the one-dimensional data of the enhanced multimodal data set in the training set into two-dimensional data using the data signal conversion layer includes:

[0044] The single data sample signal in the enhanced multimodal data set is divided into multiple short time frames, and each frame is windowed. The expression is:

[0045] y frame [n] = y[mH+n]·w[n];

[0046] Perform fast Fourier transform on each frame after windowing, the expression is:

[0047]

[0048] The Mel filter is used to map the spectrum of each frame signal after the fast Fourier transform to the Mel frequency scale, and the Mel spectrum energy of each frame signal is extracted. The expression is:

[0049]

[0050] The Mel spectrum energy of all frames is concatenated to obtain two-dimensional frequency domain signal data;

[0051] Wherein, N represents the length of each frame signal, H represents the step length of the overlapping part between frames, and m represents the mth frame signal.

[0052] Furthermore, the process of obtaining the attention weight feature includes:

[0053] The important time segment capture layer divides the audio sample data in the training set into several small segments, and splices the several small segments into the bit matrix A;

[0054] The matrix A is weighted for attention processing, and the expression is:

[0055]

[0056] Perform weighted summation on the obtained attention weight matrix A to obtain the attention weight feature matrix E;

[0057] Among them, Q, K, V all represent weight matrices, d k Represents the weight parameter.

[0058] Furthermore, a weighted loss function is set to optimize the parameters in the training process of the small sample fault detection model, and the expression is:

[0059]

[0060] Among them, x represents the sample data in the training set, e represents the logarithm of the natural number, and a represents the preset coefficient.

[0061] In the above technical solution, the data signal conversion layer can effectively convert the one-dimensional time domain signal in the training set into a two-dimensional frequency domain signal, thereby improving the model's recognition ability for the frequency domain signal; the important time segment capture layer can capture important features of the audio data in the training set, and in the process of model training, perform weighted processing on the relevant parameters of the model, thereby improving the model's recognition and analysis capabilities for specific data, thereby improving the model's training effect; the LSTM network layer can effectively extract the relevant time series features based on the one-dimensional time domain signal in the training set, and solve the problem of gradient vanishing and gradient explosion of traditional RNN when processing long-term dependencies; the CNN network layer can effectively extract the data based on the extracted two-dimensional frequency domain signal. Deep two-dimensional data features, thereby improving the model's two-dimensional feature recognition and analysis capabilities; the attention residual network layer can effectively extract the fault features of the samples in the training set based on the extracted time series features and deep two-dimensional data features, using the residual fast and attention mechanism, so that the model can more accurately identify and respond to potential fault conditions in complex engineering environments; the fault detection result classification layer can identify the fault type of the samples in the training set according to the attention weight features and fault features, and update the weight parameters in real time according to the set loss function to cope with more changes in mechanical equipment fault detection, while improving the service life of the model, thereby effectively reducing the manpower and material resources for mechanical equipment fault detection.

[0062] A small sample fault detection system based on a fusion network, the system comprising:

[0063] Data acquisition module, used to collect multi-modal data of mechanical equipment operation in real time;

[0064] A data processing module, used for preprocessing the multimodal data;

[0065] Model building module, used to build a small sample fault detection model;

[0066] A model training module is used to divide the preprocessed multimodal data into a training set and a verification set, use the training set to perform fault detection training on the small sample fault detection model, use the verification set to verify the trained small sample fault detection model, and obtain the trained small sample fault detection model after verification;

[0067] The fault detection module is used to perform real-time fault detection on running mechanical equipment using the trained small sample fault detection model and output the detection results.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] The present invention proposes a small sample fault detection method and system based on a fusion network. By preprocessing the collected multimodal data, the practicability and reliability of the data can be improved, while reducing the computational requirements for data processing. The constructed small sample fault detection model can obtain a feature of the multimodal data changes in mechanical equipment after training with a training set and verifying with a verification set, so as to use the trained small sample fault detection model to perform fault detection on the running mechanical equipment according to the multimodal data collected in real time, and output accurate and reliable detection results, so as to improve the accuracy of fault detection and the service life of the network model, thereby effectively reducing the manpower and material resources for mechanical equipment fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart of a small sample fault detection method based on a fusion network provided in an embodiment of the present application;

[0071] Figure 2 A schematic diagram of the basic steps of training a small sample fault detection model provided in an embodiment of the present application;

[0072] Figure 3 A schematic diagram of a network structure of a small sample fault detection model provided in an embodiment of the present application;

[0073] Figure 4 A flowchart of online learning steps in the small sample fault detection model training process provided in an embodiment of the present application;

[0074] Figure 5 A schematic diagram of the structure of a small sample fault detection system based on a fusion network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0077] Embodiment 1:

[0078] This embodiment provides a small sample fault detection method based on a fusion network. Figure 1 , the method comprises the following steps:

[0079] Step S1: collecting multimodal data of the operation of the mechanical equipment in real time, and preprocessing the multimodal data;

[0080] Step S2: construct a small sample fault detection model, and divide the preprocessed multimodal data into a training set and a verification set, use the training set to perform fault detection training on the small sample fault detection model, and use the verification set to verify the trained small sample fault detection model, and obtain the trained small sample fault detection model after verification;

[0081] Step S3: Use the trained small sample fault detection model to perform real-time fault detection on the running mechanical equipment and output the detection results.

[0082] In step S1, the process of preprocessing the multimodal data includes:

[0083] Step S11: performing data cleaning and data slicing processing on the collected multimodal data;

[0084] Step S12: constructing a conditional adversarial network, and using the conditional adversarial network to generate a new set of multimodal data according to the multimodal data;

[0085] Step S13: combining the multimodal data after data cleaning and data slicing with the generated multimodal data to obtain an enhanced multimodal data set, and normalizing the enhanced multimodal data set;

[0086] Among them, the collected multimodal data includes audio data and vibration data of mechanical equipment.

[0087] Specifically, multiple sensors are used to collect audio data and vibration data of different axes of the instrument arm, and the operating status is manually judged and used as the data label. Multimodal data of the same machine are packaged. The advantage of multimodal data over single-modal data is that information of different modes is complementary, and more information is received. Multimodal input has better fault tolerance than single-modal data, which can improve the robustness of the network.

[0088] In step S11, data cleaning, the audio data collected initially contains invalid data, such as audio data collected when the machine is not moving. The audio data is processed by setting a sliding window and a threshold, and the sliding window size is set to 2200 data sizes, that is, 0.1 seconds, the sliding step is set to 1100, and the threshold is set to 0.1; when the mean of all data in the window is less than the threshold, the data in the window is determined to be invalid data when the robot arm is in a static state, and the data is deleted. Data set organization, in order to facilitate model network learning, the audio data of different lengths are sliced ​​and processed, integrated into audio samples of the same size, and the data is sliced ​​by a sliding window. The sliding window size is set to 22050*30, that is, 30 seconds of audio data, and the step size is set to the same as the sliding window size, that is, the data between samples do not overlap. The data in the sliding window is stored separately to produce new samples, and the degree of fault in the file name is used as the label of the sample. In this way, a long file is sliced ​​into several small files.

[0089] In step S12, the process of generating a new set of multimodal data according to the multimodal data using the conditional adversarial network includes:

[0090] The constructed conditional adversarial network includes a generator, a discriminator, and a loss function optimizer;

[0091] Set up several rounds of optimization training to iteratively optimize the conditional adversarial network;

[0092] The generator generates a multimodal signal based on the Gaussian noise in the multimodal data;

[0093] The discriminator determines the probability value of the multimodal signal being true data or false data based on the multimodal signal, and outputs the judgment result;

[0094] The loss function optimizer calculates the loss of the discriminator on real data and fake data and updates the discriminator parameters. It also calculates the loss of the generator and updates the generator parameters.

[0095] When the iterative optimization round ends or the loss function converges, a trained conditional adversarial network is obtained, and a new set of multimodal data is generated using the trained conditional adversarial network.

[0096] Furthermore, the loss function optimizer adopts the binary cross entropy loss function, which is expressed as:

[0097]

[0098] The Adam optimizer is used to iteratively optimize the conditional adversarial network during the optimization training process;

[0099] Among them, y represents the true label, Represents the predicted probability.

[0100] Specifically, the generator is essentially a neural network, and its main structure includes deconvolution layer and upsampling layer. The input is Gaussian noise and category label, and the output is multimodal signal. The discriminator is essentially a neural network, and its main structure is convolutional neural network and LSTM module. The input is multimodal data, and the output is the probability value of the data being the real data and the probability value of its corresponding label. The loss function uses binary cross entropy loss, and the formula is

[0101]

[0102] y is the true label (value is 1), is the predicted probability. The optimizer uses the Adam optimizer

[0103] When Gaussian noise is input into the generator, its probability density function (PDF) is

[0104]

[0105] Where u is the mean of the noise, which is usually set to 0, and σ is the standard deviation of the noise, which is used to control the noise amplitude. Get fake data and mark it as a fake label (0), calculate the loss of the discriminator on real data and fake data, and update the discriminator parameters. Calculate the discriminator's judgment result on fake data, calculate the generator loss, and update the generator parameters. After the adversarial network training is completed, generate a batch of data to strengthen the data set. The adversarial network is used to solve the problem of small number of faulty data samples. At the same time, the generated data increases the robustness of the network.

[0106] Furthermore, the expression for normalizing the enhanced multimodal dataset is:

[0107]

[0108] Among them, X represents the original sample data, X min represents the minimum value of the sample data set, X max Indicates the maximum value of the sample data set, X ′ Represents normalized data.

[0109] It can be understood that data cleaning of the collected multimodal data can effectively eliminate invalid data in the original collected data, thereby improving the practicality of the data; data slicing of the collected multimodal data can divide the data into several small sample data, thereby making it easier to operate the data and improving the efficiency of data processing; using the conditional adversarial network to generate a new set of multimodal data based on the multimodal data; and combining the multimodal data after data cleaning and data slicing with the generated multimodal data to obtain an enhanced multimodal data set, which can effectively solve the problem of fewer fault samples or a relatively single fault type in the original sampled multimodal data, thereby improving the model's ability to recognize and classify fault sample data features during model training, so as to improve the model's ability to capture faults, and at the same time, the generated data increases the robustness of the network; normalizing the enhanced multimodal data set can effectively reduce the requirements for computing performance during data calculation, thereby improving the efficiency of data processing.

[0110] In this embodiment, preprocessing the collected multimodal data can improve the practicality and reliability of the data, while reducing the computational requirements for data processing; the constructed small sample fault detection model can obtain a feature of the changes in multimodal data in mechanical equipment after training with a training set and verification with a verification set, so as to use the trained small sample fault detection model to perform fault detection on the running mechanical equipment according to the multimodal data collected in real time, and output accurate and reliable detection results, so as to improve the accuracy of fault detection and the service life of the network model, thereby effectively reducing the manpower and material resources for mechanical equipment fault detection.

[0111] Embodiment 2:

[0112] This embodiment further describes steps S2 to S3 in the first embodiment, as follows:

[0113] In step S2, the process of performing fault detection training on the small sample fault detection model using the training set includes:

[0114] See also Figure 2 and Figure 3 The constructed small sample fault detection model includes data signal conversion layer, important time segment capture layer, LSTM network layer, CNN network layer, attention residual network layer and fault detection result classification layer;

[0115] Step S21: setting several rounds of optimization training to optimize the parameters of the small sample fault detection model;

[0116] Step S22: using a data signal conversion layer to convert the one-dimensional data of the enhanced multimodal data set in the training set into two-dimensional data;

[0117] Step S23: using the important time segment capture layer to capture important features of the sample data in the training set, and performing attention weighting processing on the captured features to obtain attention weight features;

[0118] Step S24: using the LSTM network layer to extract the time series features of the one-dimensional data in the training set;

[0119] Step S25: extracting deep two-dimensional data features from the two-dimensional data using the CNN network layer;

[0120] Step S26: using the attention residual network layer to extract the fault features of the samples in the training set according to the extracted time series features and deep two-dimensional data features, using the residual block and attention mechanism;

[0121] Step S27: The fault detection result classification layer identifies the fault type of the samples in the training set according to the attention weight features and the fault features, and outputs the identification results;

[0122] Step S28: using the validation set to validate the small sample fault detection model after each round of training, and setting the loss function to optimize the parameters of the small sample fault detection model after each round of training, when the round of iterative training ends or the loss function converges, the trained small sample fault detection model is obtained;

[0123] Among them, the fault types include normal operation, minor fault, moderate fault and severe fault.

[0124] In step S28, see Figure 2 , the process of using the data signal conversion layer to convert the one-dimensional data of the training set to strengthen the multimodal data set into two-dimensional data includes:

[0125] The single data sample signal in the enhanced multimodal data set is divided into multiple short-time frames, each frame length is N, the length of the overlapping part between frames is H, and the length of the audio signal is L. Then the number of frames is:

[0126]

[0127] And perform windowing processing (Haming window function) on each frame, the expression is:

[0128]

[0129] Each frame after windowing can be expressed as:

[0130] y frame [n] = y[mH+n]·w[n];

[0131] Perform fast Fourier transform on each frame after windowing, the expression is:

[0132]

[0133] Create a set of Mel filters to map the spectrum to the Mel frequency scale. The scale formula is

[0134]

[0135] The Mel filter is used to map the spectrum of each frame signal after the fast Fourier transform to the Mel frequency scale, and the Mel spectrum energy of each frame signal is extracted. The expression is:

[0136]

[0137] The Mel spectrum energy of all frames is concatenated to obtain two-dimensional frequency domain signal data;

[0138] Specifically, M Mel filters are used, each covering a certain frequency range. The center frequencies of the Mel filters are equally spaced on the Mel scale. The Mel filter bank is applied to the spectrum of each frame to obtain the energy value on the Mel scale. Assume that the frequency response of the mth Mel filter is H m [k], then the Mel spectrum energy E m as follows:

[0139]

[0140] Take the logarithm of each Mel frequency energy value to transform the amplitude to a logarithmic scale. This step can suppress large amplitude changes, making the data more suitable for further processing and analysis: E′ m =log(E m +ε), ε is a small constant, and finally the logarithmic Mel frequency energy values ​​of all frames are concatenated together to form a two-dimensional time-frequency matrix, with the horizontal axis representing time (frame) and the vertical axis representing Mel frequency. This two-dimensional time-frequency matrix is ​​the Mel frequency spectrum.

[0141] Wherein, N represents the length of each frame signal, H represents the step length of the overlapping part between frames, and m represents the mth frame signal.

[0142] In step S23, see Figure 2 , the process of obtaining the attention weight feature includes:

[0143] The important time segment capture layer divides the audio sample data in the training set into several small segments, and splices the several small segments into the bit matrix A;

[0144] The matrix A is weighted for attention processing, and the expression is:

[0145]

[0146] Perform weighted summation on the obtained attention weight matrix A to obtain the attention weight feature matrix E;

[0147] Among them, Q, K, V all represent weight matrices, d k Represents the weight parameter.

[0148] Specifically, important time segments are captured by using a window of length 4410 units and a sliding step of 4410. The 22050*30 audio segment is divided into 150 small segments, and the 150 segments are spliced ​​into a 150×4410 matrix A. Three matrices of size 4410×d are initialized. k Weight parameter matrices WQ, WK, WV. Multiply the three weight matrices by matrix A to get the Q, K, V moments, and finally the formula:

[0149]

[0150] Get the weight matrix, multiply the attention weight matrix A with the value matrix V, and get the output matrix E (i.e., attention weight feature matrix E) by weighted summation. This output matrix E contains the weighted representation of each element of the input sequence, where the weight is determined by the attention weight matrix A. Each element in the E matrix is ​​the weighted sum of all elements in the input sequence, reflecting the relevance of each element to the current query. The higher the weight value, the higher the impact of the fragment on the final result. Save the output E for the final recognition network to weight.

[0151] In step S24, see Figure 3 , the LSTM network layer is used to extract the time series features of the one-dimensional data in the training set. Specifically: the LSTM network layer (the input is one-dimensional time series data, and the number of channels is four) is a special recursive neural network specifically used to process and predict time series data. LSTM solves the problem of gradient vanishing and gradient exploding when traditional RNN processes long-term dependencies by introducing a gating mechanism. The basic unit of LSTM is called LSTM unit or LSTM cell. Each LSTM unit is mainly composed of the following parts: forgetting, input, candidate memory, output and cell state. By calculating the forgetting gate, calculating the output gate and candidate memory, updating the cell state, and calculating the output gate, the important information in the long time series can be effectively memorized to obtain the characteristics of the time series.

[0152] In step S25, see Figure 3, use the CNN network layer to extract deep two-dimensional data features in two-dimensional data. Specifically: the CNN network layer is a convolutional network, the input is two-dimensional frequency domain data, the number of channels is four (the X-axis, Y-axis, Z-axis of vibration data and audio data respectively), and the convolution layer in the CNN network layer can capture the local features in the two-dimensional data through the convolution kernel. Through multi-layer convolution, deep information can be extracted, thereby obtaining important deep two-dimensional data features.

[0153] In step S26, see Figure 3 , using the attention residual network layer to extract the fault features of the samples in the training set according to the extracted time series features and deep two-dimensional data features, the residual block and attention mechanism are used. Specifically, the attention residual network layer includes convolutional layer, residual block, attention module and the final fully connected layer. Residual module, as the network continues to deepen, the neural network will have degradation problems. The residual module introduces jump connections to solve this problem. The residual module contains a main path and a jump connection. The main path contains several convolutional layers, pooling, normalization, and activation functions. The jump connection can directly transfer the input to the output. In deep networks, it is very difficult to directly learn complex mappings from input to output. By introducing the residual module, the network only needs to learn the residual between the input and output, which is usually much simpler than directly learning the complete mapping.

[0154] Attention module, set the attention module to the end of the network, input the features extracted by the network into the attention mechanism, and the attention mechanism will adjust the weights of each feature to make the entire network focus on features that are more relevant to the task. These weights reflect the relative importance of features in fault state identification and are dynamically adjusted through back propagation and optimization algorithms. The attention mechanism is not only a method to selectively process input data, but also a mechanism to dynamically adjust and optimize the degree of attention to key fault features. This capability enables the fault detection system to more accurately identify and respond to potential fault conditions in complex engineering environments to extract fault features of samples in the training set.

[0155] In step S27, see Figure 2 and Figure 4 The fault detection result classification layer identifies the fault type of the samples in the training set according to the attention weight feature and the fault feature. Specifically, the processed data is input into the network to complete the identification. When the result shows a moderate fault or a severe fault, an alarm is immediately sounded and then inspection and maintenance are carried out. The data is labeled and then passed through the weighted loss function. The weighted function (sigmoid) is as follows This function makes the weight change in the range of [0, 1], and can smoothly adjust the weight. Finally, it is put into the network for learning to complete the network parameter update. When the result is normal operation or a minor fault, the network continues to observe for 24 hours. In this 24 hours, if the mechanical operation state remains unchanged or the fault degree only deepens by one level (referring to normal operation to a minor fault, or a minor fault to a moderate fault), the data is labeled, and then through the weighted loss function, it is finally put into the network for learning to complete the network parameter update. If the operation state deepens by two or three levels (referring to the operation state jumping from normal to a moderate fault or a severe fault, or a minor fault jumping to a severe fault) or the operation state improves (referring to a minor fault to normal operation), it means that there is a high possibility of misjudgment in the previous detection data, so the data is deleted and not used for updating the network weight parameters. The online learning method can increase the service life of the system and reduce the number of manual updates of the system, thereby saving manpower and material resources.

[0156] In step S28, a weighted loss function is set to optimize the parameters in the small sample fault detection model training process, and the expression is:

[0157]

[0158] Among them, x represents the sample data in the training set, e represents the logarithm of the natural number, and a represents the preset coefficient.

[0159] In the process of training the small sample fault detection model, the data set is divided into training set, test set, and validation set in the ratio of 80%, 10%, and 10%. The training set data is input into the network model. At the same time, through manual judgment, the EPOCH with small training loss and high accuracy is selected for truncation to prevent the network from overfitting due to too many training times. The network parameters of the EPOCH are saved for fault detection of real-time data.

[0160] In this embodiment, the data signal conversion layer can effectively convert the one-dimensional time domain signal in the training set into a two-dimensional frequency domain signal, thereby improving the model's recognition ability for the frequency domain signal; the important time segment capture layer can capture important features of the audio data in the training set, and in the process of model training, perform weighted processing on the relevant parameters of the model, thereby improving the model's recognition and analysis capabilities for specific data, thereby improving the model's training effect; the LSTM network layer can effectively extract the relevant time series features based on the one-dimensional time domain signal in the training set, and solve the problems of gradient vanishing and gradient explosion in traditional RNN when processing long-term dependencies; the CNN network layer can effectively extract the deep information in the data based on the extracted two-dimensional frequency domain signal. The attention residual network layer can effectively extract the fault features of the samples in the training set according to the extracted time series features and deep two-dimensional data features, and use the residual fast and attention mechanism to enable the model to more accurately identify and respond to potential fault conditions in complex engineering environments; the fault detection result classification layer can identify the fault types of the samples in the training set according to the attention weight features and fault features, and update the weight parameters in real time according to the set loss function to cope with more changes in mechanical equipment fault detection, while improving the service life of the model, thereby effectively reducing the manpower and material resources for mechanical equipment fault detection.

[0161] Embodiment three:

[0162] This implementation provides a small sample fault detection system based on fusion network, see Figure 5 , the system comprising:

[0163] Data acquisition module, used to collect multi-modal data of mechanical equipment operation in real time;

[0164] A data processing module, used for preprocessing the multimodal data;

[0165] Model building module, used to build a small sample fault detection model;

[0166] A model training module is used to divide the preprocessed multimodal data into a training set and a verification set, use the training set to perform fault detection training on the small sample fault detection model, use the verification set to verify the trained small sample fault detection model, and obtain the trained small sample fault detection model after verification;

[0167] The fault detection module is used to perform real-time fault detection on running mechanical equipment using the trained small sample fault detection model and output the detection results.

[0168] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A small sample fault detection method based on fusion network, characterized in that: The method comprises the following steps: Collect multimodal data of mechanical equipment in operation in real time and pre-process the multimodal data; Constructing a small sample fault detection model, and dividing the preprocessed multimodal data into a training set and a validation set, using the training set to perform fault detection training on the small sample fault detection model, and using the validation set to verify the trained small sample fault detection model, and obtaining a trained small sample fault detection model after verification; The trained small sample fault detection model is used to perform real-time fault detection on running mechanical equipment and output the detection results.

2. The small sample fault detection method based on fusion network according to claim 1 is characterized in that: The process of preprocessing the multimodal data includes: Perform data cleaning and data slicing on the collected multimodal data; Constructing a conditional adversarial network, and using the conditional adversarial network to generate a new set of multimodal data based on the multimodal data; Combining the multimodal data after data cleaning and data slicing with the generated multimodal data to obtain an enhanced multimodal data set, and normalizing the enhanced multimodal data set; Among them, the collected multimodal data includes audio data and vibration data of mechanical equipment.

3. The small sample fault detection method based on fusion network according to claim 2 is characterized in that: The process of generating a new set of multimodal data based on the multimodal data using the conditional adversarial network includes: The constructed conditional adversarial network includes a generator, a discriminator, and a loss function optimizer; Set up several rounds of optimization training to iteratively optimize the conditional adversarial network; The generator generates a multimodal signal based on the Gaussian noise in the multimodal data; The discriminator determines the probability value of the multimodal signal being true data or false data based on the multimodal signal, and outputs the judgment result; The loss function optimizer calculates the loss of the discriminator on real data and fake data and updates the discriminator parameters. It also calculates the loss of the generator and updates the generator parameters. When the iterative optimization round ends or the loss function converges, a trained conditional adversarial network is obtained, and a new set of multimodal data is generated using the trained conditional adversarial network.

4. The small sample fault detection method based on fusion network according to claim 3 is characterized in that: The loss function optimizer uses the binary cross entropy loss function, which is expressed as: The Adam optimizer is used to iteratively optimize the conditional adversarial network during the optimization training process; Among them, y represents the true label, represents the predicted probability.

5. The small sample fault detection method based on fusion network according to claim 2 is characterized in that: The expression for normalizing the enhanced multimodal dataset is: Among them, X represents the original sample data, X min Represents the minimum value of the sample data set, X max Indicates the maximum value of the sample data set, X ′ Represents normalized data.

6. The small sample fault detection method based on fusion network according to claim 2 is characterized in that: The process of performing fault detection training on the small sample fault detection model using the training set includes: The constructed small sample fault detection model includes data signal conversion layer, important time segment capture layer, LSTM network layer, CNN network layer, attention residual network layer and fault detection result classification layer; Set up several rounds of optimization training to optimize the parameters of the small sample fault detection model; The data signal conversion layer is used to convert the one-dimensional data of the enhanced multimodal data set in the training set into two-dimensional data; The important time segment capture layer is used to capture the important features of the sample data in the training set, and the captured features are weighted to obtain the attention weight features; Use the LSTM network layer to extract the time series features of the one-dimensional data in the training set; Use CNN network layers to extract deep two-dimensional data features from two-dimensional data; The attention residual network layer is used to extract the fault features of the samples in the training set based on the extracted time series features and deep two-dimensional data features, and the residual block and attention mechanism are used; The fault detection result classification layer identifies the fault type of the samples in the training set based on the attention weight features and fault features, and outputs the identification results; The validation set is used to validate the small sample fault detection model after each round of training, and the loss function is set to optimize the parameters of the small sample fault detection model after each round of training. When the round of iterative training ends or the loss function converges, the trained small sample fault detection model is obtained; Among them, the fault types include normal operation, minor fault, moderate fault and severe fault.

7. The small sample fault detection method based on fusion network according to claim 6 is characterized in that: The process of using the data signal conversion layer to convert the one-dimensional data of the training set to strengthen the multimodal data set into two-dimensional data includes: The single data sample signal in the enhanced multimodal data set is divided into multiple short time frames, and each frame is windowed. The expression is: y frame [n]=y[mH+n]·w[n]; Perform fast Fourier transform on each frame after windowing, the expression is: The Mel filter is used to map the spectrum of each frame signal after the fast Fourier transform to the Mel frequency scale, and the Mel spectrum energy of each frame signal is extracted. The expression is: The Mel spectrum energy of all frames is concatenated to obtain two-dimensional frequency domain signal data; Wherein, N represents the length of each frame signal, H represents the step length of the overlapping part between frames, and m represents the mth frame signal.

8. The small sample fault detection method based on fusion network according to claim 6 is characterized in that: The process of obtaining the attention weight feature includes: The important time segment capture layer divides the audio sample data in the training set into several small segments, and splices the several small segments into the bit matrix A; The matrix A is weighted for attention processing, and the expression is: Perform weighted summation on the obtained attention weight matrix A to obtain the attention weight feature matrix E; Among them, Q, K, V all represent weight matrices, d k Represents the weight parameter.

9. The small sample fault detection method based on fusion network according to claim 6 is characterized in that: The weighted loss function is set to optimize the parameters in the small sample fault detection model training process. The expression is: Among them, x represents the sample data in the training set, e represents the logarithm of the natural number, and a represents the preset coefficient.

10. A small sample fault detection system based on fusion network, characterized in that: The system comprises: Data acquisition module, used to collect multi-modal data of mechanical equipment operation in real time; A data processing module, used for preprocessing the multimodal data; Model building module, used to build a small sample fault detection model; A model training module is used to divide the preprocessed multimodal data into a training set and a verification set, use the training set to perform fault detection training on the small sample fault detection model, use the verification set to verify the trained small sample fault detection model, and obtain the trained small sample fault detection model after verification; The fault detection module is used to perform real-time fault detection on running mechanical equipment using the trained small sample fault detection model and output the detection results.

Images