A method and apparatus for detecting device failure based on vibration images

Abstract

The embodiment of the application provides a kind of device fault detection method and device based on vibration image, the vibration image of equipment is obtained by image acquisition, after carrying out feature extraction to vibration image, the frequency amplitude two-dimensional histogram of vibration frequency and amplitude value about vibration image is obtained;According to and pre-set through classification convolutional neural network model, the vibration image, frequency amplitude histogram obtained is classified and calculated, and the classification result for vibration image and frequency amplitude histogram is obtained, finally, according to classification result, equipment is judged to be abnormal fault;Around the vibration image feature of improved classification convolutional neural network model analysis equipment, with non-contact information acquisition method, equipment vibration is analyzed, and vibration signal detection means for industrial equipment is expanded.

Description

Technical Field

[0001] This invention relates to the field of machine vision (CV) technology, and in particular to a method, apparatus, electronic device, and readable storage medium for equipment fault detection based on vibration images. Background Technology

[0002] Industrial equipment, as an ever-increasing source of production, can greatly improve the efficiency of labor production and has the advantages of stability and reliability. During equipment operation, the mechanical activities of various internal functional components are often accompanied by a certain amount of vibration and noise. By collecting and analyzing the acoustic characteristics of the noise, the operating status of the equipment can be effectively analyzed, which is of great significance for equipment inspection and maintenance.

[0003] In related technologies, vibration sensors are typically used to capture vibration signals generated during the operation of industrial equipment. The collected vibration signals are then analyzed and processed to determine the operating status of the equipment and to identify any abnormal faults.

[0004] However, existing solutions require connecting the sensor world to the device and collecting signals through contact deployment, making the entire detection process very cumbersome and limiting the application methods. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and readable storage medium for equipment fault detection based on vibration images, in order to solve the problem that the existing detection methods for industrial equipment are too limited.

[0006] To solve the above-mentioned technical problems, the embodiments of the present invention are implemented as follows:

[0007] In a first aspect, embodiments of the present invention provide a device fault detection method based on vibration images, the method comprising:

[0008] Acquire vibration images of the equipment;

[0009] Feature extraction is performed on the vibration image to obtain the frequency-amplitude histogram of the vibration image;

[0010] Based on the vibration image, the frequency amplitude histogram, and a preset classification convolutional neural network model, classification results are obtained for the vibration image and the frequency amplitude histogram.

[0011] The classification results are used to determine whether the equipment is faulty.

[0012] Optionally, obtaining the classification result for the vibration image and the frequency amplitude histogram based on the vibration image, the frequency amplitude histogram, and a preset classification convolutional neural network model includes:

[0013] The image channels of the vibration image and the image channels of the frequency amplitude histogram are fused to obtain the target fused image;

[0014] The target fused image is classified using a pre-defined classification convolutional neural network model to obtain classification results.

[0015] Optionally, both the vibration image and the frequency amplitude histogram are two-dimensional histograms; the step of fusing the image channels of the vibration image and the image channels of the frequency amplitude histogram to obtain the target fused image includes:

[0016] Based on the image channel information of the vibration image, a collection of vibration image matrices is obtained, including at least one vibration image matrix, wherein a single vibration image matrix includes all content information of one of the image channels of the vibration image;

[0017] Based on the image channel information of the frequency amplitude histogram, a set of frequency amplitude histogram matrices is obtained, including at least one frequency amplitude histogram matrix. Each frequency amplitude histogram matrix includes all content information of one of the image channels of the frequency amplitude histogram.

[0018] The vibration image matrix set is fused with the image channels of the frequency amplitude histogram matrix set to obtain a fused matrix set; wherein, the fused matrix set includes all content information of the image channels of the vibration images and the image channels of the frequency amplitude histogram.

[0019] The target fused image is generated based on the fusion matrix set.

[0020] Optionally, the classification convolutional neural network model includes at least one classifier; the step of classifying the target fused image using a preset classification convolutional neural network model to obtain a classification result includes:

[0021] The target fused image is evaluated and scored by a classifier in a classification convolutional neural network model.

[0022] Obtain the arithmetic mean of all scores from the classifier;

[0023] The arithmetic mean is used as the classification result.

[0024] Optionally, determining whether the device is faulty based on the classification result includes:

[0025] If the arithmetic mean of the target fused image in the classification result is greater than or equal to a first preset threshold, then it is determined that the device is faulty;

[0026] If the arithmetic mean of the target fused image is less than a first preset threshold, then it is determined that the device is not faulty.

[0027] Optionally, the vibration images include: normal vibration image samples and abnormal vibration image samples;

[0028] Before performing feature extraction on the vibration image to obtain the frequency-amplitude histogram of the vibration image, the method further includes:

[0029] The number of samples in the vibration image is adjusted by a sample sampling function so that the ratio of the number of abnormal vibration image samples to the number of normal vibration image samples is equal to a first preset ratio threshold.

[0030] Optionally, after classifying the target fused image using a preset classification convolutional neural network model and obtaining the classification result, the method further includes:

[0031] The classifier of the classification convolutional neural network model is trained using a loss function to optimize the computational parameters used by the classifier when classifying target fused images.

[0032] Secondly, embodiments of the present invention provide a device for detecting equipment faults based on vibration images, the device comprising:

[0033] The image acquisition module is used to acquire vibration images of the device;

[0034] The feature extraction module is used to extract features from the vibration image to obtain the frequency amplitude histogram of the vibration image;

[0035] The image classification module is used to obtain classification results for the vibration image and the frequency amplitude histogram based on the vibration image, the frequency amplitude histogram, and a preset classification convolutional neural network model.

[0036] The equipment fault determination module is used to determine whether the equipment has a fault based on the classification results.

[0037] Optionally, the image classification module further includes:

[0038] The image fusion acquisition submodule is used to fuse the image channels of the vibration image with the image channels of the frequency amplitude histogram to obtain a target fused image;

[0039] The classification algorithm execution submodule is used to classify the target fused image using a preset classification convolutional neural network model to obtain the classification result.

[0040] Optionally, the fused image acquisition submodule further includes:

[0041] The vibration image channel matrix acquisition unit is used to obtain a collection of vibration image matrices including at least one vibration image matrix based on the image channel information of the vibration image. Each vibration image matrix includes all content information of one image channel of the vibration image.

[0042] The frequency amplitude image channel matrix acquisition unit is used to obtain a set of frequency amplitude histogram matrices including at least one frequency amplitude histogram matrix based on the image channel information of the frequency amplitude histogram, wherein a single frequency amplitude histogram matrix includes all content information of the image channel of one of the frequency amplitude histograms;

[0043] A matrix fusion unit is used to fuse the image channels of the vibration image matrix set with the image channels of the frequency amplitude histogram matrix set to obtain a fused matrix set; wherein, the fused matrix set includes all content information of the image channels of the vibration image and the image channels of the frequency amplitude histogram;

[0044] A fused image generation unit is used to generate the target fused image based on the fusion matrix set.

[0045] Optionally, the classification algorithm execution submodule further includes:

[0046] The classification result acquisition unit is used to evaluate and score the target fused image by using the classifier in the classification convolutional neural network model;

[0047] The classification result averaging unit is used to obtain the arithmetic mean of all scores from the classifier;

[0048] The classification result generation unit is used to take the arithmetic mean as the classification result.

[0049] Optionally, the device further includes:

[0050] The sample ratio adjustment module is used to adjust the number of samples in the vibration image through a sample sampling function, so that the ratio of the number of abnormal vibration image samples to the number of normal vibration image samples in the vibration image is equal to a first preset ratio threshold.

[0051] Optionally, the device further includes:

[0052] The network training execution module is used to train the classifier of the classification convolutional neural network model through a loss function, and optimize the computational parameters used by the classifier when classifying target fused images.

[0053] Thirdly, embodiments of the present invention provide an electronic device, including: a processor;

[0054] Memory used to store the processor's executable instructions;

[0055] The processor is configured to execute the instructions to implement any of the methods described above.

[0056] Fourthly, embodiments of the present invention provide a storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the methods described above.

[0057] In this embodiment of the invention, vibration images of the equipment are first acquired through image acquisition. After feature extraction of the vibration images, a two-dimensional frequency-amplitude histogram of the vibration frequency and amplitude values ​​is obtained. Based on a preset classification convolutional neural network model, the acquired vibration images and frequency-amplitude histograms are classified and calculated to obtain classification results for the vibration images and frequency-amplitude histograms. Finally, the equipment is judged to have abnormal faults based on the classification results. By analyzing the vibration image features of the equipment using an improved classification convolutional neural network model, and using a non-contact information acquisition method to analyze equipment vibration, the means of vibration signal detection for industrial equipment is expanded.

[0058] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0059] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0060] Figure 1 A simplified flowchart of a device fault detection method based on vibration images provided in an embodiment of the present invention;

[0061] Figure 2 A detailed flowchart illustrating the steps of a device fault detection method based on vibration images provided in this invention.

[0062] Figure 3 A complete implementation logic diagram of a device fault detection method based on vibration images provided in this embodiment of the invention;

[0063] Figure 4This is a schematic diagram of the module composition of a device fault detection device based on vibration images provided in an embodiment of the present invention;

[0064] Figure 5 A functional component relationship diagram of an electronic device provided in an embodiment of the present invention;

[0065] Figure 6 This is a functional component relationship diagram of another electronic device provided in an embodiment of the present invention. Detailed Implementation

[0066] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0067] Reference Figure 1 This diagram illustrates a simplified flowchart of a device fault detection method based on vibration images provided by an embodiment of the present invention; as shown below. Figure 1 As shown, the method steps may include:

[0068] Step 101: Obtain vibration images of the equipment.

[0069] During the operation of industrial equipment, complex vibration signals are generated due to the relative movement of various components within the facility. These vibration signals contain a wealth of information about equipment abnormalities or malfunctions, serving as important feedback indicators of the equipment's operating status. Capturing and analyzing vibration signals plays a crucial role in monitoring the equipment's operating status and analyzing equipment anomalies.

[0070] This invention provides a device fault detection method based on vibration images. First, a third device continuously acquires vibration images generated by industrial equipment during operation. Then, vibration feature information is extracted from the images for analysis. The vibration image represents the amplitude and frequency of vibration at each pixel on the equipment.

[0071] Specifically, in this embodiment of the invention, a high-precision camera is used to acquire vibration images of the device under test. Compared to traditional methods that capture vibration signals by installing contact vibration sensors on the device, the camera-based acquisition method is more flexible and practical, and can also switch between different shooting angles to capture the operating state of industrial equipment from multiple dimensions to obtain vibration images.

[0072] Step 102: Extract features from the vibration image to obtain the frequency amplitude histogram of the vibration image.

[0073] For vibration images captured by high-definition cameras, the contents of the images need to be normalized before analysis and calculation. This leads to feature extraction of the vibration images and the acquisition of frequency amplitude histograms.

[0074] It's important to note that image histograms are crucial pixel statistics in image processing, reflecting the probability distribution of image pixels. Histograms obtained through feature extraction differ from the original captured vibration image; they no longer represent texture information to display image content, but rather provide statistical information about the image pixels. Since the same object has the same grayscale value in an image regardless of rotation or translation, histograms possess advantages such as translation invariance and scaling invariance.

[0075] The histogram obtained through feature extraction is a statistical chart with the image grayscale value on the horizontal axis and the number of grayscale values ​​or the proportion of grayscale values ​​on the vertical axis.

[0076] Specifically, in this embodiment of the invention, the frequency amplitude histogram obtained by feature extraction is a statistical histogram that contains frequency amplitude information of the gray values ​​of pixels in the vibration image.

[0077] Step 103: Based on the vibration image, the frequency amplitude histogram, and the preset classification convolutional neural network model, obtain the classification results for the vibration image and the frequency amplitude histogram.

[0078] Following step 102, after obtaining the vibration image and frequency amplitude histogram, feature analysis processing of the image can begin.

[0079] The MobileNet V3 network model is used for image parsing and processing. The MobileNet series of lightweight networks is a low-weight deep convolutional computation network proposed for embedded devices such as mobile phones. Its core idea is depthwise separable convolution, which can accurately segment the region of the object to be measured through object region segmentation. This neural network improves the quality of the representation produced by the network by explicitly modeling the interdependencies between the network's convolutional feature channels.

[0080] It is worth noting that the Mobile Net V3 network model used in this embodiment employs a multi-head classification method, enabling the model network to capture richer feature information. Each selection head is an independent classifier with different selection and classification calculation parameters. The classification result for the two-dimensional image is the specific score generated by the classification algorithm. The final classification result is obtained by arithmetically averaging the scores from multiple classification heads within the model.

[0081] Step 104: Determine whether the equipment is faulty based on the classification results.

[0082] The classification results obtained from the Mobile Net V3 classification neural network are used to determine the equipment status of industrial equipment and whether there are any abnormal faults in the equipment.

[0083] In practical implementation, the classification result of each classifier in the Mobile Net V3 classification neural network model is an arbitrary decimal between 0 and 1, such as 0.4, 0.6, 0.8, etc. The result obtained by arithmetically averaging the scores of all classification heads is determined if the average value is greater than a certain judgment threshold, such as 0.5 points. This indicates that the operating status of the industrial equipment from which the captured image originated is abnormal.

[0084] In summary, the device fault detection method based on vibration images provided by this invention first acquires vibration images of the device through image acquisition. After feature extraction from the vibration images, a two-dimensional frequency-amplitude histogram is obtained, representing the vibration frequency and amplitude values. Based on a preset classification convolutional neural network model, the acquired vibration images and frequency-amplitude histogram are classified and calculated to obtain classification results. Finally, the device is judged to have abnormal faults based on the classification results. This method analyzes the vibration image features of the device using an improved classification convolutional neural network model, employing a non-contact information acquisition method to analyze device vibration, thus expanding the means of vibration signal detection for industrial equipment.

[0085] Reference Figure 2 This diagram illustrates a detailed implementation step flow of a device fault detection method based on vibration images provided by an embodiment of the present invention; as follows: Figure 2 As shown, the method steps may specifically include:

[0086] Step 201: Acquire vibration images of the equipment.

[0087] For details of this step, please refer to step 101 above. This embodiment will not repeat the details here.

[0088] In one optional embodiment, step 201 may further include:

[0089] Sub-step 2011: Using a sample sampling function, adjust the number of samples in the vibration image so that the ratio of the number of abnormal vibration image samples to the number of normal vibration image samples is equal to a first preset ratio threshold.

[0090] It is generally believed that malfunctions or abnormal phenomena in industrial equipment are low-probability events, and during the vast majority of daily operation, the vibration images generated by industrial equipment are normal. However, for neural network models, accurately extracting and classifying the image feature information of abnormal vibration waveforms in vibration images requires a large amount of real-world data to continuously train the model and optimize the computational parameters of the classification convolutional neural network.

[0091] In practical applications, a high-precision camera typically acquires approximately 10,000 normal vibration images of an industrial motor per unit time, while only about 100 images exhibit abnormal vibrations. Using this as the computational target for a classification convolutional neural network contradicts the intended purpose of anomaly detection. Therefore, it is necessary to adjust the ratio of normal to abnormal images in the input image samples so that abnormal images become the primary computational target in the processing samples of the classification convolutional neural network.

[0092] Specifically, a sample sampling function is used to extract all the captured vibration images to ensure that the ratio of abnormal samples to normal samples in a batch of image samples is approximately 10:1.

[0093] Step 202: Extract features from the vibration image to obtain the frequency amplitude histogram of the vibration image.

[0094] For details of this step, please refer to step 102 above. This embodiment will not repeat the details here.

[0095] Step 203: Fuse the image channel of the vibration image with the image channel of the frequency amplitude histogram to obtain the target fused image.

[0096] Before performing classification convolutional neural network calculations, it is necessary to fuse the acquired vibration images and frequency amplitude histograms, and further combine the image features of the original image and the histogram.

[0097] Optionally, step 203 may specifically include:

[0098] Sub-step 2031: Based on the image channel information of the vibration image, obtain a collection of vibration image matrices including at least one vibration image matrix, wherein a single vibration image matrix includes all content information of one of the image channels of the vibration image.

[0099] First, the channel information contained in the acquired vibration images is obtained. Image channels are used to characterize the proportional relationship of a basic image element that constitutes the image within the image.

[0100] For example, for a primary color image, the three primary color channels RGB—red, green, and blue—that constitute the complete image content can be referenced. Combining them together can restore all the content information in the original image. Within each primary color channel, the proportion of the current element in the image is represented; for example, if the original image has a large number of red pixels, this will be reflected in the monochrome channel as a larger number of red pixels.

[0101] In this embodiment of the invention, for the vibration image generated by the captured vibration of industrial equipment, the channel information in the image is represented by an image matrix; each element (number) in the matrix represents the channel information contained in the pixel at the current position in the image. A single image channel corresponds to a single image matrix, and the current image can be divided into several image channels, which correspond to several image matrices. All image matrices constitute a set of vibration image matrices.

[0102] Sub-step 2032: Based on the image channel information of the frequency amplitude histogram, obtain a set of frequency amplitude histogram matrices including at least one frequency amplitude histogram matrix, wherein a single frequency amplitude histogram matrix includes all content information of one of the image channels of the frequency amplitude histogram.

[0103] Similarly, for the frequency amplitude histogram, the main content displayed in its image channel is the two-dimensional statistical information of the frequency amplitude in the vibration image, and the collection of frequency amplitude histogram matrices can be obtained based on this statistical information.

[0104] Sub-step 2033: Fuse the vibration image matrix set with the image channels in the frequency amplitude histogram matrix set to obtain a fused matrix set; wherein, the fused matrix set includes all content information of the image channels of the vibration images and the image channels of the frequency amplitude histogram.

[0105] The vibration image and the frequency-amplitude histogram are merged into a fused matrix set. For example, the original vibration image can be divided into an image matrix consisting of two sub-image channels corresponding to frequency and amplitude respectively, and the frequency-amplitude histogram can be divided into an image matrix consisting of one sub-image channel. The final matrix fusion result is 2 + 1 = 3 image matrices.

[0106] By fusing the image channels of different images side by side, a fused image containing all the image feature information of multiple images can be obtained.

[0107] Sub-step 2034: Generate the target fused image based on the fusion matrix set.

[0108] The vibration image and all image matrices of the frequency amplitude histogram are merged, and a new target fused image is generated based on the merged fusion matrix set.

[0109] As can be seen from sub-step 2031, the original image can be split into different image channels. By superimposing and merging all the image channels, all the information in the original image can be restored.

[0110] Similarly, the fused matrix set includes all image channel information of the original vibration image and frequency amplitude histogram. By superimposing all image matrices according to the fused matrix set, an image that simultaneously possesses all image features of the original vibration image and frequency amplitude histogram can be obtained.

[0111] Step 204: Based on the target fused image and the preset classification convolutional neural network model, obtain the classification result for the target fused image.

[0112] For details of this step, please refer to step 103 above. This embodiment will not repeat the details here.

[0113] Sub-step 2041: Evaluate and score the target fused image using a classifier in a classification convolutional neural network model.

[0114] The target fused image obtained after restoration by the fusion matrix is ​​then classified using the MobileNet V3 network model employed in this embodiment of the invention, utilizing a multi-head classification method. This allows the model network to capture richer feature information. Each selection head is an independent classifier with different selection and classification calculation parameters. The classification result for the two-dimensional image is the specific score generated by the classification algorithm.

[0115] Sub-step 2042: Obtain the arithmetic mean of all scores from the classifier.

[0116] The final classification result can be obtained by summing the scores of multiple classification heads in the model using an arithmetic mean.

[0117] For example, for a classification convolutional neural network model using a 5-classifier head, the scores are 0.3, 0.5, 0.5, 0.7, 0.6, and 0.4 respectively. The final arithmetic mean classification result is (0.3+0.5+0.5+0.7+0.6+0.4) / 5=0.6.

[0118] Sub-step 2043: Use the arithmetic mean as the classification result.

[0119] After obtaining the arithmetic mean, it is used as the final classification result. By comparing the magnitude of the final result with the preset judgment threshold, it is determined whether there are abnormal factors in the vibration waveform of the equipment, thereby determining whether there is a fault or abnormality in the industrial equipment.

[0120] Step 205: Determine whether the equipment is faulty based on the classification results.

[0121] For details of this step, please refer to step 104 above. This embodiment will not repeat the details here.

[0122] Sub-step 2051: If the arithmetic mean of the target fused image in the classification result is greater than or equal to a first preset threshold, then it is determined that the device is faulty.

[0123] In this embodiment of the invention, if the final calculated arithmetic mean is greater than or equal to 0.5, the current industrial equipment can be considered to have an malfunction. In practical applications, maintenance personnel can modify the specific value of the fault malfunction judgment threshold according to different application scenarios, and this embodiment of the invention does not impose any restrictions here.

[0124] Sub-step 2052: If the arithmetic mean of the target fused image is less than the first preset threshold, then it is determined that the device is not faulty.

[0125] If the final arithmetic mean is less than 0.5, it can be considered that there is no equipment malfunction in the current industrial equipment, and the characteristic waveforms contained in the captured vibration image are all vibration waveforms generated under normal working conditions.

[0126] Step 206: Train the classifier of the classification convolutional neural network model using a loss function to optimize the computational parameters used by the classifier when classifying the target fused image.

[0127] Reference Figure 3 This diagram illustrates the complete implementation logic of a device fault detection method based on vibration images provided by an embodiment of the present invention; as follows: Figure 3 As shown, it includes:

[0128] First, vibration images of the device under test are acquired using a high-precision camera, and a two-dimensional histogram of frequency and amplitude is extracted from the vibration images. Then, the vibration images and histograms are fused by channel fusion, and the fused result is fed into an improved Mobile NetV3 classification convolutional network model to obtain the classification result. Finally, the network model is trained using a weighted sigmoid loss function.

[0129] The Sigmoid loss function, also known as the Logistic function, is used for the output of the hidden layer. Its value range is (0,1). It can map a real number to the interval (0,1). It can be used for binary classification and performs well when the features are complex or not particularly large.

[0130] In the equipment fault detection method based on vibration images provided in this embodiment of the invention, since there are more normal samples and fewer abnormal samples in industrial equipment, a weighted loss function is sampled during training to increase the weight of abnormal samples. In order to enable the Mobile Net V3 classification network model used in the classification process to extract and classify the feature information of vibration images more accurately, a sigmoid function is introduced after each classification judgment to optimize the calculation parameters in the model.

[0131] Specifically, the sigmoid function is a type of logistic loss function, suitable for binary classification tasks, and requires one of the assumptions to be met: the data follows a Bernoulli distribution. θ (x) represents the probability that the sample is predicted to be of the positive class, 1-h θ (x) represents the probability of predicting a sample as the negative class. The entire model can be expressed as: h θ (x, θ) = p, where, Where θ is the weight value, x is the input variable, and the exponential function represents the model's output, the final expression for logistic regression is obtained. The loss function of logistic regression is its maximum likelihood function.

[0132] In the process of training the model using the Sigmoid loss function in this embodiment of the invention, the weight of abnormal samples is set to 10, and the weight of normal samples is set to 1.

[0133] In summary, the device fault detection method based on vibration images provided by this invention first acquires vibration images of the device through image acquisition. After feature extraction from the vibration images, a two-dimensional frequency-amplitude histogram is obtained, representing the vibration frequency and amplitude values. Based on a preset classification convolutional neural network model, the acquired vibration images and frequency-amplitude histogram are classified and calculated to obtain classification results. Finally, the device is judged to have abnormal faults based on the classification results. This method analyzes the vibration image features of the device using an improved classification convolutional neural network model, employing a non-contact information acquisition method to analyze device vibration, thus expanding the means of vibration signal detection for industrial equipment.

[0134] Reference Figure 4 This diagram illustrates the module composition of a device fault detection device based on vibration images provided in an embodiment of the present invention; as shown below. Figure 4 As shown, the device includes:

[0135] Image acquisition module 301 is used to acquire vibration images of the device;

[0136] Feature extraction module 302 is used to extract features from the vibration image to obtain a frequency amplitude histogram of the vibration image;

[0137] The image classification module 303 is used to obtain a classification result for the vibration image and the frequency amplitude histogram based on the vibration image, the frequency amplitude histogram and a preset classification convolutional neural network model;

[0138] The equipment fault determination module 304 is used to determine whether the equipment has a fault based on the classification results.

[0139] Optionally, the image classification module 303 further includes:

[0140] The image fusion acquisition submodule is used to fuse the image channels of the vibration image with the image channels of the frequency amplitude histogram to obtain a target fused image;

[0141] The classification algorithm execution submodule is used to classify the target fused image using a preset classification convolutional neural network model to obtain the classification result.

[0142] Optionally, the fused image acquisition submodule further includes:

[0143] The vibration image channel matrix acquisition unit is used to obtain a collection of vibration image matrices including at least one vibration image matrix based on the image channel information of the vibration image. Each vibration image matrix includes all content information of one image channel of the vibration image.

[0144] The frequency amplitude image channel matrix acquisition unit is used to obtain a set of frequency amplitude histogram matrices including at least one frequency amplitude histogram matrix based on the image channel information of the frequency amplitude histogram, wherein a single frequency amplitude histogram matrix includes all content information of the image channel of one of the frequency amplitude histograms;

[0145] A matrix fusion unit is used to fuse the image channels of the vibration image matrix set with the image channels of the frequency amplitude histogram matrix set to obtain a fused matrix set; wherein, the fused matrix set includes all content information of the image channels of the vibration image and the image channels of the frequency amplitude histogram;

[0146] A fused image generation unit is used to generate the target fused image based on the fusion matrix set.

[0147] Optionally, the classification algorithm execution submodule further includes:

[0148] The classification result acquisition unit is used to evaluate and score the target fused image by using the classifier in the classification convolutional neural network model;

[0149] The classification result averaging unit is used to obtain the arithmetic mean of all scores from the classifier;

[0150] The classification result generation unit is used to take the arithmetic mean as the classification result.

[0151] Optionally, the device further includes:

[0152] The sample ratio adjustment module is used to adjust the number of samples in the vibration image through a sample sampling function, so that the ratio of the number of abnormal vibration image samples to the number of normal vibration image samples in the vibration image is equal to a first preset ratio threshold.

[0153] Optionally, the device further includes:

[0154] The network training execution module is used to train the classifier of the classification convolutional neural network model through a loss function, and optimize the computational parameters used by the classifier when classifying target fused images.

[0155] In summary, the device for detecting equipment faults based on vibration images provided by this invention first acquires vibration images of the equipment through image acquisition. After feature extraction from the vibration images, a two-dimensional frequency-amplitude histogram of the vibration frequency and amplitude values ​​is obtained. Based on a preset classification convolutional neural network model, the acquired vibration images and frequency-amplitude histogram are classified and calculated to obtain classification results. Finally, the device is judged to have abnormal faults based on the classification results. By analyzing the characteristics of equipment vibration images using an improved classification convolutional neural network model, and employing a non-contact information acquisition method, the device analyzes equipment vibrations, thus expanding the means of vibration signal detection for industrial equipment.

[0156] Additionally, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the sample container unloading method described above.

[0157] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described sample container unloading method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0158] Figure 5 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0159] Reference Figure 5 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0160] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0161] Memory 604 is used to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0162] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0163] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0164] Audio component 610 is used to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) used to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0165] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0166] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0167] Communication component 616 facilitates wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0168] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the device fault detection method based on vibration images provided in this embodiment of the invention.

[0169] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0170] Figure 6This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. (Refer to...) Figure 6 The electronic device 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by a memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform a device fault detection method based on vibration images provided in embodiments of the present invention.

[0171] Electronic device 700 may also include a power supply component 726 configured to perform power management of electronic device 700, a wired or wireless network interface 750 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 758. Electronic device 700 may operate on an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0172] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0173] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for equipment fault detection based on vibration images, characterized in that, The method includes: Acquire vibration images of the equipment; Feature extraction is performed on the vibration image to obtain the frequency-amplitude histogram of the vibration image; Based on the vibration image, the frequency amplitude histogram, and a preset classification convolutional neural network model, classification results are obtained for the vibration image and the frequency amplitude histogram. Determine whether the equipment is faulty based on the classification results; The step of obtaining classification results for the vibration image and the frequency amplitude histogram based on the vibration image, the frequency amplitude histogram, and a preset classification convolutional neural network model includes: The image channels of the vibration image and the image channels of the frequency amplitude histogram are fused to obtain the target fused image; The target fused image is classified using a pre-defined classification convolutional neural network model to obtain classification results.

2. The method according to claim 1, characterized in that, Both the vibration image and the frequency amplitude histogram are two-dimensional histograms; the process of fusing the image channels of the vibration image and the image channels of the frequency amplitude histogram to obtain a target fused image includes: Based on the image channel information of the vibration image, a collection of vibration image matrices is obtained, including at least one vibration image matrix, wherein a single vibration image matrix includes all content information of one of the image channels of the vibration image; Based on the image channel information of the frequency amplitude histogram, a set of frequency amplitude histogram matrices is obtained, including at least one frequency amplitude histogram matrix. Each frequency amplitude histogram matrix includes all content information of one of the image channels of the frequency amplitude histogram. The vibration image matrix set is fused with the image channels of the frequency amplitude histogram matrix set to obtain a fused matrix set; wherein, the fused matrix set includes all content information of the image channels of the vibration images and the image channels of the frequency amplitude histogram. The target fused image is generated based on the fusion matrix set.

3. The method according to claim 1, characterized in that, The classification convolutional neural network model includes at least one classifier; the step of classifying the target fused image using a preset classification convolutional neural network model to obtain a classification result includes: The target fused image is evaluated and scored by a classifier in a classification convolutional neural network model. Obtain the arithmetic mean of all scores from the classifier; The arithmetic mean is used as the classification result.

4. The method according to claim 3, characterized in that, Determining whether the equipment is faulty based on the classification result includes: If the arithmetic mean of the target fused image in the classification result is greater than or equal to a first preset threshold, then it is determined that the device is faulty; If the arithmetic mean of the target fused image is less than a first preset threshold, then it is determined that the device is not faulty.

5. The method according to claim 1, characterized in that, The vibration images include: normal vibration image samples and abnormal vibration image samples; Before performing feature extraction on the vibration image to obtain the frequency-amplitude histogram of the vibration image, the method further includes: The number of samples in the vibration image is adjusted by a sample sampling function so that the ratio of the number of abnormal vibration image samples to the number of normal vibration image samples is equal to a first preset ratio threshold.

6. The method according to claim 5, characterized in that, After classifying the target fused image using a preset classification convolutional neural network model and obtaining the classification result, the method further includes: The classifier of the classification convolutional neural network model is trained using a loss function to optimize the computational parameters used by the classifier when classifying target fused images.

7. A device for detecting equipment faults based on vibration images, characterized in that, The device includes: The image acquisition module is used to acquire vibration images of the device; The feature extraction module is used to extract features from the vibration image to obtain the frequency amplitude histogram of the vibration image; The image classification module is used to obtain classification results for the vibration image and the frequency amplitude histogram based on the vibration image, the frequency amplitude histogram, and a preset classification convolutional neural network model. The equipment fault determination module is used to determine whether the equipment has a fault based on the classification results. The image classification module further includes: The image fusion acquisition submodule is used to fuse the image channels of the vibration image with the image channels of the frequency amplitude histogram to obtain a target fused image; The classification algorithm execution submodule is used to classify the target fused image using a preset classification convolutional neural network model to obtain the classification result.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, When the instructions in the readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 6.

Images