System and method for monitoring conditions of a turn-around device based on image analysis

Abstract

The present application belongs to the technical field of monitoring the condition of transfer equipment, and discloses a system and method for monitoring the condition of transfer equipment based on image analysis, which comprises the following steps: collecting thermal imaging images of the transfer equipment and forming a data set; performing sharpness enhancement processing on the thermal imaging images; when an infrared thermal imager captures thermal imaging images of the transfer equipment, the thermal imaging images are subjected to ambient temperature elimination, and the thermal imaging images after ambient temperature elimination are obtained and marked as the eliminated ambient temperature thermal imaging images; the eliminated ambient temperature thermal imaging images are input into a trained defect type recognition model, and the thermal imaging images with defects are obtained; the thermal imaging images with defects are input into a target detection model, and the defect positions of the thermal imaging images with defects are obtained; and the accuracy of subsequent equipment defect and non-defect label classification is improved.

Description

Technical Field

[0001] This invention relates to the field of rotating equipment condition monitoring technology, and more specifically, to a rotating equipment condition monitoring system and method based on image analysis. Background Technology

[0002] In the prior art, for example, Chinese Patent No. CN113551775B discloses a method and system for online monitoring and alarming of equipment faults based on infrared thermal imaging, including the following steps: Step S1, under the normal operating conditions of the equipment at a reference ambient temperature, the infrared thermal imaging device takes multiple pictures of the equipment to complete the data acquisition of reference images; Step S2, based on the changes in brightness in the infrared thermal image, the bright areas, bright areas, and dark areas are identified, and the center point coordinates, brightness, and area of ​​each bright area are calculated to form calibration data and stored; Step S3, when monitoring the operating status of the equipment, the infrared thermal imaging device in Step S1 continuously takes pictures of the equipment at the same orientation and shooting angle at a preset frequency to acquire real-time images; Step S4, the real-time images are processed and identified, and the bright areas of the images are compared with the calibration data to calculate the offset. When the offset is greater than a threshold, an alarm is triggered. This invention can realize the monitoring of rotating equipment through non-contact measurement.

[0003] The following problems still exist in the existing technology:

[0004] 1. For the acquired image data, some images with uneven lighting were not enhanced. Noise and contrast are factors that affect image quality, resulting in blurriness and making it easy for the equipment to make mistakes in recognition and classification.

[0005] 2. When infrared thermal imagers capture images of rotating equipment, the temperature of the outside environment, such as summer and winter, can affect the thermal image values. In summer, the outside temperature is high, which can cause a large difference between the thermal image values ​​captured and those captured at normal temperatures, making it easy to make mistakes in judgment.

[0006] In view of this, the present invention proposes an image analysis-based system and method for monitoring the condition of rotating equipment to solve the above problems. Summary of the Invention

[0007] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution:

[0008] Image analysis-based methods for monitoring the condition of rotating equipment include:

[0009] Acquire thermal imaging images of the transfer equipment and construct a dataset;

[0010] The thermal imaging image is then processed to enhance its sharpness.

[0011] When an infrared thermal imager captures thermal images of a rotating device, it performs ambient temperature removal on the thermal images to obtain an ambient temperature-removed thermal image, which is then marked as an ambient temperature-removed thermal image.

[0012] The thermal imaging image of the ambient temperature is input into the trained defect type recognition model to obtain the thermal imaging image of the defect.

[0013] The defective thermal imaging image is input into the target detection model to obtain the numerical code of the defective part of the thermal imaging image and determine the defective part.

[0014] Furthermore, the defective parts of the rotating equipment are identified, the first hazard level is determined based on the defective parts, the standard operating temperature of the corresponding defective parts is obtained, and the actual operating temperature is obtained based on the thermal imaging image of the defective parts. The second hazard level is determined based on the actual operating temperature and the standard operating temperature. Based on the specific types of the first and second hazard levels that the defective parts simultaneously possess, the priority warning level of the defective parts is determined.

[0015] Furthermore, the method for enhancing the sharpness of the thermal imaging image includes:

[0016] S1. Adjust the thermal imaging image size to a uniform size;

[0017] S2. Normalize the pixel values ​​of the thermal imaging image;

[0018] S3. Based on the normalized thermal imaging image, ResNet-50 in the deep residual network model of the neural network model is used to enhance the clarity of the thermal imaging image.

[0019] Furthermore, in S3, the method for setting up the deep residual network model includes:

[0020] S301, The input layer is used to input the normalized thermal imaging image into the deep residual network;

[0021] S302. The convolutional layer performs a convolution operation on the input thermal imaging image using a 7×7 convolutional kernel, with a stride of 2 and an output channel count of 64.

[0022] S303, the batch normalization layer is used to perform batch normalization on the thermal imaging images output by the convolutional layer;

[0023] S304, the activation function layer activates the batch-normalized thermal imaging image in S303 by using the ReLU activation function;

[0024] S305. The max pooling layer performs max pooling operation using a 3×3 pooling kernel, with a step size of 2, to reduce the size of the feature map, which is the thermal imaging image activated in S304.

[0025] S306. The network is constructed using H residual blocks, where H is an integer greater than 1. Each residual block consists of two convolutional layers and one skip connection. The first convolutional layer has a kernel size of 1×1 and 64 output channels. The second convolutional layer has a kernel size of 3×3 and 64 output channels. The skip connection directly adds the thermal imaging image as input to the output of the residual block.

[0026] S307. Repeat the residual block three times in sequence. Repeat the first residual block and set the number of output channels of the convolution kernels of the first and second convolutional layers to 128. Repeat the second residual block and set the number of output channels of the convolution kernels of the first and second convolutional layers to 256. Repeat the third residual block and set the number of output channels of the convolution kernels of the first and second convolutional layers to 512.

[0027] S308, the average pooling layer reduces the dimensionality of the feature map by using global average pooling;

[0028] S309, the output layer, is used to output the thermal imaging image after sharpness enhancement processing.

[0029] Furthermore, methods for eliminating ambient temperature in thermal imaging images include:

[0030] S10. Create a difference image by subtracting the grayscale values ​​of the original thermal imaging image from the sharpness-enhanced thermal imaging image to obtain the difference image.

[0031] S20. Detect and label temperature change regions in the differential image using a threshold segmentation algorithm;

[0032] S30. Use the marked temperature change area to adjust the corresponding area in the original thermal imaging image to eliminate the influence of ambient temperature in the thermal imaging image, and obtain the thermal imaging image after ambient temperature elimination, which is marked as the ambient temperature eliminated thermal imaging image.

[0033] Furthermore, the threshold segmentation algorithm is the maximum inter-class variance method. Methods for detecting and labeling temperature change regions in difference images include:

[0034] The difference image is set to a specific size and contains several gray levels. Based on the gray level probability distribution, a gray level threshold is selected, and all pixels of the difference image are divided into two sets according to their gray level values. The first set contains pixels whose gray level values ​​are in the range from the lowest level to the threshold, and the second set contains pixels whose gray level values ​​are in the range from the threshold plus one to the highest level, and the highest level is greater than the threshold plus one.

[0035] Calculate the proportion of pixels in the first set in the entire image, which is the sum of the gray level probabilities of all pixels in the first set; calculate the average gray level of the first set, which is the sum of the products of the gray level values ​​of all pixels in the set and their respective gray level probabilities, and then divide by the proportion of pixels in the first set.

[0036] Calculate the proportion of pixels in the second set in the entire image, which is the sum of the gray level probabilities of all pixels in the set, and the sum of this proportion and the pixel proportion of the first set is 1; calculate the average gray level of the second set, which is the sum of the products of the gray level values ​​of all pixels in the set and their respective gray level probabilities, and then divide by the pixel proportion of the second set.

[0037] Calculate the cumulative mean corresponding to the grayscale threshold, which is the sum of the products of the grayscale values ​​of all pixels in the first set and their respective grayscale probabilities; calculate the average grayscale of the entire difference image, which is the sum of the products of the grayscale values ​​of all pixels and their respective grayscale probabilities, and the product of the pixel proportion of the first set and the average grayscale of the first set, plus the product of the pixel proportion of the second set and the average grayscale of the second set, equals the average grayscale of the entire image.

[0038] The inter-class variance is defined as: the pixel percentage of the first set multiplied by the square of the difference between the average gray level of the first set and the average gray level of the whole image, plus the pixel percentage of the second set multiplied by the square of the difference between the average gray level of the second set and the average gray level of the whole image.

[0039] Thresholding is performed on the difference image using the thresholding function provided by OpenCV in Python. The processing mode of the function is set to the mode corresponding to the maximum inter-class variance. The pixel ratio and average gray level of the two sets are calculated respectively. All gray level values ​​of the entire difference image are traversed to find the gray level threshold that maximizes the inter-class variance, and then the temperature change region is detected and marked.

[0040] Furthermore, the steps to eliminate the influence of ambient temperature in thermal imaging images include:

[0041] S31. Based on the marked temperature change area, determine the original thermal imaging image area that needs to be adjusted, calculate the difference between the gray value of each pixel in the original thermal imaging image and the gray value of its neighboring pixels, and adjust the pixel value of the corresponding area in the original thermal imaging image according to the difference.

[0042] S32. By adjusting the corresponding area in the original thermal imaging image, the gray value of the corresponding area pixels is adjusted to the same level as the surrounding environment to obtain the thermal imaging image after the ambient temperature is eliminated, and it is marked as the thermal imaging image after the ambient temperature is eliminated.

[0043] S33. Save or display the thermal imaging image after eliminating ambient temperature.

[0044] Furthermore, the training methods for the defect type identification model include:

[0045] The thermal imaging images with ambient temperature removed were divided into training and testing sets; thermal imaging images classified as defective were labeled as 0, and thermal imaging images classified as defect-free were labeled as 1.

[0046] The thermal imaging images of the environment with removed temperature from the training set are used as input data for the neural network model, and the annotations in the thermal imaging images of the environment with removed temperature are used as output data for the neural network model. The neural network model is trained to obtain an initial neural network model. The initial neural network model is validated using a test set, and the initial neural network model that meets the preset test accuracy is output as the defect type identification model. The neural network model is one of CNN and RNN.

[0047] Furthermore, the training methods for object detection models include:

[0048] The defective thermal imaging images are converted into a set of corresponding feature vectors, which are used as input to the object detection model. The object detection model outputs the predicted defect location numerical index of the defective thermal imaging images and uses the actual defect location numerical index corresponding to each set of defective thermal imaging images as the prediction target of the object detection model. The training objective is to minimize the loss function value of the object detection model. Training stops when the loss function value of the object detection model is less than or equal to the preset target loss value. The object detection model is the MobileNet_SSD object detection model.

[0049] Furthermore, the specific types of the first hazard level include the first severe mark, the first moderate mark, and the first minor mark; the specific types of the second hazard level include the second severe mark, the second moderate mark, and the second minor mark.

[0050] Methods for determining the priority warning level of a defect location based on the specific types of both the primary and secondary hazard levels it simultaneously possesses include:

[0051] The warning levels include the first warning order, the second warning order, and the third warning order, with the priority of the warning order decreasing progressively from the first warning order to the third warning order.

[0052] Case A is defined as the defect location having both the first severe mark and the second severe mark simultaneously; Case B is defined as the defect location having both the first severe mark and the second moderate mark simultaneously; Case C is defined as the defect location having both the first moderate mark and the second severe mark simultaneously; Case D is defined as the defect location having any of the following conditions: Case A, Case B, and Case C simultaneously.

[0053] If the defective part has condition A, a first warning instruction is generated; if the defective part has condition B or C, a second warning instruction is generated; if the defective part has condition D, a third warning instruction is generated.

[0054] Furthermore, the method for determining the second hazard level based on the actual working temperature and the standard working temperature includes: firstly, calculating the temperature difference between the actual working temperature and the standard working temperature, comparing the temperature difference with the corresponding preset range temperature threshold of the defect location, and if the temperature difference is less than the minimum value of the preset range temperature threshold, then marking it as a second minor hazard level.

[0055] If the temperature difference is greater than or equal to the minimum value of the preset temperature threshold and less than the maximum value of the preset temperature threshold, it is marked as the second medium mark.

[0056] If the temperature difference is greater than or equal to the maximum value of the preset temperature threshold, it is marked as the second most serious.

[0057] The image analysis-based transfer equipment condition monitoring system, used to implement the image analysis-based transfer equipment condition monitoring method, includes:

[0058] The image data acquisition module acquires thermal imaging images of the transfer equipment and forms a dataset;

[0059] The image enhancement algorithm module performs sharpness enhancement processing on the thermal imaging image;

[0060] The ambient temperature elimination module performs ambient temperature elimination on the thermal imaging image when the infrared thermal imager captures thermal imaging images of the rotating equipment, and obtains the ambient temperature eliminated thermal imaging image, which is marked as the ambient temperature eliminated thermal imaging image.

[0061] The defect type identification module inputs the eliminated ambient temperature thermal imaging image into the trained defect type identification model to obtain a defective thermal imaging image.

[0062] The equipment target detection module inputs the defective thermal imaging image into the target detection model, obtains the numerical number of the defective part of the thermal imaging image, and determines the defective part.

[0063] The technical effects and advantages of the image analysis-based monitoring system and method for monitoring the condition of rotating equipment of this invention are as follows:

[0064] By incorporating an image enhancement algorithm module, the acquired thermal imaging images are enhanced in clarity and noise is processed. Leveraging the deep neural network with its multiple skip connections, it can learn more complex mapping relationships and prevent gradient vanishing or exploding. In contrast enhancement, the deep residual network improves the dynamic range and contrast of the image, making it clearer and enhancing its visual quality. This addresses the problem of blurry images caused by the lack of enhancement processing (i.e., noise and temperature reduction preprocessing) in images with uneven lighting, reducing the classification error of transfer equipment. It also improves the accuracy of subsequent defective and non-defective label classification, significantly enhancing the accuracy and efficiency of monitoring, maintenance, and management of transfer equipment.

[0065] A Poisson fusion algorithm is used to merge thermal imaging images after temperature-induced degradation with enhanced thermal imaging images. The existing Mobilenet_SSD target detection network is then used to train and detect defective thermal imaging images, identifying defective locations in the equipment. Based on the hazardous location and temperature difference, a priority warning level for each defect is determined, allowing maintenance personnel to prioritize maintenance of equipment with higher risks. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the image analysis-based equipment condition monitoring system of the present invention;

[0067] Figure 2 This is a flowchart of the method for enhancing the sharpness of thermal imaging images according to the present invention;

[0068] Figure 3 This is a flowchart of the method for eliminating ambient temperature from thermal imaging images according to the present invention.

[0069] Figure 4 This is a flowchart of the method for eliminating the influence of ambient temperature in thermal imaging images according to the present invention;

[0070] Figure 5 This is a flowchart of the image analysis-based equipment condition monitoring method of the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Example 1

[0073] Please see Figure 1 As shown, this embodiment provides an image analysis-based equipment condition monitoring system, including an image data acquisition module, an image enhancement algorithm module, an ambient temperature elimination module, a defect type identification module, an equipment target detection module, and an early warning analysis module. The modules are connected by wired and / or wireless means.

[0074] The image data acquisition module is used to acquire thermal imaging images of the transfer equipment and form a dataset; and to send the acquired thermal imaging images to the image enhancement algorithm module.

[0075] The thermal imaging images are categorized into two types: defective and non-defective. The thermal imaging images are acquired in real time by using an infrared thermal imager mounted on a drone for autonomous patrol.

[0076] The image enhancement algorithm module is used to enhance the sharpness of the thermal imaging image; it can improve the visual effect of the image and improve the image quality.

[0077] Please see Figure 2 As shown, the method for enhancing the sharpness of the thermal imaging image is as follows:

[0078] S1. First, adjust the size of the thermal imaging image to a uniform size. In convolutional neural networks, due to the fully connected layers, the size of the input thermal imaging image must be fixed. The thermal imaging image can be scaled directly by using the OpenCV resize function. The NTER AREA (local pixel resampling) algorithm is selected as the parameter in the function to achieve better results (avoiding ripples). For example, the shape of the resized image is (256, 256, 3).

[0079] S2. Normalize the pixel values ​​of the thermal imaging image. Due to differences in acquisition and imaging factors from different cameras, identical images may exhibit inconsistencies in grayscale information. Image normalization is an image conversion method that reduces or even eliminates grayscale inconsistencies while preserving valuable grayscale differences. Image normalization is more conducive to automatic computer analysis and processing. Image normalization involves a series of transformations to convert the original image to be processed into a corresponding standard form.

[0080] In step S2, the method for normalizing the pixel values ​​of the thermal imaging image includes:

[0081] Using min-max normalization, the formula is:

[0082]

[0083] In the formula, X normalizationdenoted as , where is the normalized pixel value of the thermal imaging image; x is the unnormalized pixel value of the thermal imaging image. By traversing all pixel values ​​of the thermal imaging image, the specific values ​​of Min and Max are determined, where Min is the minimum value of all pixel values ​​traversed, and Max is the maximum value of all pixel values ​​traversed. This method is applicable to most types of data and can better maintain the original data distribution structure.

[0084] S3. Based on the normalized thermal imaging image, the ResNet-50 deep residual network model of the neural network is used to enhance the image sharpness. The deep residual network has a deep neural network with multiple skip connections, which can learn more complex mapping relationships and prevent gradient vanishing or exploding. In contrast enhancement, the deep residual network can improve the dynamic range and contrast of the image, making the image clearer.

[0085] The core idea of ​​this model is to solve the vanishing and exploding gradient problems in deep neural networks by introducing residual blocks, making the network deeper and easier to train. Specifically, a shortcut connection is added to the traditional convolutional neural network, allowing the network to directly learn the residuals, thus making training easier. The deep residual network model is preferably a neural network with 50-101 layers.

[0086] In step S3, the method for setting up the deep residual network model includes:

[0087] S301, Input Layer: Input the normalized thermal imaging image into the deep residual network;

[0088] S302, Convolutional Layer: The input thermal imaging image is convolved using a 7×7 convolutional kernel, with a stride of 2 and an output channel count of 64.

[0089] S303, Batch Normalization Layer: Performs batch normalization on the thermal imaging images output by the convolutional layer to improve the convergence speed of the network.

[0090] S304, Activation Function Layer: The ReLU activation function is used to activate the batch normalized thermal imaging images in step S303.

[0091] S305, Max Pooling Layer: Max pooling is performed using a 3×3 pooling kernel with a step size of 2 to reduce the size of the feature map, which is the thermal imaging image activated in step S304.

[0092] S306, Residual Blocks: The network is constructed using H residual blocks, where H is an integer greater than 1. Each residual block consists of two convolutional layers and one skip connection. The first convolutional layer has a kernel size of 1×1 and 64 output channels. The second convolutional layer has a kernel size of 3×3 and 64 output channels. The skip connection directly adds the thermal imaging image as input to the output of the residual block.

[0093] S307. Reusing the residual block: Reusing the residual block three times in sequence. When reusing the first residual block, the number of output channels of the convolution kernels of the first and second convolutional layers is set to 128. When reusing the second residual block, the number of output channels of the convolution kernels of the first and second convolutional layers is set to 256. When reusing the third residual block, the number of output channels of the convolution kernels of the first and second convolutional layers is set to 512.

[0094] S308, Average Pooling Layer: Uses global average pooling to reduce the dimensionality of the feature map.

[0095] S309, Output layer: Thermal imaging image after sharpness enhancement processing.

[0096] The normalized thermal imaging image is enhanced using a deep residual network model to obtain a sharpness-enhanced thermal imaging image.

[0097] The ambient temperature elimination module is mainly used to eliminate the influence of external temperature when infrared thermal imagers capture thermal images of the transfer equipment, which may lead to misjudgments of subsequent transfer equipment faults. Therefore, it is necessary to eliminate the ambient temperature in the thermal image.

[0098] Please see Figure 3 As shown, the method for eliminating ambient temperature in thermal imaging images is as follows:

[0099] S10. Create a difference image by subtracting the grayscale values ​​of the original thermal imaging image from the sharpness-enhanced thermal imaging image to obtain the difference image. Assuming that a thermal imaging image has undergone sharpness enhancement, this process usually enhances the contrast of the thermal imaging image, making temperature differences more obvious. That is, if an object is hotter or colder than its surroundings, sharpness enhancement will make the edges of the object sharper and the difference from its surroundings greater.

[0100] Then, by subtracting the enhanced thermal image from the original thermal image, the result of the subtraction will show the areas of temperature difference between the two thermal images. Because the temperature difference between the target object (transfer equipment) and the surrounding environment has been enhanced in the enhanced thermal image, the differences in these areas will be more obvious after the subtraction.

[0101] S20. Detect and label temperature change regions in the difference image. In the difference image, temperature change regions are presented in a specific way. For example, if the temperature of the target object is higher than the surrounding environment, then the pixel value of this region will be greater than the pixel value of the surrounding environment; conversely, if the temperature of the target object is lower than the surrounding environment, then the pixel value of this region will be less than the pixel value of the surrounding environment. This temperature difference is called a "temperature change region". This can be achieved through a threshold segmentation algorithm, namely the Otsu algorithm, also known as the maximum inter-class variance method. The maximum inter-class variance method (Otsu algorithm) includes the following:

[0102] S21. Assume the difference image is M×N in size and has Q gray levels, where p represents the gray level probability function (p is an existing gray level probability function). i Let i represent the components of the normalized histogram of the difference image, i represent the gray value, and select a gray threshold T(k) [k represents the gray level of T, 0 < k < Q] to divide the gray values ​​of the difference image into two sets D1 (gray values ​​in the interval [0, k]) and D2 (gray values ​​in the interval [k+1, Q-1]), where Q > k+1.

[0103] The probability P1(k) that a pixel value is assigned to D1 is:

[0104]

[0105] The average grayscale value m1(k) of the pixel value assigned to D1:

[0106]

[0107] The probability P2(k) that a pixel value is assigned to D2 is:

[0108]

[0109] The average pixel grayscale value m2(k) assigned to D2 by the pixel value:

[0110]

[0111] The cumulative mean m(k) of gray level k is:

[0112] P1(k) + P2(k) = 1;

[0113] The average gray level m of the entire difference image F for:

[0114] P1(k)m1(k)+P2(k)m2(k)=m F ;

[0115] Define inter-class variance for:

[0116]

[0117] S22. The `cv.threshold` function provided by OpenCV in Python can be used to perform thresholding on the difference image. Set the parameter `type` of the `cv.threshold` function to `cv.THRESH_OTSU`, calculate the probabilities P1(k) and P2(k) in sets D1 and D2 respectively, and the average gray values ​​m1(k) and m2(k). Iterate through the gray values ​​of the entire difference image to reduce the inter-class variance. To reach the maximum, the area of ​​temperature change is detected and marked.

[0118] S30. Use the marked temperature change areas to adjust the corresponding areas in the original thermal imaging image. (See [link to relevant documentation]) Figure 4 As shown: Local adjustments are made to the corresponding areas in the original thermal image. This adjustment process is based on the relationship between the gray value of each pixel and its neighboring pixels. The difference between the gray value of each pixel and the gray value of its neighboring pixels is calculated, and the pixel values ​​of the corresponding areas in the original thermal image are adjusted according to this difference.

[0119] The adjustment methods include: traversing each pixel in the original thermal image using a contrast enhancement adjustment method, increasing or decreasing its grayscale value based on the difference between its grayscale value and that of its neighboring pixels. For example, if the target object is hotter than its surroundings, the grayscale value of that area can be increased to make it more prominent.

[0120] By adjusting the corresponding areas in the original thermal imaging image, the influence of ambient temperature on the thermal imaging image can be eliminated. This is because these areas represent the temperature difference between the target object and the surrounding environment, rather than the temperature of the target object itself. By adjusting the grayscale values ​​of the pixels in the corresponding areas to the same level as the surrounding environment, interference caused by changes in ambient temperature can be eliminated, thereby highlighting the features of the target object in the image; this is marked as an ambient temperature-reduced thermal imaging image.

[0121] Save or display thermal imaging images that eliminate ambient temperature.

[0122] Image merging combines a thermal imaging image (with ambient temperature removed) with an enhanced thermal imaging image. This can be achieved using the Poisson fusion algorithm, an image processing technique that seamlessly merges two images, resulting in a more natural-looking merged image. Its principle is based on the Poisson equation, achieving a smooth transition through optimization. The Poisson fusion algorithm has wide applications in image synthesis, image restoration, and image editing. The core idea of ​​the Poisson fusion algorithm is to achieve seamless image fusion using the Poisson equation. The Poisson equation is a partial differential equation used to describe the equilibrium state of a physical system. In image processing, the Poisson equation is applied to gradient calculation and color transfer. In the Poisson fusion algorithm, a target region is first determined, and then the thermal imaging image (with ambient temperature removed, denoted as image A) is fused with the enhanced thermal imaging image (denoted as image B). This process can be divided into the following steps:

[0123] 1. Gradient Calculation: First, we need to calculate the gradients of images A and B. The gradient represents the direction and intensity of color changes in an image. We use the Sobel function from the OpenCV library in Python to calculate the gradients. By calculating the gradients, we can determine which regions in image B need to be blended.

[0124] 2. Boundary Conditions: After determining the regions in image B that need to be fused, boundary conditions need to be specified. These boundary conditions refer to the constraints that the pixel values ​​of the regions in image B must satisfy during the fusion process. These constraints can be determined by the pixel values ​​of the corresponding regions in image A. Specifically, the boundary color of the fusion region should be the same as the boundary color of the corresponding region in image A. The `cv2.findContours` function from the OpenCV library in Python can be used to find the contours of images A and B, determine the boundaries of the fusion region, and obtain these boundary pixel values.

[0125] 3. Solve the Poisson equation: Based on the results of the first two steps, the Poisson equation can be established and solved. The Poisson equation is a partial differential equation that describes how the pixel value of each pixel in an image changes. The general form of the Poisson equation is:

[0126]

[0127] Where u represents the function to be solved, and f represents a known function. It's the Laplacian operator. In the Poisson image fusion algorithm, u represents the fused image, and f represents the pixel value difference between image A and image B. In Poisson fusion, this equation is solved using a discretized form. For example, the equation can be discretized using the finite difference method, and then solved iteratively. Each iteration updates the pixel values ​​in the fused region.

[0128] 4. Image fusion: Finally, the pixel values ​​in image A are fused into the corresponding regions of image B based on the obtained Poisson equation. This method achieves seamless fusion of images A and B.

[0129] The Poisson fusion algorithm has the advantage of preserving image details and textures, and achieving smooth image transitions. It has wide applications in image compositing and restoration. For example, in image compositing, an object can be extracted from one image and fused into another, making the merged image look natural and realistic. In image restoration, a damaged area can be repaired to obtain a complete image.

[0130] The defect type identification module inputs the eliminated ambient temperature thermal imaging image into the trained defect type identification model to obtain a defective thermal imaging image.

[0131] Defect type identification model training methods include:

[0132] The thermal imaging images with ambient temperature removed were divided into training and testing sets; thermal imaging images classified as defective were labeled as "0", and thermal imaging images classified as defect-free were labeled as "1".

[0133] The thermal imaging images of the environment with removed temperature from the training set are used as input data for the neural network model, and the annotations in the thermal imaging images of the environment with removed temperature are used as output data for the neural network model. The neural network model is trained to obtain an initial neural network model. The initial neural network model is validated using a test set, and the initial neural network model that meets the preset test accuracy is output as the defect type identification model. The neural network model is one of CNN and RNN.

[0134] The equipment target detection module is used to input defective thermal imaging images into the target detection model to obtain the numerical number of the defective parts of the defective thermal imaging images and determine the defective parts.

[0135] Training methods for object detection models include:

[0136] The defective thermal imaging images are converted into a set of corresponding feature vectors, which are used as input to the target detection model. The target detection model outputs the predicted defect location numerical numbers of the defective thermal imaging images, uses the actual defect location numerical numbers corresponding to each set of defective thermal imaging images as the prediction target of the target detection model, and minimizes the loss function value of the target detection model as the training objective. Training stops when the loss function value of the target detection model is less than or equal to the preset target loss value.

[0137] Furthermore, the early warning analysis module: acquires the defective parts of the transfer equipment, determines the first hazard level based on the defective parts, acquires the standard operating temperature of the corresponding defective parts, acquires the actual operating temperature of the defective parts based on the thermal imaging images of the defective parts, determines the second hazard level based on the actual operating temperature and the standard operating temperature, and determines the priority early warning level of the defective parts based on the specific types of the first hazard level and the second hazard level that the defective parts simultaneously possess.

[0138] The specific types of the first hazard level include the first severe mark, the first moderate mark, and the first minor mark; the specific types of the second hazard level include the second severe mark, the second moderate mark, and the second minor mark.

[0139] Methods for determining the priority warning level of a defect location based on the specific types of both the primary and secondary hazard levels it simultaneously possesses include:

[0140] The warning levels include the first warning order, the second warning order, and the third warning order.

[0141] Case A is defined as the defect location having both the first severe mark and the second severe mark simultaneously; Case B is defined as the defect location having both the first severe mark and the second moderate mark simultaneously; Case C is defined as the defect location having both the first moderate mark and the second severe mark simultaneously; and Case D is defined as the defect location having any of the following conditions: Case A, Case B, and Case C.

[0142] If the defective part has condition A, a first warning instruction is generated; if the defective part has condition B or C, a second warning instruction is generated; if the defective part has condition D, a third warning instruction is generated.

[0143] Among them, the priority warning level of the first warning order, the second warning order, and the third warning order gradually decreases.

[0144] Methods for determining the first hazard level based on the location of the defect include:

[0145] Identify rotating equipment such as electric motors, generators, etc. In this embodiment, the rotating equipment is an electric motor as an example. Defects in electric motors include problems such as poor contact, overload, and insulation defects.

[0146] The hazard level of overload is greater than that of poor contact, and the hazard level of poor contact is greater than that of insulation defect; overload is marked as the first severe hazard level; poor contact is marked as the first moderate hazard level; and insulation defect is marked as the first minor hazard level.

[0147] Methods for determining the second hazard level based on the actual operating temperature and the standard operating temperature include:

[0148] First, calculate the temperature difference between the actual working temperature and the standard working temperature. Then, compare the temperature difference with the corresponding preset range temperature threshold for the defect location. If the temperature difference is less than the minimum value of the preset range temperature threshold, it is marked as a second minor mark.

[0149] If the temperature difference is greater than or equal to the minimum value of the preset temperature threshold and less than the maximum value of the preset temperature threshold, it is marked as the second medium mark.

[0150] If the temperature difference is greater than or equal to the maximum value of the preset temperature threshold, it is marked as the second severe mark; the preset temperature threshold is determined by those skilled in the art based on a large number of experiments.

[0151] To facilitate understanding of the technical solution in this embodiment, the following examples are provided:

[0152] Overload: The motor overheats under prolonged or excessive load conditions, and thermal imaging can detect the high-temperature areas of the motor.

[0153] Slightly hot areas appeared on the surface of the motor, with the temperature rise being less than 10°C, which is considered slight.

[0154] The electric motor has a relatively large high-temperature range, with temperature rises between 10°C and 30°C, which is considered moderate.

[0155] Significantly high temperatures appear over a large area of ​​the motor surface, with temperatures exceeding 30°C, which may cause localized burnout or deformation, indicating a serious condition.

[0156] Poor contact: Loose, corroded, or burned connections in the motor can lead to poor contact and overheating. Thermal imaging can detect high-temperature areas at the connection points and determine the severity of the fault based on the degree of temperature rise and the size of the affected area.

[0157] Among them, the connection part showed a slight high temperature, which was less than 10°C and was considered slight.

[0158] There is a large area of ​​high temperature at the connection point, with the temperature rising between 10°C and 30°C, which is considered moderate.

[0159] Significantly high temperatures were observed at the connection points, with temperatures exceeding 30°C, leading to loose bolts, corrosion of connecting wires, or arc discharge, indicating a serious condition.

[0160] Insulation defects: Aging, damage, or cracking of the insulation material in an electric motor can lead to current leakage and localized heating. Thermal imaging can detect high-temperature areas in the insulation, and the degree and extent of temperature rise can be used to determine the severity of the insulation fault.

[0161] Among them, there was localized high temperature in the insulation part, but the temperature rise was within 10°C, which was considered minor.

[0162] A large area of ​​high temperature was observed in the insulation part, with the temperature rising between 10°C and 30°C, which is considered moderate.

[0163] Significantly high temperatures are observed in the insulation components, with temperatures exceeding 30°C, leading to damage or breakage of the insulation material, which is considered a serious condition.

[0164] The object detection model is the MobileNet_SSD object detection model, which combines the ideas of MobileNet and SSD (SingleShot MultiBox Detector). MobileNet is a lightweight convolutional neural network, while SSD is a neural network-based object detection method. The design idea of ​​the MobileNet_SSD model is to graft the SSD detector onto the MobileNet convolutional neural network. In this way, the MobileNet_SSD model can maintain high detection performance while having low computational complexity and memory consumption.

[0165] In the MobileNet_SSD model, MobileNet is first used to extract features from defective thermal imaging images, and then the SSD detector is used to detect objects based on the extracted features. The SSD detector employs a multi-box detection method, using a shared convolutional neural network to predict target boxes of different sizes and shapes. The MobileNet_SSD model is suitable for object detection tasks in mobile devices and embedded systems because it has low computational complexity and memory footprint. Furthermore, due to its neural network approach, the MobileNet_SSD model also exhibits good generalization performance and robustness. The constructed MobileNet_SSD object detection network is used to train and detect defective thermal imaging images to identify the faulty parts of the transferring equipment.

[0166] In this example, an image enhancement algorithm module is added to enhance the sharpness and reduce noise in the acquired thermal imaging images. The deep residual network, with its multiple skip connections, learns more complex mapping relationships and prevents gradient vanishing or exploding. In contrast enhancement, the deep residual network improves the dynamic range and contrast of the image, making it clearer and enhancing its visual quality. This solves the problem of blurry images caused by the lack of enhancement processing (i.e., noise and temperature removal preprocessing) in images with uneven lighting, reducing the possibility of classification errors in the identification of transfer equipment and affecting the subsequent labeling of abnormal and normal equipment. This provides significant accuracy and efficiency for the monitoring, maintenance, and management of subsequent transfer equipment. A Poisson fusion algorithm is used to merge the temperature-effect-free thermal imaging image with the enhanced thermal imaging image. The existing Mobilenet_SSD target detection network is used to train and detect defective thermal imaging images to obtain the defective parts of the transfer equipment. Based on the hazardous parts and temperature differences, the priority warning level of the defective parts is determined, which makes it easier for maintenance personnel to prioritize the maintenance of transfer equipment with greater hazards.

[0167] Example 2

[0168] Please see Figure 5 As shown, this embodiment provides a method for monitoring the condition of transfer equipment based on image analysis, including:

[0169] Acquire thermal imaging images of the transfer equipment and construct a dataset;

[0170] The thermal imaging image is then processed to enhance its sharpness.

[0171] When an infrared thermal imager captures thermal images of a rotating device, it performs ambient temperature removal on the thermal images to obtain an ambient temperature-removed thermal image, which is then marked as an ambient temperature-removed thermal image.

[0172] The thermal imaging image of the ambient temperature is input into the trained defect type recognition model to obtain the thermal imaging image of the defect.

[0173] The defective thermal imaging image is input into the target detection model to obtain the numerical code of the defective part of the thermal imaging image and determine the defective part.

[0174] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0175] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for monitoring the condition of rotating equipment based on image analysis, characterized in that, include: Acquire thermal imaging images of the transfer equipment and construct a dataset; The thermal imaging image is then processed to enhance its sharpness. When an infrared thermal imager captures thermal images of a rotating device, it performs ambient temperature removal on the thermal images to obtain an ambient temperature-removed thermal image, which is then marked as an ambient temperature-removed thermal image. The thermal imaging image of the ambient temperature is input into the trained defect type recognition model to obtain the thermal imaging image of the defect. The defective thermal imaging image is input into the target detection model to obtain the numerical code of the defective part of the thermal imaging image and determine the defective part. Methods for eliminating ambient temperature in thermal imaging images include: S10. Create a difference image by subtracting the grayscale values ​​of the original thermal imaging image from the sharpness-enhanced thermal imaging image to obtain the difference image. S20. Detect and label temperature change regions in the differential image using a threshold segmentation algorithm; S30. Use the marked temperature change area to adjust the corresponding area in the original thermal imaging image to eliminate the influence of ambient temperature in the thermal imaging image, and obtain the thermal imaging image after ambient temperature elimination, which is marked as the ambient temperature eliminated thermal imaging image. The steps to eliminate the influence of ambient temperature in thermal imaging images include: S31. Based on the marked temperature change area, determine the original thermal imaging image area that needs to be adjusted, calculate the difference between the gray value of each pixel in the original thermal imaging image and the gray value of its neighboring pixels, and adjust the pixel value of the corresponding area in the original thermal imaging image according to the difference. S32. By adjusting the corresponding area in the original thermal imaging image, the gray value of the corresponding area pixels is adjusted to the same level as the surrounding environment to obtain the thermal imaging image after the ambient temperature is eliminated, and it is marked as the thermal imaging image after the ambient temperature is eliminated. S33. Save or display the thermal imaging image after eliminating ambient temperature.

2. The image analysis-based method for monitoring the condition of rotating equipment according to claim 1, characterized in that, The defective parts of the rotating equipment are identified, the first hazard level is determined based on the defective parts, the standard operating temperature of the corresponding defective parts is obtained, and the actual operating temperature of the defective parts is obtained based on the thermal imaging images of the defective parts. The second hazard level is determined based on the actual operating temperature and the standard operating temperature. The priority warning level of the defective parts is determined based on the specific types of the first and second hazard levels that the defective parts simultaneously possess.

3. The image analysis-based method for monitoring the condition of rotating equipment according to claim 2, characterized in that, The method for enhancing the sharpness of the thermal imaging image includes: S1. Adjust the thermal imaging image size to a uniform size; S2. Normalize the pixel values ​​of the thermal imaging image; S3. Based on the normalized thermal imaging image, ResNet-50 in the deep residual network model of the neural network model is used to enhance the clarity of the thermal imaging image.

4. The image analysis-based method for monitoring the condition of rotating equipment according to claim 3, characterized in that, In S3, the method for setting up the deep residual network model includes: S301, The input layer is used to input the normalized thermal imaging image into the deep residual network; S302. The convolutional layer performs a convolution operation on the input thermal imaging image using a 7×7 convolutional kernel, with a stride of 2 and an output channel count of 64. S303, the batch normalization layer is used to perform batch normalization operations on the thermal imaging images output by the convolutional layer; S304, the activation function layer activates the batch-normalized thermal imaging image in S303 by using the ReLU activation function; S305. The max pooling layer performs max pooling operation using a 3×3 pooling kernel, with a step size of 2, to reduce the size of the feature map, which is the thermal imaging image activated in S304. S306. The network is constructed using H residual blocks, where H is an integer greater than 1. Each residual block consists of two convolutional layers and one skip connection. The first convolutional layer has a kernel size of 1×1 and 64 output channels. The second convolutional layer has a kernel size of 3×3 and 64 output channels. The skip connection directly adds the thermal imaging image as input to the output of the residual block. S307. Repeat the residual block three times in sequence. Repeat the first residual block and set the number of output channels of the convolution kernels of the first and second convolutional layers to 128. Repeat the second residual block and set the number of output channels of the convolution kernels of the first and second convolutional layers to 256. Repeat the third residual block and set the number of output channels of the convolution kernels of the first and second convolutional layers to 512. S308, the average pooling layer reduces the dimensionality of the feature map by using global average pooling; S309, the output layer, is used to output the thermal imaging image after sharpness enhancement processing.

5. The method for monitoring the condition of rotating equipment based on image analysis according to claim 1, characterized in that, Thresholding segmentation algorithms, such as the maximum inter-class variance method, are used to detect and label temperature change regions in difference images. The difference image is set to a specific size and contains several gray levels. Based on the gray level probability distribution, a gray level threshold is selected, and all pixels of the difference image are divided into two sets according to their gray level values. The first set contains pixels whose gray level values ​​are in the range from the lowest level to the threshold, and the second set contains pixels whose gray level values ​​are in the range from the threshold plus one to the highest level, and the highest level is greater than the threshold plus one. Calculate the proportion of pixels in the first set in the entire image, which is the sum of the gray level probabilities of all pixels in the first set; calculate the average gray level of the first set, which is the sum of the products of the gray level values ​​of all pixels in the set and their respective gray level probabilities, and then divide by the proportion of pixels in the first set. Calculate the proportion of pixels in the second set in the entire image, which is the sum of the gray level probabilities of all pixels in the set, and the sum of this proportion and the pixel proportion of the first set is 1; calculate the average gray level of the second set, which is the sum of the products of the gray level values ​​of all pixels in the set and their respective gray level probabilities, and then divide by the pixel proportion of the second set. Calculate the cumulative mean corresponding to the grayscale threshold, which is the sum of the products of the grayscale values ​​of all pixels in the first set and their respective grayscale probabilities; calculate the average grayscale of the entire difference image, which is the sum of the products of the grayscale values ​​of all pixels and their respective grayscale probabilities, and the product of the pixel proportion of the first set and the average grayscale of the first set, plus the product of the pixel proportion of the second set and the average grayscale of the second set, equals the average grayscale of the entire image. The inter-class variance is defined as: the pixel percentage of the first set multiplied by the square of the difference between the average gray level of the first set and the average gray level of the whole image, plus the pixel percentage of the second set multiplied by the square of the difference between the average gray level of the second set and the average gray level of the whole image. The thresholding function provided by OpenCV in Python is used to perform thresholding on the difference image. The processing mode of the function is set to the mode corresponding to the maximum inter-class variance. The pixel ratio and average gray level of the two sets are calculated respectively. All gray levels of the entire difference image are traversed to find the gray level threshold that makes the inter-class variance reach the maximum, and then the temperature change region is detected and marked.

6. The method for monitoring the condition of rotating equipment based on image analysis according to claim 2, characterized in that, The specific types of the first hazard level include the first severe mark, the first moderate mark, and the first minor mark; the specific types of the second hazard level include the second severe mark, the second moderate mark, and the second minor mark. Methods for determining the priority warning level of a defect location based on the specific types of both the primary and secondary hazard levels it simultaneously possesses include: The warning levels include the first warning order, the second warning order, and the third warning order, with the priority of the warning order decreasing progressively from the first warning order to the third warning order. Case A is defined as the defect location having both the first severe mark and the second severe mark simultaneously; Case B is defined as the defect location having both the first severe mark and the second moderate mark simultaneously; Case C is defined as the defect location having both the first moderate mark and the second severe mark simultaneously; Case D is defined as the defect location having any of the following conditions: Case A, Case B, and Case C. If the defective part has condition A, a first warning instruction is generated; if the defective part has condition B or C, a second warning instruction is generated; if the defective part has condition D, a third warning instruction is generated.

7. The image analysis-based method for monitoring the condition of rotating equipment according to claim 6, characterized in that, The method for determining the second hazard level based on the actual working temperature and the standard working temperature includes: first, calculating the temperature difference between the actual working temperature and the standard working temperature, comparing the temperature difference with the corresponding preset range temperature threshold of the defect location, and if the temperature difference is less than the minimum value of the preset range temperature threshold, then marking it as a second minor hazard level. If the temperature difference is greater than or equal to the minimum value of the preset range temperature threshold and less than the maximum value of the preset range temperature threshold, it is marked as the second medium mark. If the temperature difference is greater than or equal to the maximum value of the preset temperature threshold, it is marked as the second most serious.

8. A monitoring system for the condition of rotating equipment based on image analysis, characterized in that, The method for monitoring the condition of rotating equipment based on image analysis as described in any one of claims 1-7 includes: The image data acquisition module acquires thermal imaging images of the transfer equipment and forms a dataset; The image enhancement algorithm module performs sharpness enhancement processing on the thermal imaging image; The ambient temperature elimination module performs ambient temperature elimination on the thermal imaging image when the infrared thermal imager captures thermal imaging images of the rotating equipment, and obtains the ambient temperature eliminated thermal imaging image, which is marked as the ambient temperature eliminated thermal imaging image. The defect type identification module inputs the eliminated ambient temperature thermal imaging image into the trained defect type identification model to obtain a defective thermal imaging image. The equipment target detection module inputs the defective thermal imaging image into the target detection model, obtains the numerical number of the defective part of the defective thermal imaging image, and determines the defective part; The ambient temperature elimination module is specifically used for: S10. Create a difference image by subtracting the grayscale values ​​of the original thermal imaging image from the sharpness-enhanced thermal imaging image to obtain the difference image. S20. Detect and label temperature change regions in the differential image using a threshold segmentation algorithm; S30. Use the marked temperature change area to adjust the corresponding area in the original thermal imaging image to eliminate the influence of ambient temperature in the thermal imaging image, and obtain the thermal imaging image after ambient temperature elimination, which is marked as the ambient temperature eliminated thermal imaging image. The ambient temperature elimination module is also specifically used for: S31. Based on the marked temperature change area, determine the original thermal imaging image area that needs to be adjusted, calculate the difference between the gray value of each pixel in the original thermal imaging image and the gray value of its neighboring pixels, and adjust the pixel value of the corresponding area in the original thermal imaging image according to the difference. S32. By adjusting the corresponding area in the original thermal imaging image, the gray value of the corresponding area pixels is adjusted to the same level as the surrounding environment to obtain the thermal imaging image after the ambient temperature is eliminated, and it is marked as the thermal imaging image after the ambient temperature is eliminated. S33. Save or display the thermal imaging image after eliminating ambient temperature.

Images