Method and system for early warning thermal fault of fan component

Abstract

The invention relates to a fan component thermal fault early warning method and system.The fan component thermal fault early warning method comprises the steps that after an infrared image and a visible light image are collected, the infrared image and the visible light image are registered and spliced to obtain a spliced image, and the spliced image is input into a diffusion network; training the diffusion network through an end-to-end training mode based on the structural similarity loss and the weighted loss of the denoising network, obtaining a fused image, inputting the fused image into a semantic segmentation network, and dividing each part in the infrared image by the semantic segmentation network based on a first feature map and a second feature map of the fused image, and finally, whether the fan has a thermal fault is judged in combination with a component segmentation result and a temperature region classification result, structural similarity loss is introduced into a diffusion network, a lightweight segmentation network is provided in a segmentation task, and the segmentation network comprises a cross attention module and a contour extraction module. And the model size and the reasoning time consumption are reduced under the condition of improving the segmentation precision.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for early warning of thermal failure of fan components. Background Art

[0002] A fan is a machine that relies on input mechanical energy to increase gas pressure and discharge gas. It is a driven fluid machine. Fan is the customary abbreviation for gas compression and gas delivery machinery in China. The fans usually include ventilators, blowers, and wind turbines.

[0003] Fans are widely used for ventilation in factories, mines, tunnels, cooling towers, vehicles, ships and buildings. They are also used for cooling and ventilation in air conditioning equipment and household appliances. It can be seen that fans have a wide range of applications.

[0004] Since the various internal components of the fan will generate heat when working, and the heat generated by the fan will also fluctuate when the fan is working abnormally, the heat generated by the abnormal fan is different from the heat generated by the normally working fan. Therefore, it is possible to determine whether the fan is abnormal by detecting the heat of various components of the fan.

[0005] Existing fan thermal warning methods often do not accurately divide the areas of different fan components during detection, resulting in blurred boundaries of the divided areas. It is also difficult to determine whether the corresponding components in the area have a thermal failure based on the regional temperature, making the final detection results inaccurate or difficult to issue warnings in a timely manner. Summary of the invention

[0006] Based on this, it is necessary to provide a fan component thermal failure warning method and system to address the problem that the existing fan thermal warning method is often not accurate enough in the regional division of different fan components during detection, resulting in blurred boundaries of the divided areas. It is also difficult to judge whether the component corresponding to the area has a thermal failure based on the regional temperature, making the final detection result inaccurate or difficult to issue a warning in time.

[0007] In one aspect, the present application provides a fan component thermal failure early warning method, comprising:

[0008] Acquire multiple infrared images and visible light images of the wind turbine, register the infrared images and the visible light images to obtain registered image pairs, stitch the infrared images and the visible light images in each image pair to obtain stitched images, and divide all the stitched images into training sets and test sets according to a certain ratio;

[0009] Creating a diffusion network; the diffusion network includes a diffusion model and a multi-channel module, and the diffusion model includes a forward network and a denoising network;

[0010] The spliced ​​image as a training set is input into the diffusion network, and the diffusion network is trained in an end-to-end training manner based on the structural similarity loss and the weighted loss of the denoising network, until the trained diffusion network has the ability to automatically output a fused image based on the input infrared image and visible light image;

[0011] Creating a semantic segmentation network; the semantic segmentation network includes an encoder, a cross attention module CCAM and an outline extraction module EEM;

[0012] Input the fused image output by the diffusion network into the semantic segmentation network, and output the first feature map and the second feature map of the fused image through the encoder of the semantic segmentation network;

[0013] The first feature map is subjected to affine operation and aggregation operation through the cross attention module CCAM to obtain an enhanced feature map, and the enhanced feature map is added to the first feature map to obtain a final feature map;

[0014] Inputting the second feature map into the contour extraction module EEM, and calculating the contour loss of the second feature map by the contour extraction module EEM;

[0015] The fused image is divided into components according to the final feature map and contour loss, and combined with the classification results of the infrared image, it is determined whether there are components with incorrect temperature classification;

[0016] If there are components with the wrong temperature classification, the fan is faulty;

[0017] The spliced ​​image as a training set is input into the diffusion network, and the diffusion network is trained through an end-to-end training method based on the structural similarity loss and the weighted loss of the denoising network, including:

[0018] Set the training number threshold and training number T, and set T = 0;

[0019] The loss of the denoising network is calculated based on the P2-weight idea, and the denoising network is trained using the loss of the denoising network;

[0020] The loss of the multi-channel module is calculated based on the multi-channel texture loss and the multi-channel pixel loss, and the multi-channel module is trained using the loss of the multi-channel module;

[0021] Let the number of training times T=T+1, and determine whether the number of training times is greater than or equal to the training times threshold;

[0022] If the number of training times is greater than the training times threshold, the loss of the denoising network calculated based on the P2-weight idea is returned, and the denoising network is trained using the loss of the denoising network until the number of training times is greater than or equal to the training times threshold;

[0023] If the number of training times is greater than or equal to the training times threshold, the training is exited to obtain the trained denoising network and the trained multi-channel module;

[0024] Input the stitched image into the forward network of the stitching model to obtain a noisy image;

[0025] Input the noisy image into the trained denoising network to obtain a denoised image;

[0026] The denoised image is input into the trained multi-channel module to obtain the fused image.

[0027] On the other hand, the present application also provides a fan component thermal failure warning system, the fan component thermal failure warning system comprising:

[0028] A collection component, through which infrared images and visible light images of the target are collected;

[0029] A processing component, wherein the acquisition component is in communication with the processing component, and the processing component executes the above-mentioned fan component thermal fault early warning method on the acquired infrared image and visible light image;

[0030] The alarm component is in communication with the processing component. When the processing component determines that the fan temperature is abnormal, the processing component controls the alarm component to sound an alarm.

[0031] The present application relates to a method and system for early warning of thermal failure of fan components. After collecting infrared images and visible light images, the method aligns and splices the infrared images and visible light images to obtain a spliced ​​image, which is input into a diffusion network. Based on the structural similarity loss and the weighted loss of the denoising network, the diffusion network is trained in an end-to-end training manner to obtain a fused image, which is input into a semantic segmentation network. The semantic segmentation network divides each component in the infrared image based on the first feature map and the second feature map of the fused image. Finally, the component segmentation result and the temperature area classification result are combined to determine whether the fan has a thermal failure. The present application introduces structural similarity loss in the diffusion network, and proposes a lightweight segmentation network in the segmentation task, which includes a cross attention module and a contour extraction module, which reduces the model size and inference time while improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic flow chart of a method for early warning of thermal failure of a fan component provided in one embodiment of the present application.

[0033] Figure 2 This is a schematic diagram of the structure of a fan component thermal failure warning system provided in one embodiment of the present application.

[0034] Figure 3A schematic flow chart of a diffusion network of a fan component thermal failure warning method provided in one embodiment of the present application.

[0035] Figure 4 A flowchart of a semantic segmentation network for a fan component thermal failure warning method provided in one embodiment of the present application.

[0036] Figure 5 A schematic diagram of temperature zone classification of fan components in a fan component thermal failure warning method provided in one embodiment of the present application.

[0037] Figure 6 A schematic diagram of a fan thermal failure warning method for a fan component thermal failure warning provided in an embodiment of the present application.

[0038] Reference numerals:

[0039] 100, acquisition component; 200, processing component; 300, alarm component. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The present application provides a fan component thermal failure early warning method and system.

[0042] like Figure 1 As shown, in one embodiment of the present application, a fan component thermal failure early warning method is provided, and the fan component thermal failure early warning method includes:

[0043] S100, acquiring multiple infrared images and visible light images of the wind turbine, registering the infrared images and the visible light images to obtain registered image pairs, stitching the infrared images and the visible light images in each image pair to obtain a stitched image, and dividing all the stitched images into a training set and a test set according to a certain ratio.

[0044] S200, creating a diffusion network. The diffusion network includes a diffusion model and a multi-channel module, and the diffusion model includes a forward network and a denoising network.

[0045] S300, inputs the spliced ​​image as a training set into the diffusion network, and trains the diffusion network through an end-to-end training method based on the structural similarity loss and the weighted loss of the denoising network, until the trained diffusion network has the ability to automatically output a fused image based on the input infrared image and visible light image.

[0046] S400, creating a semantic segmentation network. The semantic segmentation network includes an encoder, a cross attention module CCAM and an outline extraction module EEM.

[0047] S500, inputting the fused image output by the diffusion network into a semantic segmentation network, and outputting a first feature map and a second feature map of the fused image through an encoder of the semantic segmentation network.

[0048] S600, performing affine operation and aggregation operation on the first feature map through a cross attention module CCAM to obtain an enhanced feature map, and adding the enhanced feature map to the first feature map to obtain a final feature map.

[0049] S700: Input the second feature map into the contour extraction module EEM, and calculate the contour loss of the second feature map through the contour extraction module EEM.

[0050] S800, dividing the fused image into components according to the final feature map and the contour loss, and combining the classification result of the infrared image to determine whether there are components with incorrect temperature classification.

[0051] S900, if there is a component with the wrong temperature classification, the fan is faulty.

[0052] The S300 includes:

[0053] S310, setting a training times threshold and training times T, and setting T=0.

[0054] S320, based on the P2-weight concept, the loss of the denoising network is calculated, and the denoising network is trained using the loss of the denoising network.

[0055] S330, calculating the loss of the multi-channel module based on the multi-channel texture loss and the multi-channel pixel loss, and using the loss of the multi-channel module to train the multi-channel module.

[0056] S340, let the number of training times T=T+1, and determine whether the number of training times is greater than or equal to the training times threshold.

[0057] S350, if the number of training times is greater than the training times threshold, return to calculate the loss of the denoising network based on the P2-weight idea, and use the loss of the denoising network to train the denoising network until the number of training times is greater than or equal to the training times threshold.

[0058] S360: If the number of training times is greater than or equal to the training times threshold, the training is exited to obtain a trained denoising network and a trained multi-channel module.

[0059] S370, input the stitched image into the forward network of the stitching model to obtain a noisy image.

[0060] S380, inputting the noisy image into the trained denoising network to obtain a denoised image.

[0061] S390, input the denoised image into the trained multi-channel module to obtain a fused image.

[0062] In this embodiment, after collecting the infrared image and the visible light image, the infrared image and the visible light image are aligned and spliced ​​to obtain a spliced ​​image, which is input into the diffusion network. Based on the structural similarity loss and the weighted loss of the denoising network, the diffusion network is trained through an end-to-end training method to obtain a fused image, which is input into the semantic segmentation network. The semantic segmentation network divides the components in the infrared image based on the first feature map and the second feature map of the fused image. Finally, the component segmentation result and the temperature area classification result are combined to determine whether the fan has a thermal failure. The present application introduces structural similarity loss in the diffusion network and proposes a lightweight segmentation network in the segmentation task, which includes a cross attention module and a contour extraction module, which reduces the model size and inference time while improving the segmentation accuracy.

[0063] In one embodiment of the present application, the S100 includes:

[0064] S101, creating an initial image database.

[0065] S102, setting the collection duration, collection frequency and collection interval angle.

[0066] S103, simultaneously collecting an infrared image and a visible light image at every collection frequency and collection interval angle within the collection time, and putting the infrared image and the visible light image collected at the same time as an initial image pair into an initial image database to obtain multiple initial image pairs.

[0067] S104: randomly select an initial image pair from the initial image database.

[0068] S106, performing any one or more operations of flipping, horizontal mirroring and random cropping on the initial image pair to obtain the operated image pair and put it into the initial image database to expand the initial image database.

[0069] S107, randomly selecting an image pair from the expanded initial image database. The image pair includes the initial image pair and the operated image pair.

[0070] S108, aligning the infrared image and the visible light image in the image pair by feature point matching through a combination method of ORB feature extraction, feature matching, and RANSAC algorithm estimation of the homography matrix, to obtain a registered image pair.

[0071] Specifically, the S108 includes:

[0072] K101, converting the infrared image and the visible light image in the image pair into grayscale images. Recording the grayscale image corresponding to the infrared image as a first grayscale image, and recording the grayscale image corresponding to the visible light image as a second grayscale image.

[0073] K102, respectively extracting the edge of the first grayscale image and the edge of the second grayscale image by using a Canny edge detection algorithm.

[0074] K103, based on the edge of the first grayscale image and the edge of the second grayscale image, find the matching position and the maximum matching value of the first grayscale image and the second grayscale image through a template matching algorithm and a correlation coefficient method.

[0075] Specifically, the template matching algorithm can be executed by cv2.matchTemplate. The correlation coefficient method can be executed by cv2.TM_CCOEFF.

[0076] K104, calculating the matching coordinates of the matching area in the original image according to the matching position, and extracting the matching area in the visible light image according to the matching coordinates.

[0077] K105, scaling and cropping the matching area so that the size of the matching area is equal to the infrared image.

[0078] K106, calculates the homography matrix through ORB feature detection, and aligns the visible light image to the infrared image through the homography matrix to obtain the registered image pair.

[0079] S109, performing channel stitching on the registered image pair to obtain a stitched image.

[0080] S110, returning to randomly select an image pair from the expanded initial image database until every image pair in the initial image database is selected to obtain a plurality of stitched images.

[0081] S111, setting the division ratio.

[0082] S112, dividing all the stitched images into a training set and a test set according to a division ratio.

[0083] In this embodiment, before starting to capture the initial image, it is necessary to first set the capture frequency, capture time and capture interval angle of the capture component. For example, if the capture frequency is set to 3s / image, the capture time is 3 minutes, and the capture interval angle is 30°, then take the starting position as the initial position, and take an infrared image and a visible light image at 30° intervals clockwise or counterclockwise until the capture time is reached. A total of multiple infrared images and visible light images are obtained. A pair of infrared images and visible light images at the same time are recorded as an initial image pair to facilitate subsequent processing.

[0084] After the initial image acquisition is completed, in order to expand the data volume of the initial image, any one or more operations of flipping, horizontal mirroring and random cropping are performed to increase the total data volume in the initial image database, thereby increasing the number of training samples and further improving the training effect of the diffusion network.

[0085] When expanding, the initial image pair is still used as the basic unit, that is, the same operation is performed on the visible light image and infrared image contained in each initial image pair, so as to facilitate subsequent registration operations and stitching operations.

[0086] After the expansion is completed, the infrared image and visible light image contained in each initial image pair are registered through a combination of ORB feature extraction, feature matching, and RANSAC algorithm estimation of the homography matrix, thereby converting the initial image pair into a registered image.

[0087] Finally, the registered images are channel-stitched to obtain a stitched image, thus completing the preprocessing of the initial image.

[0088] After all the initial image pairs are processed and multiple stitched images are obtained, all the stitched images are randomly divided into the training set or the test set according to the set division ratio, for example, the training set: test set division ratio is 7:3. If there are 2000 stitched images and the training set: test set division ratio is 7:3, then the test set contains a total of 1400 stitched images and the training set contains 600 stitched images.

[0089] like Figure 3 As shown, in one embodiment of the present application, the S300 includes:

[0090] S301, adding noise to the initial image step by step according to Formula 1 and Formula 2, and obtaining a final image after adding noise after t steps of adding noise.

[0091]

[0092] Among them, I tis the noisy image at step t, I0 is the diffusion image at step 0, t is the number of forward diffusion steps, ε~N(0,Ι) is the added random noise, β t is the first noise proportional coefficient added in the forward diffusion process, a t is the second noise ratio added in the forward process, is all a from 0 to t t The product of .

[0093] Specifically, since the input stitched image has not been diffused at step 0, the diffused image I0 at step 0 is the stitched image input to the diffusion model.

[0094] Before executing S301 , the forward diffusion process of the forward network of the diffusion model is defined by formula 16.

[0095]

[0096] Among them, q(I t |I t-1 ) is the forward diffusion process of the diffusion model, t is the number of steps of forward diffusion, β t is the noise proportional coefficient added in the forward diffusion process, I t is the noisy image at step t, I t is the noisy image at step t-1, and N is a normal distribution.

[0097] In this embodiment, the forward diffusion process of the diffusion model is a process of gradually adding Gaussian noise to the input sample image to transform the data into a standard Gaussian distribution, and β t It is the noise proportional coefficient added in the forward diffusion process. Specifically, 0<β1,β2,...,β T <1,t∈[1,T], based on the formula, the formula can be derived, so that the noisy image I of any t-th step can be directly obtained from the original image I0 through the formula t .

[0098] like Figure 3 As shown, in one embodiment of the present application, the S300 further includes:

[0099] Get the final image after adding noise.

[0100] The final image after adding noise is denoised by the formula, and the final image after denoising is obtained by calculating the denoised image after t steps of denoising by formula 3.

[0101]

[0102] Among them, ε θ (I t,t) is the noise removed at each step, I' t-1 is the denoised image at step t-1, I t is the noisy image at step t, t is the number of forward diffusion steps, θ is the parameter set of the denoising network, β t is the noise proportional coefficient added in the forward diffusion process, z~Ν(0,1), ε θ (I t ,t) is the noise removed in the tth step, σ t is a hyperparameter of the noise intensity added at the t-th step in the denoising process.

[0103] In this embodiment, ε θ (I t ,t) is predicted by the denoising network, and only the current step data I is input to the denoising network t And the number of steps t, the removed noise ε can be predicted θ (I t ,t).

[0104] In the denoising process, random noise is gradually denoised, and finally a clear image is obtained.

[0105] The denoised image obtained by each denoising step is actually not the same as the denoised image of the corresponding step number. For example, if the denoised images are I1, I2 and I3, and I3 is used as the input of the denoising network, then the denoised images I'2 and I'1 can be obtained by denoising step by step. However, the denoised image I'2 is not the same as the denoised image I2, and the denoised image I'1 is not the same as the denoised image I1. The denoised image is clearer than the denoised image of the synchronous number.

[0106] In one embodiment of the present application, the S320 includes:

[0107] S321, calculating the mean square error of the noise added by the multi-channel module at each moment and the noise predicted by the denoising network through Formula 4.

[0108]

[0109] Among them, L t is the mean square error of the noise added at time t and the noise predicted by the denoising network, t is the number of diffusion steps, ε~Ν(0,Ι) is the added random noise, I0 is the diffusion image at step 0, ε θ .

[0110] S322, define the weight coefficient at time t using Formula 5.

[0111]

[0112] Among them, λ tis the weight coefficient of t, k and γ are hyperparameters, and SNR(t) is the signal-to-noise ratio at time t.

[0113] S323, calculate the loss of P2-weight using Formula 6.

[0114] L p2 =Σ t λ t L t Formula 6.

[0115] Among them, L p2 is the P2-weight loss, λ t is the weight coefficient of t, L t is the mean square error between the noise added at time t and the noise predicted by the denoising network.

[0116] Specifically, during training, the loss at each moment is not equally important. As the diffusion process proceeds, the noise in the image becomes larger and larger, and the amount of information becomes smaller and smaller. This process can be measured by the signal-to-noise ratio (SNR). When t is closer to T, the image's signal-to-noise ratio is lower and the amount of information is less. Therefore, P2-weight proposes: use the weight associated with the diffusion moment t to correct the mean square error loss in the diffusion model. The closer t is to T, the smaller the weight value.

[0117] S324, define the perceptual loss of the VGG19 network by formula 7.

[0118]

[0119] Among them, L vgg is the perceptual loss of the VGG19 network, is the jth layer of the VGG19 network, H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, ∈~Ν(0,Ι) is the added random noise, θ is the parameter set of the denoising network, I t is the noisy image at step t, and t is the number of forward diffusion steps.

[0120] S325, calculating the loss of the denoising network of the diffusion model through Formula 8.

[0121] L ddpm =L p2 +L vgg Formula 8.

[0122] Among them, L ddpm is the loss of the denoising network of the diffusion model, L p2 is the loss of P2-weight, L vgg is the perceptual loss of the VGG19 network.

[0123] S326, substituting the loss of the denoising network of the diffusion model into the denoising network to train the denoising network.

[0124] In this embodiment, the diffusion model trained with the P2-weight loss calculated under the P2-weight concept is different from the original loss function L t Compared with the diffusion model trained with p2-weight loss, the diffusion model trained with p2-weight loss has richer visual effects. The rich visual effects will bring richer texture information to the fusion results of infrared images and visible light images, improve the quality of the fused image, make the texture of the fused image richer, and make the brightness and darkness of the spliced ​​image softer.

[0125] In addition, the perceptual loss L is also used vgg To measure the authenticity of the denoised image, since the input of the VGG19 network is a three-channel feature, the infrared image with less information can be ignored during processing, and the visible light image is selected as the input of the VGG19 network.

[0126] Combining the loss of P2-weight and the perceptual loss of the VGG19 network, we can get the loss L of the denoising network of the diffusion model. ddpm , and the loss L of the denoising network based on the diffusion model ddpm To train the denoising network of the diffusion model.

[0127] In one embodiment of the present application, the S330 includes:

[0128] S331, calculate the multi-channel texture loss of the multi-channel module through Formula 9.

[0129]

[0130] Among them, L MCG is the multi-channel texture loss of the multi-channel module, H is the height of the feature map, W is the width of the feature map, represents the derivative, I f is the fused image, I ir is the infrared image, I vis is a visible light image.

[0131] S332, calculating the multi-channel pixel loss of the multi-channel module using formula 10.

[0132]

[0133] Among them, L MCI is the multi-channel pixel loss of the multi-channel module, H is the height of the feature map, W is the width of the feature map, and I f is the fused image, I ir is the infrared image, I visis a visible light image,

[0134] S333, calculate the structural similarity loss of the multi-channel module through formula 11.

[0135] L SSIM =δ(1-SSIM(I f ,I vis ))+(1-SSIM(I f ,I ir )) Formula 11.

[0136] Among them, L SSIM is the structural similarity loss, I f is the fused image, I ir is the infrared image, I vis is a visible light image, and δ is a hyperparameter.

[0137] Specifically, δ is used to dynamically control the similarity between the fused image and one of the infrared image and the visible light image.

[0138] Structural Similarity (SSIM) is an indicator to measure the similarity between two images. Among the two images used by SSIM, one is an uncompressed and distortion-free image, and the other is a distorted image. In this application, infrared images and visible light images are used, that is, the similarity between infrared images and visible light images is judged by structural similarity.

[0139] S334, calculate the loss function of the multi-channel module using formula 12.

[0140] L fusion =L MCG +L MCI +L SSIM Formula 12.

[0141] Among them, L fusion is the loss function of the multi-channel module, L MCG is the multi-channel texture loss of the multi-channel module, L MCI is the multi-channel pixel loss of the multi-channel module, L SSIM is the structural similarity loss.

[0142] S335, substituting the loss function of the multi-channel module into the multi-channel module to train the multi-channel module.

[0143] In this embodiment, the loss function of the multi-channel module is obtained by combining the multi-channel pixel loss of the multi-channel module, the multi-channel texture loss of the multi-channel module and the structural similarity loss, so that the loss function of the multi-channel module is used to train the multi-channel module.

[0144] When training the denoising network of the multi-channel module and the diffusion model, an end-to-end training method is adopted, that is, the multi-channel module and the diffusion model are trained alternately until the training times threshold is reached, so that the trained multi-channel module and the trained diffusion model have better performance, and the fused image finally output by the diffusion network is clearer.

[0145] End-to-end joint training is a training method that allows the model to simultaneously learn the direct mapping relationship between input and output without explicit intermediate steps or stages. This training method usually involves training the various parts of the model as a whole to ensure that they work together to achieve the final goal. The advantage of end-to-end training is that it simplifies the model training process and reduces the need to manually design intermediate steps, allowing the model to learn more complex dependencies and patterns from the data.

[0146] like Figure 4 As shown, in one embodiment of the present application, the S600 includes:

[0147] S601, inputting the fused image into a semantic segmentation network, and outputting a first feature map and a second feature map of the fused image through an encoder of the semantic segmentation network.

[0148] Specifically, the first feature map is denoted as f2, and the second feature map is denoted as f5.

[0149] The first feature map is recorded as f2 because the first feature map is output from the second channel of the encoder, and the second feature map f5 is output from the fifth channel of the encoder.

[0150] The criss-cross attention module is an improvement of the self-attention mechanism and is mainly used in semantic segmentation networks. This module effectively improves the quality of feature maps by capturing global context information in the horizontal and vertical directions.

[0151] S602: Perform dimension reduction on the first feature map to reduce the number of channels of the first feature map.

[0152] Specifically, reducing the number of channels of the first feature map can reduce the amount of calculation, thereby reducing the memory consumption of subsequent calculation processes.

[0153] S603: Perform 1x1 convolution on the first feature map after dimensionality reduction to obtain enhanced feature map Q, enhanced feature map K and enhanced feature map V.

[0154] Specifically, the enhanced feature map Q stands for Query. In image processing, Query represents the part where specific information or features need to be extracted. Usually, Query is a feature vector that represents the features of a certain position in the image. It can be a point or area in the convolutional feature map, which is used to ask questions or find specific patterns or objects in the image.

[0155] The enhanced feature map K stands for Key. In image processing, Key represents the feature vector in the image, which is used to match the query. Each Key represents the feature of a certain location or area of ​​the image, which is used to determine the similarity or relevance with the query.

[0156] The enhanced feature map V stands for Value. In image processing, Value represents the actual feature content. The Value vector contains specific information about the image. When the relevant Key is found, the Value information is extracted and used. Value is usually used together with Key to pass information in the matching between Query and Key.

[0157] S604, performing affine operation on the enhanced feature map Q and the enhanced feature map K according to formula 13.

[0158]

[0159] Among them, d i,u represents an affine operation, Q u ∈R C' Represents the element taken from the pixel at position u in the enhanced feature map Q, Ω u represents the set of elements of the same row and column taken from the enhanced feature map K and the same position u in the enhanced feature map Q, C' is the number of channels of the feature map of the fused image after the fusion image is reduced in dimension, H is the height of the feature map of the fused image, W is the width of the feature map of the fused image, d i,u ∈D represents Q u and between Ω u The degree of association, i represents the i-th element in the set, and i=[1,...,|Ω u |] R ,Ω u With Ω i,u relationship.

[0160] S605, pass D through the softmax function to obtain the attention matrix A, A∈R (H+W-1)×W×H

[0161] S606, the attention matrix A and the enhanced feature map V are aggregated using formula 14.

[0162]

[0163] Among them, A i,u is the i-th element of all elements taken out at the pixel position u in the attention matrix A, Φ i,u is the set of all elements in the same row and column taken from the same position u in the enhanced feature map V and the attention matrix A, I u is the element at position u in the first feature map, I' u It is the weighted adjusted value of the element at position u of the first feature map.

[0164] S607, enhance the feature map and the fused image I f Add element by element to get the final feature map.

[0165] S607: The element I at position u of the first feature graph is u The weighted adjusted value I' u Through the cross attention module CCAM module, the pixel at position u has the contextual information of all positions in the entire image.

[0166] In this embodiment, since I′ u Only the information of elements in the same row and column is associated, so the adjusted elements only combine the local context semantics. In order to pass the semantic information in the entire feature map, I_f^' is passed through a CCAM module again, so that the pixel at position u will have the context information of all positions in the entire image.

[0167] In one embodiment of the present application, the S700 includes:

[0168] S701, input the second feature map into the contour extraction module EEM, reduce the number of channels of the second feature map through the contour extraction module EEM and output the contour feature f of the second feature map e .

[0169] Specifically, the contour extraction module EEM includes a combination of multiple convolutional layers, batch normalization layers and ReLU activation functions.

[0170] S702, using the Canny operator algorithm to perform contour feature f of the second feature map e Perform contour extraction to obtain the contour feature f of the second feature map e The true value of the benchmark.

[0171] S703, the contour feature f of the second feature map is calculated by formula 15 e and the contour feature f of the second feature map e The contour feature f of the second feature map is calculated by the reference true value e Contour loss.

[0172]

[0173] Among them, L edge is the contour feature f of the second feature map e Contour loss, P(y i =1) is the contour feature f of the second feature map e The probability that the pixel at position i in is predicted to be a contour, P(y j =0) is the contour feature f of the second feature map e The probability that the pixel at position j is predicted to be background, and is the weight coefficient, U + is the set of contour pixels, U - is a collection of background pixels.

[0174] In this embodiment, in order to improve the accuracy of the component segmentation results, when segmenting through the semantic segmentation network, the contour is equivalent to auxiliary information in the segmentation process, so that when training the semantic segmentation network, segmentation can be performed based on the auxiliary information, so that the semantic segmentation network can better segment the components in the image.

[0175] like Figure 5 and Figure 6 As shown, in one embodiment of the present application, the S800 includes:

[0176] S801, selecting an infrared image and a visible light image at the same time from an initial image database.

[0177] S802 , dividing each component in the infrared image into a low-temperature image portion, a medium-temperature image portion, and a high-temperature image portion through K-means clustering.

[0178] Specifically, in this embodiment, K of K-means clustering is 3.

[0179] S803, input the infrared image and the visible light image into a diffusion network to obtain a fused image.

[0180] S804, inputting the fused image into a semantic segmentation network, and segmenting the fused image into a plurality of components through the semantic segmentation network.

[0181] S805 , comparing the multiple components included in the fused image with the low-temperature image portion, the medium-temperature image portion, and the high-temperature image portion of each component in the infrared image, and determining whether the regional temperature of each component meets the temperature of its corresponding region.

[0182] S806: If the temperature of the area of ​​at least one component does not match the temperature of its corresponding area, it is determined that the temperature of the fan is abnormal.

[0183] In this embodiment, Figure 5 As shown, since the operating temperature of each component in the fan is generally maintained within a range when it is working normally, if the operating temperature of the component exceeds this range, it can be considered that the component has failed, and an early warning needs to be issued to the user.

[0184] In the infrared image, the K-means clustering algorithm can be used to divide the infrared image area corresponding to each component in the fan into three parts: low-temperature image part, medium-temperature image part and high-temperature image part. Each component has its corresponding temperature area. Then, the infrared image and the visible light image are fused through the diffusion network to obtain a fused image. The fused image is input into the semantic segmentation network to divide the components contained in the fused image. The output image based on the semantic segmentation network is compared with the infrared image after regional classification to determine whether the actual regional temperature of the area corresponding to each component is consistent with the regional temperature of the infrared image corresponding to the normal working temperature. If not, it is considered that the component corresponding to the area with abnormal temperature has a thermal fault.

[0185] like Figure 2 As shown, in one embodiment of the present application, a fan component thermal failure warning system is provided, which is applied to the fan component thermal failure warning method of the aforementioned embodiment, and the fan component thermal failure warning system includes a collection component, a processing component and an alarm component.

[0186] The infrared image and visible light image of the target are collected by the acquisition component. The acquisition component is connected in communication with the processing component, and the fan component thermal fault warning method described in the above embodiment is executed on the collected infrared image and visible light image by the processing component. The alarm component is connected in communication with the processing component, and when the processing component determines that the fan temperature is abnormal, the processing component controls the alarm component to sound an alarm.

[0187] In this embodiment, the acquisition component in the present application includes a dual-light camera, which can acquire infrared images and visible light images.

[0188] After the acquisition component has acquired a plurality of initial image pairs, all of these initial image pairs are stored in the processing component. The processing component executes the fan component thermal fault warning method as described in the aforementioned embodiment on these images, thereby determining whether the fan has a thermal fault. If the fan has a thermal fault, the alarm component is controlled to issue a warning, thereby alerting the user.

[0189] The technical features of the above-described embodiments may be arbitrarily combined, and the execution order of the method steps is not limited. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fan component thermal failure early warning method, characterized in that: The fan component thermal failure early warning method comprises: Acquire multiple infrared images and visible light images of the wind turbine, register the infrared images and the visible light images to obtain registered image pairs, stitch the infrared images and the visible light images in each image pair to obtain stitched images, and divide all the stitched images into training sets and test sets according to a certain ratio; Creating a diffusion network; the diffusion network includes a diffusion model and a multi-channel module, and the diffusion model includes a forward network and a denoising network; The spliced ​​image as a training set is input into the diffusion network, and the diffusion network is trained in an end-to-end training manner based on the structural similarity loss and the weighted loss of the denoising network, until the trained diffusion network has the ability to automatically output a fused image based on the input infrared image and visible light image; Creating a semantic segmentation network; the semantic segmentation network includes an encoder, a cross attention module CCAM and an outline extraction module EEM; Input the fused image output by the diffusion network into the semantic segmentation network, and output the first feature map and the second feature map of the fused image through the encoder of the semantic segmentation network; The first feature map is subjected to affine operation and aggregation operation through the cross attention module CCAM to obtain an enhanced feature map, and the enhanced feature map is added to the first feature map to obtain a final feature map; Inputting the second feature map into the contour extraction module EEM, and calculating the contour loss of the second feature map by the contour extraction module EEM; The fused image is divided into components according to the final feature map and contour loss, and combined with the classification results of the infrared image, it is determined whether there are components with incorrect temperature classification; If there are components with the wrong temperature classification, the fan is faulty; The spliced ​​image as a training set is input into the diffusion network, and the diffusion network is trained through an end-to-end training method based on the structural similarity loss and the weighted loss of the denoising network, including: Set the training number threshold and training number T, and set T = 0; The loss of the denoising network is calculated based on the P2-weight idea, and the denoising network is trained using the loss of the denoising network; The loss of the multi-channel module is calculated based on the multi-channel texture loss and the multi-channel pixel loss, and the multi-channel module is trained using the loss of the multi-channel module; Let the number of training times T=T+1, and determine whether the number of training times is greater than or equal to the training times threshold; If the number of training times is greater than the training times threshold, the loss of the denoising network calculated based on the P2-weight idea is returned, and the denoising network is trained using the loss of the denoising network until the number of training times is greater than or equal to the training times threshold; If the number of training times is greater than or equal to the training times threshold, the training is exited to obtain the trained denoising network and the trained multi-channel module; Input the stitched image into the forward network of the stitching model to obtain a noisy image; Input the noisy image into the trained denoising network to obtain a denoised image; The denoised image is input into the trained multi-channel module to obtain the fused image.

2. The fan component thermal failure early warning method according to claim 1, characterized in that: The method comprises: obtaining a plurality of infrared images and visible light images of the wind turbine, registering the infrared images and the visible light images to obtain registered image pairs, splicing the infrared images and the visible light images in each image pair to obtain a spliced ​​image, and dividing all the spliced ​​images into a training set and a test set according to a certain ratio, including: Create an initial image database; Set the collection duration, collection frequency and collection interval angle; During the acquisition time, an infrared image and a visible light image are simultaneously acquired at every acquisition frequency and acquisition interval angle, and the infrared image and the visible light image acquired at the same time are put into the initial image database as initial image pairs to obtain multiple initial image pairs; Randomly select an initial image pair from the initial image database; Perform any one or more operations of flipping, horizontal mirroring and random cropping on the initial image pair to obtain the operated image pair and put it into the initial image database to expand the initial image database; An image pair is randomly selected from the expanded initial image database; the image pair includes the initial image pair and the operated image pair; The infrared image and the visible light image in the image pair are registered by matching feature points through a combination of ORB feature extraction, feature matching, and RANSAC algorithm estimation of the homography matrix to obtain a registered image pair; Perform channel stitching on the registered image pair to obtain a stitched image; Return to randomly select an image pair from the expanded initial image database until every image pair in the initial image database is selected to obtain multiple stitched images; Set the division ratio; All stitched images are divided into training set and test set according to the division ratio.

3. The fan component thermal failure early warning method according to claim 2, characterized in that: The step of inputting the spliced ​​image into a forward network of the splicing model to obtain a noisy image, and inputting the noisy image into a denoising network of the splicing model to obtain a denoised image comprises: The initial image is gradually denoised using Formula 1 and Formula 2, and the final denoised image is obtained after t steps of denoising; Among them, I t is the noisy image at step t, I0 is the diffusion image at step 0, t is the number of forward diffusion steps, ε~Ν(0,Ι) is the added random noise, β t is the first noise proportional coefficient added in the forward diffusion process, a t is the second noise ratio added in the forward process, is all a from 0 to t t The product of .

4. The fan component thermal failure early warning method according to claim 3 is characterized in that: The denoised image is input into the multi-channel module, and the structural similarity loss of the denoised image and the weighted loss of the denoising network are combined to obtain a fused image, including: Get the final image after adding noise; The final image after adding noise is denoised by the formula, and the final image after denoising is obtained by calculating the denoised image after t steps of denoising by formula 3; Among them, ε θ (I t ,t) is the noise removed at each step, I' t-1 is the denoised image at step t-1, I t is the noisy image at step t, t is the number of forward diffusion steps, θ is the parameter set of the denoising network, β t is the noise proportional coefficient added in the forward diffusion process, z~Ν(0,1), ε θ (I t ,t) is the noise removed in the tth step, σ t is a hyperparameter of the noise intensity added at the tth step in the denoising process.

5. The fan component thermal failure early warning method according to claim 4, characterized in that: The loss of the denoising network is calculated based on the P2-weight concept, and the denoising network is trained using the loss of the denoising network, including: The mean square error of the noise added by the multi-channel module at each moment and the noise predicted by the denoising network is calculated by formula 4; Among them, L t is the mean square error of the noise added at time t and the noise predicted by the denoising network, t is the number of diffusion steps, ε~Ν(0,Ι) is the added random noise, I0 is the diffusion image at step 0, ε θ ; The weight coefficient at time t is defined by formula 5; Among them, λ t is the weight coefficient of t, k and γ are hyperparameters, and SNR(t) is the signal-to-noise ratio at time t; The loss of P2-weight is calculated by formula 6; L p2 =∑ t λ t L t Formula 6; Among them, L p2 is the P2-weight loss, λ t is the weight coefficient of t, L t is the mean square error of the noise added at time t and the noise predicted by the denoising network; The perceptual loss of the VGG19 network is defined by formula 7; the perceptual loss is used to measure the authenticity of the denoised image. Among them, L vgg is the perceptual loss of the VGG19 network, is the jth layer of the VGG19 network, H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, ∈~Ν(0,Ι) is the added random noise, θ is the parameter set of the denoising network, I t is the noisy image at step t, and t is the number of forward diffusion steps; The loss of the denoising network of the diffusion model is calculated by formula 8; L ddpm =L p2 +L vgg Formula 8; Among them, L ddpm is the loss of the denoising network of the diffusion model, L p2 is the loss of P2-weight, L vgg is the perceptual loss of the VGG19 network; The loss of the denoising network of the diffusion model is substituted into the denoising network to train the denoising network.

6. The fan component thermal failure early warning method according to claim 5, characterized in that: The method of calculating the loss of the multi-channel module based on the multi-channel texture loss and the multi-channel pixel loss, and training the multi-channel module using the loss of the multi-channel module, comprises: The multi-channel texture loss of the multi-channel module is calculated by formula 9; Among them, L MCG is the multi-channel texture loss of the multi-channel module, H is the height of the feature map, W is the width of the feature map, ▽ represents the derivative, I f is the fused image, I ir is the infrared image, I vis is a visible light image; The multi-channel pixel loss of the multi-channel module is calculated by formula 10; Among them, L MCI is the multi-channel pixel loss of the multi-channel module, H is the height of the feature map, W is the width of the feature map, and I f is the fused image, I ir is the infrared image, I vis is a visible light image, The structural similarity loss of the multi-channel module is calculated by formula 11; L SSIM = δ(1 - SSIM(I f , I vis )) + (1 - SSIM(I f , I ir ) Formula 11; Among them, L SSIM is the structural similarity loss, I f is the fused image, I ir is the infrared image, I vis is a visible light image, δ is a hyperparameter; The loss function of the multi-channel module is calculated by formula 12; L fusion =L MCG +L MCI +L SSIM Formula 12; Among them, L fusion is the loss function of the multi-channel module, L MCG is the multi-channel texture loss of the multi-channel module, L MCI is the multi-channel pixel loss of the multi-channel module, L SSIM is the structural similarity loss; Substitute the loss function of the multi-channel module into the multi-channel module to train the multi-channel module.

7. The fan component thermal failure early warning method according to claim 6, characterized in that: The first feature map is subjected to affine operation and aggregation operation through a cross attention module CCAM to obtain an enhanced feature map, and the enhanced feature map is added to the first feature map to obtain a final feature map, including: The fused image is input into a semantic segmentation network, and a first feature map and a second feature map of the fused image are output through an encoder of the semantic segmentation network; Performing dimensionality reduction on the first feature map to reduce the number of channels of the first feature map; The first feature map after dimensionality reduction is subjected to 1x1 convolution to obtain enhanced feature maps Q, K and V; Perform affine operation on the enhanced feature map Q and the enhanced feature map K by using formula 13; Among them, d i,u represents an affine operation, Q u ∈R C' Represents the element taken from the pixel at position u in the enhanced feature map Q, Ω u represents the set of elements of the same row and column taken from the enhanced feature map K and the same position u in the enhanced feature map Q, C' is the number of channels of the feature map of the fused image after the fusion image is reduced in dimension, H is the height of the feature map of the fused image, W is the width of the feature map of the fused image, d i,u ∈D represents Q u and between Ω u The degree of association, i represents the i-th element in the set, and i=[1,...,|Ω u |] R ,Ω u With Ω i,u relationship; Pass D through the softmax function to get the attention matrix A, A∈R (H+W-1)×W×H Aggregate the attention matrix A and the enhanced feature map V through formula 14; Among them, A i,u is the i-th element of all elements taken out at the pixel position u in the attention matrix A, Φ i,u is the set of all elements in the same row and column taken from the same position u in the enhanced feature map V and the attention matrix A, I u is the element at position u in the first feature map, I' u is the weighted adjusted value of the element at position u of the first feature map, and R represents the set of all real numbers; The enhanced feature map and fused image I f Add element by element to get the final feature map; The weighted value I′ of the element at position u of the first feature map is u Through the cross attention module CCAM module, the pixel at position u has the contextual information of all positions in the entire image.

8. The fan component thermal failure early warning method according to claim 7, characterized in that: The step of inputting the second feature map into the contour extraction module EEM and calculating the contour loss of the second feature map by the contour extraction module EEM comprises: The second feature map is input into the contour extraction module EEM, and the contour extraction module EEM reduces the number of channels of the second feature map and outputs the contour feature f of the second feature map e ; The contour feature f of the second feature map is processed by the Canny operator algorithm. e Perform contour extraction to obtain the contour feature f of the second feature map e The true value of the benchmark; The contour feature f of the second feature map is obtained by formula 15 e and the contour feature f of the second feature map e The contour feature f of the second feature map is calculated by the reference true value e Contour loss; Among them, L edge is the contour feature f of the second feature map e Contour loss, P(y i =1) is the contour feature f of the second feature map e The probability that the pixel at position i in is predicted to be a contour, P(y j =0) is the contour feature f of the second feature map e The probability that the pixel at position j is predicted to be background, and θ are weight coefficients.

9. The fan component thermal failure early warning method according to claim 8, characterized in that: The method of dividing the fused image into components according to the final feature map and the contour loss, and judging whether there are components with temperature classification errors in combination with the classification results of the infrared image, includes: Select an infrared image and a visible light image at the same time from the initial image database; By using K-means clustering, each component in the infrared image is divided into a low-temperature image portion, a medium-temperature image portion, and a high-temperature image portion; The infrared image and the visible light image are input into a diffusion network to obtain a fused image; The fused image is input into a semantic segmentation network, and multiple components contained in the fused image are segmented by the semantic segmentation network; Comparing the multiple components included in the fused image with the low-temperature image portion, the medium-temperature image portion, and the high-temperature image portion of each component in the infrared image to determine whether the regional temperature of each component meets the temperature of its corresponding region; If the temperature of a region of at least one component does not match the temperature of its corresponding region, it is determined that the temperature of the fan is abnormal.

10. A fan component thermal failure early warning system, applied to the fan component thermal failure early warning method according to claim 1 to claim 9, characterized in that: The fan component thermal failure early warning system comprises: A collection component, through which infrared images and visible light images of the target are collected; A processing component, wherein the acquisition component is in communication with the processing component, and the fan component thermal fault warning method according to claim 1 to claim 9 is executed on the acquired infrared image and visible light image through the processing component; The alarm component is in communication with the processing component. When the processing component determines that the fan temperature is abnormal, the processing component controls the alarm component to sound an alarm.

Images