Image-based power equipment temperature monitoring method, system and electronic equipment

Abstract

The present application discloses an image-based power equipment temperature monitoring method, system and electronic device, and discloses a method including: constructing a feature segmentation layer corresponding to a visible light image and an infrared image respectively and a modal fusion layer corresponding to a preset resolution based on a preset resolution and gated fusion, constructing an image fusion model with the feature segmentation layer, the modal fusion layer and the complexity parameter; obtaining historical power equipment temperature information corresponding to a historical visible light image and infrared image set, inputting the visible light image feature set and the infrared image feature set into the image fusion model, updating the image fusion model with the historical power equipment temperature information as output; constructing an image temperature recognition model with the updated image fusion model and the feature extraction model. The beneficial effects of the present application: enhancing the display of small area detail differences, ensuring that images with more information account for a larger proportion, avoiding information loss, and improving the accuracy of power equipment temperature monitoring.

Description

Technical Field

[0001] The present application relates to the technical field of temperature monitoring of electric power equipment, and in particular to an image-based temperature monitoring method, system and electronic equipment for electric power equipment. Background Art

[0002] As a key component of the power system, the temperature status of power equipment directly reflects the operating efficiency and safety of the equipment. With the advancement of science and technology, especially the rapid development of the Internet of Things, big data and artificial intelligence technologies, it has become possible to monitor the temperature of power equipment in real time, accurately and comprehensively. A real-time power equipment temperature monitoring method that integrates multimodal data has emerged. It aims to achieve accurate monitoring and early warning of the temperature of power equipment by integrating various types of data to ensure the safe and stable operation of the power system. If real-time monitoring cannot be carried out by operation and maintenance personnel alone, the experience and attention level of the operation and maintenance personnel will directly affect the accuracy of the detection, which may lead to the omission of important problems. Using infrared thermal imaging technology alone, since infrared images mainly reflect temperature distribution and lack the texture and color information of visible light images, it is more difficult to distinguish equipment with complex structures or high similarity.

[0003] Even when infrared images and visible light images are fused in related technologies, the overall image is fused, and small area details are easily lost. However, in actual situations, the amount of information contained in infrared images and visible light images is not equal. Simply fusing with a fixed fusion weight can easily cause information loss.

[0004] The patent "Electric Power Equipment Detection Method, Device, Equipment and Medium Based on Infrared Temperature Measurement", publication number: CN118670526A, publication date: September 20, 2024, specifically discloses: obtaining infrared photos of electric power equipment in real scenes; performing temperature mapping based on the infrared photos and the virtual restoration three-dimensional model of the electric power equipment to obtain a temperature mapping three-dimensional model, and the virtual restoration three-dimensional model is digitally restored according to the actual survey of electric power equipment parts data; performing temperature detection based on the temperature mapping three-dimensional model to determine the hidden danger parts of the electric power equipment. Although this solution successfully combines infrared temperature measurement technology with three-dimensional modeling technology to achieve accurate positioning of hidden danger parts of electric power equipment, the construction of a virtual restoration three-dimensional model requires a large amount of electric power equipment parts data, including actual installation part information and geographic coordinates, etc. These data are difficult to obtain and need to be updated regularly to ensure accuracy. The cost of collecting data is high and the data processing is complex.

[0005] Patent "Infrared temperature anomaly detection method for insulator strings of power equipment based on instance segmentation", publication number: CN118470051A, publication date: August 9, 2024, specifically discloses that the infrared image of the insulator string is enhanced by DDE technology, and then segmented using the YOLACT instance segmentation model to obtain the insulator string area image with the background removed; the radial temperature distribution characteristic curve of the n insulator piece image area is drawn by infrared temperature measurement technology; the insulator string area image with the background removed is trained using the isolation forest algorithm to obtain an isolation forest model for calculating the anomaly score of the insulator piece; the radial temperature distribution characteristic curve of the n insulator piece image area is fitted to obtain a fitting curve, and the determination coefficient R2 of the fitting curve is used to comprehensively judge whether the insulator string has temperature anomaly based on the determination coefficient R2 and the anomaly score of the insulator piece. Although the YOLACT instance segmentation model and the isolation forest algorithm have excellent performance, their algorithm complexity is high, which may lead to slow processing speed in practical applications and cannot be used on front-end devices. Summary of the invention

[0006] In order to solve the problem that the temperature monitoring of power equipment in the prior art easily overlooks the details of small areas and ignores the information loss caused by the inequality of image information, the present application provides an image-based temperature monitoring method, system and electronic device for power equipment. By constructing a feature segmentation layer and a modal fusion layer corresponding to a preset resolution, the image is segmented into different resolution forms for fusion processing to enhance the display of small area detail differences. At the same time, the complexity parameter is used to perform adaptive adjustment according to the information capacity of the infrared image and the visible light image, ensuring that the image with more information has a greater weight, avoiding information loss, and improving the accuracy of the temperature monitoring of power equipment. There is no need to use more complex algorithms, and the application is wider.

[0007] To achieve the above-mentioned technical objectives, a technical solution provided by the present application is an image-based power equipment temperature monitoring method, comprising the following steps: obtaining historical visible light images and infrared image sets, training a feature extraction model with the historical visible light images and infrared image sets, and obtaining a feature extraction model and a visible light image feature set and an infrared image feature set; constructing a feature segmentation layer corresponding to the visible light image and the infrared image respectively and a modal fusion layer corresponding to the preset resolution based on a preset resolution and gated fusion, and constructing an image fusion model with the feature segmentation layer, the modal fusion layer and the complexity parameter; obtaining historical power equipment temperature information corresponding to the historical visible light images and infrared image sets, inputting the visible light image feature set and the infrared image feature set into the image fusion model, and updating the image fusion model with the historical power equipment temperature information as the output; constructing an image temperature recognition model with the updated image fusion model and the feature extraction model, and outputting real-time power equipment temperature information using the image temperature recognition model with real-time visible light images and infrared images.

[0008] Furthermore, the training of the feature extraction model with the historical visible light images and infrared image sets also includes: constructing parallel CNN streams corresponding to the visible light images and the infrared images, respectively, to obtain a feature extraction model based on a dual-stream network; inputting the aligned historical visible light images and historical infrared images into the feature extraction model based on the dual-stream network, and performing feature extraction model training.

[0009] Furthermore, the feature segmentation layers corresponding to the visible light image and the infrared image, respectively, and the modal fusion layer corresponding to the preset resolution are constructed based on the preset resolution and gated fusion, and also include: constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image based on the preset resolution; and constructing a modal fusion layer for the visible light feature segmentation layer and the infrared feature segmentation layer of the same preset resolution based on gated fusion.

[0010] Furthermore, the method of constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image based on the preset resolution also includes: obtaining a maximum preset resolution based on a maximum historical image resolution; constructing a preset ratio based on the ratio of the temperature influence area of ​​the power equipment to the overall area of ​​the power equipment; obtaining an intermediate preset resolution based on the maximum preset resolution and the preset ratio; and constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image with the maximum preset resolution and the intermediate preset resolution.

[0011] Furthermore, constructing an image fusion model using a feature segmentation layer, a modal fusion layer and complexity parameters includes: constructing a complexity parameter based on an image average gradient; and constructing an image fusion model using a feature segmentation layer, a modal fusion layer and complexity parameters.

[0012] Furthermore, it also includes: building an initial network model with yolov8s; replacing the backbone network of yolov8s with MobileNetV4-Conv-m as the backbone network for extracting features, and building a feature extraction model.

[0013] Furthermore, it also includes: obtaining a first fluctuation based on text information in the real-time temperature information of the power equipment and text information within a preset time sequence; obtaining a second fluctuation based on fused image information in the real-time temperature information of the power equipment and fused image information within a preset time sequence; judging whether the temperature of the power equipment is abnormal based on the first fluctuation and the second fluctuation, and if so, executing an alarm.

[0014] Furthermore, the method of obtaining the first fluctuation based on the text information in the real-time power equipment temperature information and the text information within the preset time sequence includes: obtaining a temperature trend curve based on the real-time power equipment temperature coordinates and the power equipment temperature coordinates within the preset time sequence; obtaining a temperature change curve corresponding to the structure type based on the structure type corresponding to the real-time power equipment temperature coordinates and the structure type corresponding to the power equipment temperature coordinates within the preset time sequence, the real-time power equipment temperature and the power equipment temperature within the preset time sequence; and taking the temperature trend curve and the temperature change curve corresponding to the structure type as the first fluctuation.

[0015] Furthermore, the method of judging whether the temperature of the power equipment is abnormal based on the first fluctuation and the second fluctuation includes: if the fault range of the temperature trend curve is greater than a preset range threshold and / or the temperature change amplitude of the temperature change curve corresponding to the structural type is greater than a preset amplitude threshold, then it is considered that the first fluctuation is greater than the first fluctuation threshold and the temperature of the power equipment is abnormal.

[0016] Furthermore, the step of inputting the visible light image feature set and the infrared image feature set into the image fusion model and updating the image fusion model with the historical power equipment temperature information as the output also includes: taking a preset resolution as the segmentation target, segmenting each feature image in the visible light image feature set and the infrared image feature set, and assigning them to the corresponding feature segmentation layer; clustering the feature images in the same feature segmentation layer using a clustering algorithm; outputting the clustering results to the corresponding modal fusion layer; the modal fusion layer performs fusion of the visible light feature image and the infrared feature image; if the fused image information output by the modal fusion layer is inconsistent with the historical power equipment temperature information, updating the image fusion model to repeat the fusion steps until the fused image information output by the modal fusion layer is consistent with the historical power equipment temperature information, and the update is completed.

[0017] Another technical solution provided by the present application is an image-based power equipment temperature monitoring system for implementing the method as described above, including: a data acquisition unit for acquiring visible light images and infrared images; a model building unit for building an image temperature recognition model based on a historical visible light image set and an infrared image set; a complexity parameter adjustment unit for adjusting complexity parameters based on the complexity of the visible light image and the infrared image; and a temperature analysis unit for outputting real-time power equipment temperature information based on a fused image output by the image temperature recognition model.

[0018] Another technical solution provided by the present application is an electronic device, which includes a processor, a memory and a battery module; the memory is used to store programs; the battery module is used to power the memory; the processor is used to execute the program and implement the image-based power equipment temperature monitoring method as described above when executing the program.

[0019] Another technical solution provided by the present application is a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processing device, the above-mentioned image-based temperature monitoring method for power equipment is implemented.

[0020] The beneficial effects of the present application are as follows: 1. By constructing a feature segmentation layer and a modal fusion layer corresponding to a preset resolution, the image is segmented into different resolution forms for fusion processing to enhance the display of detail differences in small areas. At the same time, the complexity parameter is used to perform adaptive adjustment according to the information capacity of the infrared image and the visible light image to ensure that images with more information have a greater weight, avoid information loss, and improve the accuracy of temperature monitoring of power equipment. The image feature extraction algorithm, image segmentation algorithm, image fusion algorithm, and clustering algorithm are all conventional algorithms, and there is no need to use more complex algorithms, so they are more applicable.

[0021] 2. The preset ratio is constructed by the ratio of the temperature-affected area of ​​the power equipment to the overall area of ​​the power equipment, so that there must be a feature segmentation layer corresponding to the size of the temperature-affected area of ​​the power equipment, which can improve the accuracy of monitoring the temperature-affected area of ​​the power equipment and further improve the accuracy of temperature monitoring. At the same time, a difference percentage threshold is set to construct a feature segmentation layer with a representative preset ratio, which improves the accuracy while ensuring the efficiency of temperature information output.

[0022] 3. Add complexity parameters related to the complexity of image information. When training the image fusion model, the information differences have been adaptively adjusted through complexity parameters when fusion training is performed for different pairs of images. Therefore, the image fusion model training will not be affected by the usage scenario, making the image fusion model training results more accurate.

[0023] 4. Use yolov8s as the initial model of the image temperature recognition model, use MobileNetV4-Conv-m as the backbone network for extracting features to replace the backbone network of yolov8s, and use the Neck and Head parts of yolov8s as the image fusion model part to achieve accurate output of temperature information and location information of power equipment, while improving the lightweight of the network and being able to be applied to mobile devices.

[0024] 5. The information in the image that cannot be semantically recognized is compensated through image difference fluctuations. When the difference of the remaining information that cannot be semantically recognized is too large, it is considered that there is a large impact on the current temperature monitoring of the power equipment, and it is suspected that the accuracy of the current monitoring results is low. The alarm is also executed to prompt the operation and maintenance personnel to feedback the real-time information of the power equipment according to the actual status or to correct the image fusion recognition model.

[0025] 6. There is a fault range in the temperature trend curve. The temperature within the fault range does not conform to the law of temperature radiation. It is impossible for a certain structure of the power equipment to have a temperature mutation in the adjacent time series. Once the temperature change curve of a certain structure fluctuates beyond the preset amplitude threshold, there is a temperature mutation, and the power equipment temperature abnormality prompt is issued. Two-way judgment further improves the accuracy of temperature monitoring.

[0026] 7. When fusing the visible light feature image with the infrared feature image, traversal is performed directly based on the clustering results, that is, the infrared feature image is traversed with the feature image in a cluster of the visible light feature image. After obtaining the corresponding infrared feature image, it is considered that the feature images in the two clusters correspond. At this time, the feature images in the cluster only traverse each other, and no longer traverse other clusters. After all clusters are traversed, the remaining unmatched feature images are traversed and matched to match the corresponding visible light feature image and infrared feature image, thereby reducing the number of traversals during fusion and improving the accuracy and efficiency of fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the image-based temperature monitoring method for power equipment of the present application.

[0028] Figure 2 The figure is a structural diagram of an image temperature recognition model based on an image in one embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in conjunction with the drawings and examples. It should be understood that the specific implementation method described here is only an optimal embodiment of the present application, which is only used to explain the present application and does not limit the scope of protection of the present application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0030] like Figure 1 As shown, as the first embodiment of the present application, the image-based power equipment temperature monitoring method includes the following steps:

[0031] Obtain historical visible light images and infrared image sets, train a feature extraction model with the historical visible light images and infrared image sets, and obtain a feature extraction model and a visible light image feature set and an infrared image feature set;

[0032] Based on the preset resolution and gated fusion, feature segmentation layers corresponding to visible light images and infrared images and modality fusion layers of corresponding resolutions are constructed, and an image fusion model is constructed with the feature segmentation layer, modality fusion layer and complexity parameters;

[0033] Acquire historical power equipment temperature information corresponding to historical visible light images and infrared image sets, input the visible light image feature set and the infrared image feature set into the image fusion model, and update the image fusion model with the historical power equipment temperature information as output;

[0034] An image temperature recognition model is constructed using the updated image fusion model and feature extraction model, and the image temperature recognition model is used to output real-time power equipment temperature information using real-time visible light images and infrared images.

[0035] In this embodiment, a feature extraction model is trained by using historical visible light images and infrared image sets to ensure the accuracy of feature extraction, and then feature segmentation layers of each resolution of visible light images, feature segmentation layers of each resolution of infrared images, and a modal fusion layer that fuses images of each resolution are constructed based on a preset resolution. The feature segmentation layer and the modal fusion layer are adjusted along with the visible light image feature set and the infrared image feature set based on a complexity parameter, so that the final fused image information is closer to the actual image information. At the same time, since both the feature segmentation layer and the modal fusion layer are constructed based on a preset resolution, more details of the image can be obtained according to different resolutions, the features of each part of the image can be refined, and the influence of image information can be considered at the same time, thereby further improving the accuracy of temperature monitoring of power equipment.

[0036] Firstly, historical visible light images taken by visible light cameras and historical infrared images taken by infrared cameras are obtained, and data preprocessing is performed on the historical visible light images and historical infrared images. Based on the time series, the preprocessed historical visible light images and historical infrared images are matched one by one to obtain a set of historical visible light images and infrared images.

[0037] Specifically, obtaining a set of historical visible light images and infrared images, and training a feature extraction model with the set of historical visible light images and infrared images includes:

[0038] Acquire historical visible light images and historical infrared images;

[0039] Aligning historical visible light images and historical infrared images according to spatial information;

[0040] Align historical visible light images and historical infrared images according to time series information;

[0041] Perform data normalization on historical visible light images and historical infrared images;

[0042] The feature extraction model is trained using the processed historical visible light images and historical infrared images.

[0043] Align historical visible light images and historical infrared images according to spatial information and temporal information to ensure that the visible light and infrared image pairs are strictly aligned in space and time for subsequent feature extraction and matching. Normalize historical visible light images and historical infrared images, such as 0-1 or -1 to 1, to facilitate subsequent model training. The feature extraction model can use a convolutional neural network (CNN) model, and obtain the feature extraction model by fine-tuning or training all weights.

[0044] In this embodiment, the feature extraction model is a two-stream network model, and training the feature extraction model with historical visible light images and infrared image sets also includes:

[0045] Construct parallel CNN streams corresponding to visible light images and infrared images, respectively, to obtain a feature extraction model based on a dual-stream network;

[0046] The aligned historical optical image and the historical infrared image are input into a feature extraction model based on a two-stream network to perform feature extraction model training.

[0047] Since the historical visible light images and historical infrared images have been aligned in time and space in the early stage, they appear in pairs at this time. The CNN stream corresponding to the visible light images focuses on processing the visible light images and extracting the appearance features and detail information of the power equipment. The CNN stream corresponding to the infrared images focuses on processing the infrared images and extracting the thermal features and temperature distribution information of the power equipment. The two CNN streams process the paired images in parallel to ensure that the output feature vectors are aligned in time and space, while improving the feature extraction efficiency and ensuring the real-time performance of temperature monitoring.

[0048] After the feature extraction model training is completed, a visible light image feature set corresponding to the historical visible light images and an infrared image feature set corresponding to the historical infrared images are obtained.

[0049] Synchronously constructing a feature segmentation layer and a modal fusion layer based on resolution and gated fusion. Specifically, constructing a feature segmentation layer corresponding to a visible light image and an infrared image and a modal fusion layer corresponding to a preset resolution based on a preset resolution and gated fusion also includes:

[0050] Constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image based on a preset resolution;

[0051] Based on gated fusion, a modal fusion layer is constructed for the visible light feature segmentation layer and the infrared feature segmentation layer with the same preset resolution.

[0052] The preset resolution can be set based on experience, such as setting the preset resolution to 160 160 resolution, 80 80 resolution, 40 40 resolution. At this time, the visible light image feature segmentation layer corresponds to 160 160 resolution visible light feature segmentation layer, 80 80 resolution visible light feature segmentation layer, 40 The visible light feature segmentation layer has a resolution of 40; the infrared image feature segmentation layer has a resolution of 160 160 resolution infrared image feature segmentation layer, 80 80 resolution infrared image feature segmentation layer, 40 40 resolution infrared image feature segmentation layer; the modality fusion layer corresponds to 160 160 resolution modal fusion layer, 80 80 resolution modal fusion layer, 40 Modality fusion layer at 40 resolution.

[0053] In other cases, constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image based on a preset resolution further includes:

[0054] Get the maximum preset resolution based on the maximum historical image resolution;

[0055] Obtaining an intermediate preset resolution based on the maximum preset resolution and the preset ratio;

[0056] A visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image are constructed with a maximum preset resolution and an intermediate preset resolution.

[0057] In this case, a fixed resolution is not set, but an intermediate preset resolution is obtained by using the maximum preset resolution and the preset ratio, and the maximum preset resolution is obtained by using the maximum historical image resolution, which is the maximum image resolution of the historical visible light image and the historical infrared image. Multiple preset ratios can be set, corresponding to multiple intermediate preset resolutions, and a feature segmentation layer is constructed according to the maximum preset resolution and the intermediate preset resolution, thereby improving the accuracy of image analysis and further improving the accuracy of temperature monitoring.

[0058] In other embodiments, the preset ratio is set according to the actual situation of the power equipment. At this time, the visible light feature segmentation layer corresponding to the visible light image and the infrared feature segmentation layer corresponding to the infrared image are constructed based on the preset resolution and further include:

[0059] Get the maximum preset resolution based on the maximum historical image resolution;

[0060] Establishing a preset ratio based on the ratio of the temperature impact area of ​​the power equipment to the overall area of ​​the power equipment;

[0061] Obtaining an intermediate preset resolution based on the maximum preset resolution and the preset ratio;

[0062] A visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image are constructed with a maximum preset resolution and an intermediate preset resolution.

[0063] The preset ratio is constructed by the ratio of the temperature influence area of ​​the power equipment to the overall area of ​​the power equipment, so that the calculated intermediate preset resolution includes the temperature influence area of ​​the power equipment. For example, it is assumed that a power equipment is captured in a visible light image / infrared image, but the temperature influence area of ​​the power equipment only covers 1 / 24 of the overall area of ​​the power equipment. If the overall temperature state of the power equipment is obtained by fusion of the entire visible light image features and infrared image features, it is easy to ignore the details in the temperature influence area of ​​the power equipment, thereby causing errors or omissions in temperature monitoring. In this embodiment, the preset ratio is constructed by the ratio of the temperature influence area of ​​the power equipment to the overall area of ​​the power equipment, so that there must be a feature segmentation layer corresponding to the size of the temperature influence area of ​​the power equipment, which can improve the accuracy of monitoring the temperature influence area of ​​the power equipment and further improve the accuracy of temperature monitoring. In the case of monitoring multiple power equipment at the same time, several preset ratios are constructed based on the ratio of the power equipment temperature influence area of ​​each power equipment to the overall area of ​​the power equipment. At this time, a corresponding table of power equipment and preset ratios can be constructed in advance, and the corresponding preset ratios can be directly retrieved according to the power equipment in the image. In the preset ratio corresponding table, the preset ratio difference percentage is at least greater than 5%. Of course, in other cases, the difference percentage threshold can be set by itself. Here, it is only to avoid the preset ratios with small differences leading to the additional construction of multiple feature segmentation layers. For example, if the two preset ratios differ by only 1%, the feature segmentation layer constructed with any preset ratio can effectively reflect the required regional details without the need to construct two feature segmentation layers. Therefore, the difference percentage threshold is set, and the feature segmentation layer is constructed with a representative preset ratio, which improves the accuracy while ensuring the efficiency of temperature information output. For example, when the difference percentage between multiple preset ratios is less than or equal to the difference percentage threshold, the mean can be used as the preset ratio.

[0064] Furthermore, a modal fusion layer is constructed based on gated fusion for the visible light feature segmentation layer and the infrared feature segmentation layer of the same preset resolution. Gated fusion is a feature fusion strategy that uses a gated unit to control the weights of different modal data, automatically learning and fusing information from different sources. In this embodiment, a sigmoid function is used to control the fusion weight coefficient of the output data of the visible light feature segmentation layer and the infrared feature segmentation layer, and a modal fusion layer is provided for each preset resolution, thereby highlighting the fusion details of each area size, avoiding ignoring the fusion features of small areas during overall fusion, and further improving the accuracy of monitoring the temperature of power equipment through infrared images and visible light images.

[0065] It can be understood that although the visible light feature segmentation layer, infrared feature segmentation layer, and modal fusion layer are used for description in this embodiment, it does not mean that the processing of each feature data must be performed in different layers of the neural network. Visible light feature segmentation and infrared feature segmentation can perform data separation and fusion at the same level, and for model parallel processing, visible light feature segmentation and infrared feature segmentation can also be processed at two parallel processing levels, or the number of parallel lines can be obtained by using visible light feature segmentation, infrared feature segmentation, and resolution number, and parallel processing at multiple levels can be performed to improve the effect of model processing.

[0066] The sigmoid function is used to control the fusion weight coefficient of the output data of the visible light feature segmentation layer and the infrared feature segmentation layer:

[0067] ;

[0068] in, is the fusion weight coefficient, b is the bias term, It is a sigmoid function, x is the visible light feature, y is the infrared feature, and gate is the fused image information.

[0069] In other cases, a fully connected layer is used to fuse the visible light feature image and the infrared feature image, and a fully connected layer is set corresponding to the visible light feature image. , set the fully connected layer corresponding to the infrared feature image :

[0070] .

[0071] The image fusion model is constructed with feature segmentation layer, modality fusion layer and complexity parameters, including:

[0072] Construct complexity parameters based on the average gradient of the image;

[0073] The image fusion model is constructed with feature segmentation layer, modality fusion layer and complexity parameters.

[0074] In different usage scenarios, the importance of different image information is not consistent. For example, in poor light at night, visible light images lack texture information and various image features. In fact, thermal imaging image information is more important. Therefore, complexity parameters related to the complexity of image information are added. When training the image fusion model, the information difference has been adaptively adjusted through complexity parameters during fusion training for different pairs of images. Therefore, the fusion weight coefficient training of the image fusion model will not be affected by the usage scenario, making the image fusion model training results more accurate.

[0075] In this embodiment, the complexity parameter constructed based on the image average gradient is:

[0076] ;

[0077] ;

[0078] ;

[0079] in, is the average gradient of feature map F, M is the number of rows of feature map F, N is the number of columns of feature map F, i is the row index of feature map F, j is the column index of feature map F, is the complexity parameter of the visible light feature, is the complexity parameter of infrared characteristics, is the average gradient of the visible light feature image, is the average gradient of the infrared feature image.

[0080] The average gradient is an important indicator of image clarity or detail contrast expression. The larger the average gradient, the more layers in the image, the richer the details, and the higher the information complexity of the image.

[0081] The image fusion model is constructed with feature segmentation layer, modality fusion layer and complexity parameters as follows:

[0082] .

[0083] During the training process, the historical power equipment temperature information corresponding to the historical visible light image and infrared image set is obtained, the visible light image feature set and the infrared image feature set are input into the image fusion model, and the image fusion model is updated with the historical power equipment temperature information as the output. The historical power equipment temperature information is the true value of the power equipment in the historical time series, that is, the fused image information output by the image fusion model should be consistent with the historical power equipment temperature information. During the training process, the optimization algorithm and loss function can be used to adjust the fusion weight coefficient according to the error between the predicted output of the image fusion model and the historical power equipment temperature information. Through continuous iterative training, the accuracy and reliability of the image fusion model can be improved.

[0084] In some cases, the historical power equipment temperature information is a historical power equipment temperature annotation map constructed by setting legends according to the power equipment temperature conditions such as temperature, image coordinates, and occurrence area. In this case, the image fusion model includes a legend annotation layer. In the process of training the image fusion model, the corresponding legend annotation is matched according to the feature conditions, and the visible light features and infrared features are fused to make their fused image consistent with the historical power equipment temperature annotation map. At this time, the output result of the image fusion model is a fused image, so that the operation and maintenance personnel can intuitively obtain the power equipment temperature information, that is, directly obtain the temperature anomaly position from the temperature distribution and equipment details on the image, so as to intuitively locate the power equipment temperature and improve the efficiency of subsequent temperature anomaly processing.

[0085] In other cases, the historical temperature information of power equipment is text information, i.e., text information such as power equipment coordinate points, temperature, and power equipment structure type. At this time, the image fusion model also includes a semantic recognition layer. By corresponding to the text information and fusing the image information, the image fusion model can directly output the temperature text information and the corresponding temperature coordinates of the power equipment, which is convenient for temperature data statistics.

[0086] Of course, in other cases, the image fusion model may include a semantic recognition layer and a legend annotation layer. In this case, the image fusion model outputs text information and fused image information with legend annotations, and the historical power equipment temperature information also includes text information and a historical power equipment temperature annotation map constructed by setting legends according to power equipment temperature conditions such as temperature, image coordinates, and occurrence area. In this case, it can ensure that the operation and maintenance personnel can intuitively obtain the temperature status of the power equipment and directly perform temperature data statistics for a period of time.

[0087] Thus, an image temperature recognition model is constructed with the updated image fusion model and feature extraction model, and the real-time power equipment temperature information is output using the image temperature recognition model with real-time visible light images and infrared images. By inputting the real-time visible light image and infrared image into the image temperature recognition model, the real-time power equipment temperature information is output according to the pixel temperature information and position information in the output fusion image.

[0088] As a second embodiment of the present application, the image-based power equipment temperature monitoring method further includes: before the feature extraction model training, there is:

[0089] The feature extraction model is constructed by replacing the backbone network of the initial network model with a lightweight network.

[0090] In order to adapt to real-time reasoning on low-computing front-ends, the backbone network of the initial network model is replaced with a lightweight network to improve the efficiency of model extraction.

[0091] In this embodiment, replacing the initial network model backbone with a lightweight network to construct a feature extraction model includes:

[0092] Use yolov8s to build the initial network model;

[0093] Use MobileNetV4-Conv-m as the backbone network for extracting features to replace the backbone network of yolov8s and build a feature extraction model.

[0094] In other embodiments, lightweight networks such as MobileNet and EfficientNet can also be used as backbone networks for extracting features. MobileNetV4-Conv-m is used as the backbone network for extracting features, so that it is easy to use on mobile devices. Among them, the output feature map size and number of channels of MobileNetV4-Conv-m must match the output of the original backbone network of yolov8s.

[0095] like Figure 2 As shown in the figure, yolov8s can also be used as the initial model of the image temperature recognition model, MobileNetV4-Conv-m can be used as the backbone network for extracting features to replace the backbone network of yolov8s, and the Neck and Head parts of yolov8s can be used as the image fusion model part, so as to achieve accurate output of temperature information and location information of power equipment, and can be applied to mobile devices at the same time.

[0096] The Neck part of yolov8s is an improved version based on the feature pyramid network (FPN), which is used to fuse feature maps of different scales as the visible light feature segmentation layer, infrared feature segmentation layer, and modal fusion layer in the image fusion model. The Head part of yolov8s separates the regression branch and the classification branch, and corresponds the fused feature map to the text information. The regression branch will output the bounding box coordinates of the target, including the center point position, width, height and other information, that is, the position information corresponding to the temperature of the power equipment. The classification branch will output the probability that the target belongs to each category, which is used to judge the category attribute of the target, that is, the structural category of the power equipment.

[0097] In this embodiment, the image-based power equipment temperature monitoring method further includes:

[0098] Acquire a first fluctuation according to text information in the real-time power equipment temperature information and text information in a preset time sequence;

[0099] Acquire a second fluctuation according to the fused image information in the real-time power equipment temperature information and the fused image information in a preset time sequence;

[0100] Whether the temperature of the power equipment is abnormal is determined based on the first fluctuation and the second fluctuation. If so, an alarm is issued.

[0101] The text information at least includes the temperature coordinates of the power equipment, the structure type corresponding to the temperature coordinates of the power equipment, and the temperature of the power equipment. The first fluctuation includes at least the coordinate fluctuation, the structure type fluctuation, and the temperature fluctuation. The second fluctuation includes at least the image difference fluctuation, and the difference value fluctuation is obtained by the difference value of two fused images connected in time series, and the difference value fluctuation is used as the image difference fluctuation.

[0102] Judging whether the temperature of the power equipment is abnormal according to the first fluctuation and the second fluctuation includes:

[0103] If the first fluctuation is greater than the first fluctuation threshold and / or the second fluctuation is greater than the second fluctuation threshold, it is considered that the temperature of the power equipment is abnormal.

[0104] The change in the temperature coordinates of the power equipment, the change in the structural type corresponding to the temperature coordinates of the power equipment, and the change in the temperature of the power equipment each correspond to a first fluctuation threshold. When any change exceeds the first fluctuation threshold, it is considered that the first fluctuation is greater than the first fluctuation threshold. The first fluctuation threshold can be preset according to actual conditions, or it can be used as the first fluctuation threshold based on the maximum thresholds of each change in the historical normal temperature sequence of the power equipment. The second fluctuation threshold can be preset according to actual conditions, or it can be used as the second fluctuation threshold based on the maximum threshold of the image change in the historical normal temperature sequence of the power equipment. The information in the image that cannot be semantically recognized is compensated by image difference fluctuations. When the difference in the remaining information that cannot be semantically recognized is too large, it is considered that the current temperature monitoring of the power equipment has a large impact, and it is suspected that the accuracy of the current monitoring results is low. An alarm is also executed to prompt the operation and maintenance personnel to feedback the real-time information of the power equipment according to the actual status or to correct the image fusion recognition model.

[0105] In other cases, obtaining the first fluctuation according to the text information in the real-time power equipment temperature information and the text information in the preset time sequence includes:

[0106] Obtain temperature trend curve based on real-time power equipment temperature coordinates;

[0107] Based on the structure type corresponding to the real-time power equipment temperature coordinates and the structure type corresponding to the power equipment temperature coordinates in a preset time sequence, the real-time power equipment temperature and the power equipment temperature in a preset time sequence, obtaining a temperature change curve corresponding to the structure type;

[0108] The temperature profile and the temperature change curve corresponding to the structural type are used as the first fluctuation.

[0109] The temperature trend curve shows the radiation change of the temperature area of ​​the power equipment, and the temperature change curve corresponding to the structure type shows the change of the structure temperature of the power equipment. Therefore, judging whether the temperature of the power equipment is abnormal based on the first fluctuation and the second fluctuation includes:

[0110] If the fault range of the temperature trend curve is greater than the preset range threshold and / or the temperature change amplitude of the temperature change curve corresponding to the structure type is greater than the preset amplitude threshold, it is considered that the first fluctuation is greater than the first fluctuation threshold and the temperature of the power equipment is abnormal.

[0111] Since temperature cannot appear or disappear suddenly, and temperature radiation cannot produce excessive fault radiation, if there is a fault range in the temperature trend curve, the temperature in the fault range does not conform to the temperature radiation law, and the fault range is greater than the preset range threshold, then considering that the deviation of some small data is not enough to affect the normal operation of the power equipment, it is considered that the temperature data is abnormal. Similarly, a certain structure of the power equipment cannot have a sudden temperature change in adjacent time series. Once the temperature change curve of a certain structure fluctuates beyond the preset amplitude threshold, there is a sudden temperature change, and it is necessary to prompt the power equipment temperature abnormality.

[0112] As a third embodiment of the present application, the visible light image feature set and the infrared image feature set are input into the image fusion model, and the image fusion model is updated with the historical power equipment temperature information as output, and further includes:

[0113] Taking the preset resolution as the segmentation target, each feature image in the visible light image feature set and the infrared image feature set is segmented and assigned to the corresponding feature segmentation layer;

[0114] Clustering the feature images in the same feature segmentation layer using a clustering algorithm;

[0115] Output the clustering results to the corresponding modality fusion layer;

[0116] The modality fusion layer performs the fusion of the visible light feature image and the infrared feature image;

[0117] If the fused image information output by the modal fusion layer is inconsistent with the historical power equipment temperature information, the image fusion model is updated to repeat the fusion steps until the fused image information output by the modal fusion layer is consistent with the historical power equipment temperature information, and the update is completed.

[0118] In this embodiment, the visible light feature image and the infrared feature image are segmented according to a preset resolution, such as dividing the 360 The 360 ​​visible light feature image and infrared feature image are divided into 160 160 resolution, 80 80 resolution, 40 The feature images with 40 resolution are clustered by clustering algorithms such as K-means and hierarchical clustering. The image blocks with similar features are divided into the same cluster. The clustering results are output to the modal fusion layer. The visible light feature image and the infrared feature image are fused according to the clustering results. The fusion weight coefficient of the image fusion model is adjusted according to the historical temperature information of the power equipment until the output fused image information is consistent with the historical temperature information of the power equipment. Among them, the feature image is first segmented, and then cluster analysis is performed for different segmentation sizes to improve the accuracy of analysis of local area features of the image. At the same time, clustering is performed using a clustering algorithm to obtain different cluster clusters in the same area. Since images in the same area have a higher similarity, images in the same area are more likely to be clustered in the same cluster. Therefore, when fusing the visible light feature image with the infrared feature image, traversal can be performed directly based on the clustering result, that is, the infrared feature image is traversed with a feature image in a cluster of the visible light feature image. After obtaining the corresponding infrared feature image, it is considered that the feature images in the two cluster clusters correspond. At this time, the feature images in the cluster cluster only traverse each other, and no longer traverse other cluster clusters. When all cluster clusters are traversed, the remaining unmatched feature images are traversed and matched to match the corresponding visible light feature image and infrared feature image, thereby reducing the number of traversals during fusion and improving the accuracy and efficiency of fusion.

[0119] Among them, since there are multiple modal fusion layers, the update is considered to be completed only when the modal fusion layer corresponding to any preset resolution reaches the point where the fused image information output by the modal fusion layer is consistent with the historical power equipment temperature information. Since the visible light feature image and the infrared feature image are fused first, the local detail features are reflected during the fusion, and the reflected local detail features will not be lost after the overall image fusion. For example, some mutation points are easily ignored due to their small proportion in the overall fusion, and the local fusion can reflect whether the mutation points of the visible light feature image and the infrared feature image correspond to each other. If so, the mutation point can be displayed without being ignored. At this time, when the real-time visible light image and the real-time infrared image are input into the image temperature recognition model, the modal fusion layers corresponding to different resolutions output information of different area sizes. At this time, not only a large range of temperature information can be obtained, but also a small range of temperature information can be obtained, which is convenient for distinguishing different temperature abnormalities. For example, in some cases, an abnormal recognition model can be constructed based on abnormal information and temperature distribution, and the temperature information of different size areas can be input into the abnormal recognition model, so as to intuitively and accurately obtain the abnormal points of power equipment failure, and facilitate maintenance by operation and maintenance personnel. In other cases, large-scale temperature recognition results may ignore extremely small temperature anomalies. At this time, when the temperature recognition information output by any modal fusion layer is abnormal, a temperature anomaly alarm is executed to ensure the accuracy of temperature monitoring while taking into account local anomalies and overall abnormal conditions. At the same time, the image temperature recognition model is updated according to the judgment results of the operation and maintenance personnel. For example, if the operation and maintenance personnel judge that the current temperature of the power equipment is normal, the modal fusion layer and the corresponding feature segmentation layer with normal output temperature recognition information are frozen, and the modal fusion layer and the feature segmentation layer with abnormal output temperature recognition are adjusted according to the judgment results. Otherwise, the modal fusion layer and the corresponding feature segmentation layer with abnormal output temperature recognition information are frozen, and the modal fusion layer and the feature segmentation layer with normal output temperature recognition are adjusted according to the judgment results.

[0120] In other embodiments, secondary fusion may be performed based on the fused image output by the modal fusion layer, that is, all fusion information is reflected in one fused image, so that operation and maintenance personnel can intuitively obtain the temperature information of the power equipment.

[0121] As a fourth embodiment of the present application, an image-based power equipment temperature monitoring system includes:

[0122] A data acquisition unit, used for acquiring visible light images and infrared images;

[0123] A model building unit, used for building an image temperature recognition model based on a historical visible light image set and an infrared image set;

[0124] A complexity parameter adjustment unit, used for adjusting the complexity parameter based on the complexity of the visible light image and the infrared image;

[0125] The temperature analysis unit is used to output real-time power equipment temperature information based on the fused image information output by the image temperature recognition model.

[0126] The data acquisition unit includes at least a visible light camera and an infrared camera. Some power systems are equipped with visible light cameras and infrared cameras. In this case, the visible light cameras and infrared cameras of the power system can be directly connected to reduce the redundancy of equipment settings.

[0127] The temperature analysis unit can be an interactive unit that directly displays the fused image information output by the image temperature recognition model to the operation and maintenance personnel, or it can be equipped with an anomaly recognition model to output an anomaly recognition result based on the fused image information.

[0128] As the fifth embodiment of the present application, the electronic device includes a processor, a memory and a battery module; the memory is used to store programs; the battery module is used to power the memory; the processor is used to execute the program and implement the image-based power equipment temperature monitoring method as described above when executing the program.

[0129] As the sixth embodiment of the present application, a computer-readable storage medium is used to store a computer program or instruction. When the computer program or instruction is executed by a processing device, the above-mentioned image-based temperature monitoring method of power equipment is implemented. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state hard disk), etc.

[0130] The specific implementation methods described above are preferred implementation methods of the image-based power equipment temperature monitoring method, system and electronic device of the present application, and are not intended to limit the specific implementation scope of the present application. The scope of the present application includes but is not limited to the specific implementation methods. All equivalent changes made in accordance with the shape and structure of the present application are within the protection scope of the present application.

Claims

1. An image-based power equipment temperature monitoring method, characterized in that: The steps include: Obtaining historical visible light images and infrared image sets, training a feature extraction model with the historical visible light images and infrared image sets, and obtaining a feature extraction model and a visible light image feature set and an infrared image feature set; Based on the preset resolution and gated fusion, a feature segmentation layer corresponding to the visible light image and the infrared image and a modality fusion layer corresponding to the preset resolution are constructed, and an image fusion model is constructed with the feature segmentation layer, the modality fusion layer and the complexity parameter; Acquire historical power equipment temperature information corresponding to historical visible light images and infrared image sets, input the visible light image feature set and the infrared image feature set into the image fusion model, and update the image fusion model with the historical power equipment temperature information as output; An image temperature recognition model is constructed using the updated image fusion model and feature extraction model, and the image temperature recognition model is used to output real-time power equipment temperature information using real-time visible light images and infrared images; In the image fusion model, perform: The clustering algorithm is used to cluster the feature images in the same feature segmentation layer. First, the clustering clusters of the visible light image feature set and the infrared image feature set are mutually traversed to obtain the clustering cluster correspondence results, and then the feature images are fused according to the clustering cluster correspondence results. Wherein, the first fluctuation is obtained according to the text information in the real-time power equipment temperature information and the text information in the preset time sequence; the second fluctuation is obtained according to the fused image information in the real-time power equipment temperature information and the fused image information in the preset time sequence; whether the temperature of the power equipment is abnormal is determined according to the first fluctuation and the second fluctuation, and if so, an alarm is executed; The step of constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image based on a preset resolution further includes: Get the maximum preset resolution based on the maximum historical image resolution; Establishing a preset ratio based on the ratio of the temperature impact area of ​​the power equipment to the overall area of ​​the power equipment; Obtaining an intermediate preset resolution based on the maximum preset resolution and the preset ratio; A visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image are constructed with a maximum preset resolution and an intermediate preset resolution.

2. The image-based temperature monitoring method for power equipment according to claim 1, characterized in that: The feature extraction model trained with the historical visible light image and infrared image set also includes: Construct parallel CNN streams corresponding to visible light images and infrared images, respectively, to obtain a feature extraction model based on a dual-stream network; The aligned historical visible light images and historical infrared images are input into the feature extraction model based on the two-stream network to perform feature extraction model training.

3. The image-based temperature monitoring method for power equipment according to claim 1, characterized in that: The method of constructing a feature segmentation layer corresponding to the visible light image and the infrared image and a modality fusion layer corresponding to the preset resolution based on the preset resolution and gated fusion also includes: Constructing a visible light feature segmentation layer corresponding to the visible light image and an infrared feature segmentation layer corresponding to the infrared image based on a preset resolution; Based on gated fusion, a modal fusion layer is constructed for the visible light feature segmentation layer and the infrared feature segmentation layer with the same preset resolution.

4. The image-based temperature monitoring method for power equipment according to claim 1, characterized in that: The image fusion model constructed by using the feature segmentation layer, the modality fusion layer and the complexity parameter includes: Construct complexity parameters based on the average gradient of the image; The image fusion model is constructed with feature segmentation layer, modality fusion layer and complexity parameters.

5. The image-based temperature monitoring method for power equipment according to claim 1, characterized in that: Also includes: Use yolov8s to build the initial network model; Use MobileNetV4-Conv-m as the backbone network for extracting features to replace the backbone network of yolov8s and build a feature extraction model.

6. The image-based temperature monitoring method for power equipment according to claim 1, characterized in that: The obtaining of the first fluctuation according to the text information in the real-time power equipment temperature information and the text information in the preset time sequence comprises: Acquire a temperature trend curve based on the real-time temperature coordinates of the power equipment and the temperature coordinates of the power equipment within a preset time sequence; Based on the structure type corresponding to the real-time power equipment temperature coordinates and the structure type corresponding to the power equipment temperature coordinates in a preset time sequence, the real-time power equipment temperature and the power equipment temperature in a preset time sequence, obtaining a temperature change curve corresponding to the structure type; The temperature profile and the temperature change curve corresponding to the structural type are used as the first fluctuation.

7. The image-based temperature monitoring method for power equipment according to claim 6, characterized in that: The determining whether the temperature of the power equipment is abnormal according to the first fluctuation and the second fluctuation includes: If the fault range of the temperature trend curve is greater than the preset range threshold and / or the temperature change amplitude of the temperature change curve corresponding to the structure type is greater than the preset amplitude threshold, it is considered that the first fluctuation is greater than the first fluctuation threshold and the temperature of the power equipment is abnormal.

8. The image-based temperature monitoring method for power equipment according to claim 1, characterized in that: The step of inputting the visible light image feature set and the infrared image feature set into the image fusion model and updating the image fusion model with the historical power equipment temperature information as output further includes: Taking the preset resolution as the segmentation target, each feature image in the visible light image feature set and the infrared image feature set is segmented and assigned to the corresponding feature segmentation layer; Clustering the feature images in the same feature segmentation layer using a clustering algorithm; Output the clustering results to the corresponding modality fusion layer; The modality fusion layer performs the fusion of the visible light feature image and the infrared feature image; If the fused image information output by the modal fusion layer is inconsistent with the historical power equipment temperature information, the image fusion model is updated to repeat the fusion steps until the fused image information output by the modal fusion layer is consistent with the historical power equipment temperature information, and the update is completed.

9. An image-based power equipment temperature monitoring system, used to implement the method according to any one of claims 1 to 8, characterized in that: include: A data acquisition unit, used for acquiring visible light images and infrared images; A model building unit, used for building an image temperature recognition model based on a historical visible light image set and an infrared image set; A complexity parameter adjustment unit, used for adjusting the complexity parameter based on the complexity of the visible light image and the infrared image; The temperature analysis unit is used to output real-time power equipment temperature information based on the fused image output by the image temperature recognition model.

10. An electronic device, characterized in that: The electronic device includes a processor, a memory and a battery module; The memory is used to store programs; The battery module is used to supply power to the memory; The processor is used to execute the program and implement the image-based temperature monitoring method for power equipment according to any one of claims 1 to 8 when executing the program.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or instruction. When the computer program or instruction is executed by a processing device, the image-based temperature monitoring method for power equipment as described in any one of claims 1 to 8 is implemented.

Images