A method and system for arrester fault thermal imaging image recognition combined with gaze prediction

Abstract

This invention relates to the field of image recognition technology, specifically to a method and system for recognizing arrester faults using thermal imaging images combined with gaze prediction. The method includes: downsampling the original thermal imaging image to a fixed size, and sequentially passing it through multiple convolutional and Spatial Softmax layers to output a predicted gaze point trajectory sequence; extracting key and non-key regions based on the density of the gaze point trajectory sequence; reconstructing the key regions to obtain compressed and recombined target regions; inputting the non-key regions into a variational autoencoder to obtain low-dimensional latent feature vectors; and inputting the image-structure joint feature representation into an EffiCroprViT model to finally obtain the arrester fault classification result. This invention reduces computational complexity by efficiently fusing local and global features and effectively suppresses background noise interference, achieving real-time and accurate identification and early warning of power equipment fault states, thereby ensuring the safety and stability of power grid operation.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method and system for recognizing thermal imaging images of surge arrester faults by combining gaze prediction. Background Technology

[0002] With the rapid development of deep learning technology, image classification has become one of the fundamental tasks in the field of computer vision. Traditional convolutional neural networks (CNNs) rely on local receptive fields during feature extraction. Although they have the advantage of linear computational complexity and can efficiently capture local features, they are limited by neighborhood convolution operations and cannot effectively model long-range dependencies across regions, resulting in insufficient global contextual information capture capabilities and shortcomings in global information modeling. In contrast, the Transformer-based visual model ViT, which has emerged in recent years, uses a self-attention mechanism to obtain global perception capabilities. However, it causes high computational complexity when processing high-resolution images, resulting in significant computational redundancy in high-resolution industrial inspection scenarios, especially with secondary computational burdens, which severely restricts real-time requirements.

[0003] To address the problem of infrared image recognition of power equipment, existing technologies mainly employ the following solutions:

[0004] (1) Manual inspection and traditional image processing methods

[0005] Traditional methods for detecting faults in power equipment often rely on manual infrared detection, analyzing thermal imaging data to determine the equipment's operating status. However, this method is limited by human factors, and its real-time performance and accuracy are both limited. [Reference: Ren Dajiang, Ye Haipeng, Li Jianping. Analysis of the causes of a 500kV metal oxide zinc surge arrester fault [J]. Electric Porcelain Surge Arrester, 2020, (03): 127-132.]

[0006] (2) Contact sensor monitoring

[0007] Contact sensors, based on thermal convection and conduction, achieve thermal equilibrium through direct contact with the object being measured, thereby displaying the object's temperature. [Reference: Zhang Zelin. Development of Non-Contact Voltage Measurement Device Based on Electric Field Coupling [D]. 2023.]

[0008] (3) Offline test analysis

[0009] Offline test analysis involves collecting data from equipment using various testing instruments and then processing the parameters to assess the equipment's condition. [Reference: Dai Liqing, Xu Tiesheng. Fault diagnosis analysis and treatment of 330kV zinc oxide surge arrester [J]. Mechanical and Electrical Engineering Technology, 2023, 52(07):172-176.]

[0010] (4) Infrared image detection based on traditional neural networks

[0011] Using traditional visual neural networks such as CNN and ViT, fault and normal operation are identified and classified in thermal imaging images of equipment. [Reference: Cui Xiaowei. Image Fault Detection Technology of Electrical Equipment Based on CNN [J]. Automation Application, 2025, 66(02):85-87.]

[0012] The aforementioned existing technology has the following problems:

[0013] (1) Limitations of manual detection methods: Traditional methods relying on manual infrared detection suffer from problems such as being affected by subjective factors, slow response, and low efficiency, making it difficult to adapt to the increasingly complex operating environment of power plant equipment.

[0014] (2) Limitations of contact sensor methods: Contact sensors are expensive to deploy and difficult to monitor around the clock.

[0015] (3) Limitations of offline test analysis: Parameters need to be measured and processed after the equipment is offline, and it is impossible to capture the real-time dynamic characteristics of the equipment's operating status.

[0016] (4) Limitations of CNN and ViT infrared detection: CNN networks lack a global field of view, and ViT networks suffer from non-linear growth in computational complexity due to their self-attention mechanism, resulting in excessive computational load when dealing with large images. At the same time, infrared images are difficult to classify due to their low contrast and thermal diffusion effects, making it hard to guarantee accuracy. Summary of the Invention

[0017] Purpose of the invention: In view of the above-mentioned problems, the present invention proposes a surge arrester fault thermal imaging image recognition method combined with gaze prediction. This method makes creative improvements to address the problems of untimely early detection of potential faults and high risk of fault evolution in power plant equipment during operation, as well as the shortcomings of existing manual inspection and traditional image recognition methods in terms of real-time performance, accuracy and computational efficiency. The present invention also provides a surge arrester fault thermal imaging image recognition system combined with gaze prediction.

[0018] Summary of the Invention: This invention provides a method for identifying surge arrester faults using thermal imaging images combined with gaze prediction. The method includes:

[0019] The original thermal imaging image is downsampled to a fixed size and then passed through multiple convolutional and Spatial Softmax layers in sequence to output a predicted gaze point trajectory sequence. Key and non-key regions are extracted based on the density of the gaze point trajectory sequence.

[0020] The key region is compressed into low-resolution high-semantic features by an encoder. The spatial order of the low-resolution high-semantic features is modeled by a swin transformer model and then decoded and reconstructed into the compressed and recombined target region.

[0021] Non-critical regions are input into a variational autoencoder to obtain low-dimensional latent feature vectors;

[0022] The compressed and recombined target region and the low-dimensional latent feature vector are concatenated to form the image-structure joint feature representation. This image-structure joint feature representation is then input into the EffiCroprViT model. The EffiCroprViT model extracts features from the input feature vector and performs dynamic token pruning using a low-rank spatial channel mixing operation. The dynamically pruned set is then combined with the activation density matrix for prototype matching. Through prototype matching, each token is matched to the most suitable semantic prototype. The model is then trained according to the set loss function to establish a prototype library of objects to be identified. By matching the image to be detected with the prototype library, the final lightning arrester fault classification result is obtained.

[0023] Furthermore, including:

[0024] The process involves sequentially passing through multiple convolutional and Spatial Softmax layers to output a predicted gaze point trajectory sequence. Key and non-key regions are extracted based on the density of this gaze point trajectory sequence, including:

[0025] The model formed by multiple convolutional and Spatial Softmax layers is trained. The training set used is a set of thermal imaging images labeled with gaze coordinates. The gaze coordinates are two-dimensional coordinate sequences formed by human operators based on the focus position and gaze behavior.

[0026] The model is trained based on the training set formed above to output an image with anchor boxes. The anchor boxes are regions formed by channel-based coordinate sequences obtained through multiple convolutional and Spatial Softmax layers. The regions inside the anchor boxes are considered key regions, and the regions outside the anchor boxes are considered non-key regions.

[0027] Furthermore, including:

[0028] The process of training the model based on the training set formed above to output an image with anchor boxes includes:

[0029] The images in the training set are sequentially fed into five convolutional layers. After five convolutional layers, the convolutional values ​​of each position in the current image under multiple channels are obtained. The corresponding values ​​are then fed into the Spatial Softmax layer under the corresponding channel. The Spatial Softmax layer transforms the convolutional values ​​of each position in the image of each channel into a weight value and outputs the maximum weight value. Thus, the position corresponding to the maximum weight value can be known, and the coordinates of that position can be obtained. Therefore, the coordinates corresponding to all channels can be collected together to obtain the channel-based coordinate sequence.

[0030] Using channel-based coordinate sequences, clustering or fitting algorithms are employed to determine the degree of clustering or dispersion of the corresponding sequences. If the sequence is determined to be clustered, it is marked with a small anchor box. If the sequence is determined to be relatively dispersed, it is marked with a medium anchor box. Otherwise, if the sequence is determined to be quite dispersed, it is marked with a large anchor box. The dimensions of the small, medium, and large anchor boxes are obtained based on a preset fixed aspect ratio.

[0031] Furthermore, including:

[0032] The process of compressing the key region into low-resolution, high-semantic features using an encoder, modeling the spatial order of these low-resolution, high-semantic features using a Swin transformer model, and then decoding and reconstructing them into the compressed and recombined target region includes:

[0033] The encoder includes a multi-layer graph convolutional network model and a mapping. Based on the multi-layer graph convolutional network, the key regions of the current input are patched and feature extracted to obtain graph node features X and an adjacency matrix A. Here, X represents the feature nodes in the key region image, and A is used to represent the degree of association between nodes. Based on X and A, feature extraction and dimensionality reduction are achieved through the multi-layer graph convolutional network, gradually compressing it into a low-dimensional graph, represented as a feature map Z∈R. N×d Where N×d is the dimension of the space;

[0034] The feature map Z is projected onto two semantic embedding spaces M = f through two independent mapping branches. M (Z)∈R N ×dm N = f N (Z)∈R N×dn N×dm represents the dimension of the semantic embedding space M, and N×dn represents the dimension of the semantic embedding space N. The semantic embedding space is essentially a set of high-dimensional expected vectors of image nodes, which is used to construct a dual-branch embedding structure.

[0035] The embedding vectors of the two spaces are reconstructed using a cross-correlation kernel to obtain the structural correlation matrix, which describes the reconstructive structural relationships between image regions. The calculation formula is expressed as follows:

[0036]

[0037] Subsequently, a structure-guided bias term is added to the encoder's output feature Z. The attention calculation formula, integrated into the reorganization module of the Swin Transformer, is as follows:

[0038]

[0039] Here, β is a parameter that can be adjusted during training to control the position. The matrix weights in the attention score, where d is the query / key dimension, and Q, K, and V are obtained by linearly transforming the feature Z:

[0040] Q = ZW Q K = ZW K V=ZW V Among them, W Q W K and W V Weights used during linear transformation;

[0041] Swin Transformer employs a staggered window partitioning strategy in adjacent layers, which allows features from neighboring regions that originally belonged to different windows in the previous layer to be fused into the same window in the next layer to calculate the value of Attn. This breaks the receptive field limitation between windows, promotes cross-regional feature interaction and fusion, and thus obtains the enhanced feature Z'.

[0042] The decoder is a mirror image of the encoder, used to reconstruct the output feature Z', thereby obtaining the reconstructed key region feature map R_f'.

[0043] Furthermore, including:

[0044] The process of compressing the key region into low-resolution, high-semantic features using an encoder, modeling the spatial order of these low-resolution, high-semantic features using a Swin transformer model, and then decoding and reconstructing them into the compressed and recombined target region further includes:

[0045] The residuals of the reconstructed key region feature map R_f' and the original key region image are calculated, and the residual values ​​of different regions are binarized. The binarization results include 1 and 0, where 1 represents a value with large image differences in the region, that is, the first reconstruction result is not ideal, and 0 represents a value with small image differences in the region, that is, the reconstruction effect is good. Therefore, only the regions with values ​​of 1 are reconstructed, that is, the above steps of encoding, mapping and decoding are repeated to obtain the second reconstruction result. This result is accumulated with the image of the previous round and reconstructed again until the set number of iterations is reached or the iteration ends according to the compression ratio or other indicators after reconstruction. The final feature map is denoted as R_f. n , where n is the number of iterations.

[0046] Furthermore, including:

[0047] The EffiCroprViT model includes a training phase and a recognition phase, wherein...

[0048] Training phase: The image-structure joint feature representation is divided into image blocks and the embedded features of each image block are extracted. The extracted features are mixed in low-rank channels and then mapped back to features in high-dimensional space. Dynamic pruning is performed on these features. Based on the constructed activation density matrix and the high-confidence token subset obtained by dynamic pruning, prototype features are matched to build a prototype library. The construction of the prototype library includes: the corresponding elements in the activation density matrix increase with the number of times the feature prototype is matched during training, and a joint loss function is constructed based on the variance of the activation density matrix. During the continuous iterative training process, each prototype feature is uniformly activated until the iteration ends, and finally a category-level prototype library is obtained, which includes categories and corresponding prototype features.

[0049] Recognition phase: Using the prototype library built during training and the image to be detected x, a similarity response matrix is ​​obtained. Based on the obtained similarity response matrix, the posterior probability of the current category is calculated. This posterior probability reflects the matching probability between the image to be detected and the corresponding category.

[0050] Furthermore, including:

[0051] The process of performing low-rank channel blending on the extracted features and then mapping them back to the high-dimensional space, followed by dynamic pruning, includes:

[0052] Let the input feature map after feature extraction be denoted as X∈R. H×W×C First, using the projection matrix W proj ∈R C×d Transform the feature map into X′∈R H×W×dIn this low-dimensional space, channel mixing is performed, and then the features Y∈R are mapped back to the high-dimensional space. H×W×C Then, for the feature map Y∈R H×W×C Dynamic pruning is performed, where d is the number of channels less than C, H is the length of the feature map, W is the width of the feature map, and C is the number of channels; the channel blending operation is an operation after feature extraction, which interacts and fuses information from different channels, enabling the model to capture the correlation between channels.

[0053] Through a cross-attention network, a credit score is assigned to each token, represented as:

[0054]

[0055] Where q is the feature vector to be queried, d is the vector dimension corresponding to q, and T i Let i be the i-th token in Y, and let score be the i-th token in Y. i This represents the confidence level of the i-th token. The higher the output score, the more important the corresponding token is. Therefore, the K highest-scoring, high-confidence tokens are retained to form a subset.

[0056] Furthermore, including:

[0057] The prototype library is constructed by matching prototype features based on the constructed activation density matrix and the high-confidence token subset obtained by dynamic pruning, including:

[0058] The activation density matrix B∈R y×M This matrix represents the frequency with which the prototype feature corresponding to each category is matched, where y is the number of categories and M is a subset formed based on the K high-confidence tokens in this embodiment. The dimension of the feature vector set obtained by combining clustering algorithms;

[0059] During training, the initial values ​​of the elements of the activation density matrix are set to zero. Each time the model is trained, the number of times the current category recognizes a certain prototype feature is recorded. If the match with the feature is successful, the current element value is incremented by one; otherwise, it remains unchanged. During training, the variance of the elements in the matrix is ​​always kept close to zero. The variance of the matrix is ​​introduced into the joint loss function to control the variance during training, thereby obtaining the prototype library.

[0060] Furthermore, including:

[0061] The joint loss function includes:

[0062] The reconstruction loss L1 uses L2 distance to measure the pixel difference between the image generated by the Diffusion model and the original input image, in order to maintain the structural consistency of the prototype generation.

[0063] Perceptual loss L2 is introduced into the distance evaluation of the pre-trained visual network in the feature space, which improves the semantic fidelity and detail perception of the generated image.

[0064] The prototype classification loss L3 is a class discrimination loss guided by the Bayesian posterior probability distribution, which ensures that each sample can be accurately matched to its corresponding Gaussian prototype cluster.

[0065] The prototype mining loss L4 encourages the model to learn differential features in non-salient image regions through a prototype expansion mechanism guided by sub-salient regions, thereby improving prototype diversity and fine-grained expressive ability.

[0066] The auxiliary metric loss L5 includes contrastive learning loss and prototype alignment loss, which enhances the clustering of similar samples and the consistency of prototype expression, and strengthens the inter-class distinction boundary.

[0067] The discriminant loss L6 introduces an adversarial learning mechanism, which uses a discriminator to evaluate the distinction between the generated image and the real sample, thereby further improving the realism and discriminative driving ability of the prototype image.

[0068] The activation density variance loss L7 is a parameter introduced to enable the prototype library to learn each feature prototype evenly during the construction process. In other words, the variance value of the corresponding similarity density matrix is ​​added to the end of other loss functions.

[0069] The joint loss function described above is then normalized, as shown below:

[0070]

[0071] Where i∈[1,7], ε is a numerically stable term, and normalization can prevent a certain term from dominating the training process and promote collaborative learning of each subtask under the same optimization scale. mean is the mean calculation.

[0072] Therefore, the final joint loss function is defined as:

[0073]

[0074] Furthermore, including:

[0075] The process involves using the prototype library built during training and the image to be detected x to obtain a similarity response matrix, and then calculating the posterior probability of the current category based on the obtained similarity response matrix, including:

[0076] The image to be detected is encoded into N token feature vectors, and the prototype library includes D prototype feature vectors. Therefore, the elements of the similarity response matrix are represented as follows:

[0077]

[0078] Among them, T i For the image to be detected, P is encoded into the i-th token feature vector among several token features. j Let be the j-th prototype feature vector in the prototype library, where i∈N, j∈D. Then, based on the calculated element values, the corresponding similarity response matrix is ​​obtained, which reflects the probability that the prototype feature belongs to a certain category.

[0079] The posterior probability is calculated by combining the prior importance weights of each prototype feature to construct the overall class conditional likelihood.

[0080]

[0081] Where, ω y,j Using the weight values, the posterior probability of the image x to be detected corresponding to the category y is finally obtained through the Bayesian inference formula:

[0082]

[0083] The classification result is predicted as follows: if p(y|x) is greater than the set threshold, the image to be detected is judged to match the category; otherwise, it is judged not to match.

[0084] On the other hand, the present invention also provides a surge arrester fault thermal imaging image recognition system that combines gaze prediction, comprising:

[0085] The gaze prediction module is used to downsample the original thermal imaging image to a fixed size, and output a predicted gaze trajectory sequence after passing it through multiple convolutional and Spatial Softmax layers in sequence. Key regions and non-key regions are extracted based on the density of the gaze trajectory sequence.

[0086] The compression processing module is used to compress the key region into low-resolution high semantic features through an encoder. The low-resolution high semantic features are modeled into spatial order by a swin transformer model and then decoded and reconstructed into the compressed and recombined target region.

[0087] The image dimensionality reduction module is used to input non-critical regions into the variational autoencoder to obtain low-dimensional latent feature vectors;

[0088] The image recognition and localization module is used to concatenate the compressed and recombined target region and the low-dimensional latent feature vector, denoted as the image-structure joint feature representation. The image-structure joint feature representation is then input into the EffiCroprViT model. The EffiCroprViT model extracts features from the input feature vector and performs dynamic token pruning by combining low-rank spatial channel mixing operations. The dynamically pruned set is then combined with the activation density matrix for prototype matching. After prototype matching, each token is matched to the most suitable semantic prototype. The model is trained according to the set loss function to establish a prototype library of objects to be identified. By matching the image to be detected with the prototype library, the final lightning arrester fault classification result is obtained.

[0089] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0090] (1) The training set of the present invention is a set of thermal imaging images labeled with gaze coordinates. The gaze coordinates are formed by human operators performing eye gaze behavior based on the focus position, forming a set of coordinate sequences. The model is trained based on the training set formed above to output images with anchor boxes. The anchor boxes are the regions obtained by the channel-based coordinate sequence set of the training set after multiple convolutions and SpatialSoftmax layers. The regions inside the anchor boxes are regarded as key regions, and the regions outside are regarded as non-key regions. Therefore, by using the biomimetic trajectory of human operators' gaze, only a very simple neural network can determine the approximate region of the object to be identified and its proportion in the image, complete one classification and localization, reduce the subsequent computational load and improve the recognition accuracy.

[0091] (2) In the processing of key regions, the encoder part of this invention no longer adopts the traditional convolution and residual stacking structure, but uses an improved split cross-graph autoencoder design. Specifically, two independent embedding spaces M and N are generated from the encoder output, the extracted vectors are fed into these two independent spaces, and the structure is reconstructed through the cross-correlation kernel. Secondly, a swin transformer is added in the middle to restore the spatial continuity and boundary integrity of the target structure from the compressed feature map. In addition, multiple rounds of iterative reconstruction are introduced in the image reconstruction, and residual feedback loop optimization is performed to improve the reconstruction quality. When iterating, the swin transformer can not only make the decoder pay attention to the unrestored area through its attention mechanism, but also guide the spatial recombination order, making the reconstruction process more orderly. Therefore, the reconstruction process describes the reconstructive structural relationship between image regions. This mechanism is more expressive than the commonly used autocorrelation method, especially when dealing with symmetrical structures, acyclic island regions and potential directional relationships in the image, it shows stronger robustness and flexibility. Since the lightning arrester in this embodiment presents a symmetrical structure in some images, this invention is very suitable for using this method, which can reduce the processing cost of invalid background areas, retain key information and optimize resource utilization.

[0092] (3) The present invention uses a compression and recombination model to retain semantic features in key areas, while using a variational autoencoder to directly compress and reduce dimensions in non-key areas. This weakens the interference of background information and makes full use of image information, achieving a balance between strengthening key information and weakening the background.

[0093] (4) The recognition model EffiCroprViT of this invention is an innovative visual Transformer model that combines graph structure modeling and dynamic token pruning into a hybrid prototype learning paradigm. This breaks through the bottlenecks of traditional visual models in terms of inference speed, computational complexity, prototype redundancy, and global-local information fusion. For channel mixing operations, channel mixing is placed in a low-rank state space, and low-dimensional mapping is used to reduce computation. In addition, to address the problem that some prototypes are ignored during the prototype library construction process, a density activation matrix is ​​established. The variance of the density activation matrix is ​​used to realize the anti-attention mechanism of the model, so that each prototype is uniformly activated, ensuring that key information is not lost during token pruning. This achieves efficient fault localization and classification without relying on traditional bounding box regression or fixed prototypes, and has higher accuracy and interpretability. It is particularly suitable for handling fault recognition tasks of small objects in complex backgrounds and infrared images.

[0094] (5) This invention is designed for power image scenarios and adapts to engineering constraints. The model fully utilizes the spatial priors and structural constraints of power equipment layout, enabling it to adapt to the topological characteristics and physical structure features of typical thermal imaging images in power plants / substations. It relies less on large-scale general data training and is easy to transfer and highly interpretable, making it suitable for engineering scenarios such as inspection, early warning, and defect identification in target tasks. In summary, this invention reduces computational complexity by efficiently fusing local and global features and effectively suppresses background noise interference, achieving real-time and accurate identification and early warning of power equipment fault states. This ensures the safety and stability of power grid operation and provides more reliable and efficient technical support for fault early warning and safe operation of power plant equipment. Attached Figure Description

[0095] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a flowchart of a lightning arrester fault thermal imaging image recognition method combined with gaze prediction provided in an embodiment of the present invention.

[0097] Figure 2 This is a flowchart of the anchor frame marking method according to an embodiment of the present invention;

[0098] Figure 3 This is a flowchart of the key region extraction method described in an embodiment of the present invention;

[0099] Figure 4 This is a flowchart of the non-critical region extraction method described in an embodiment of the present invention;

[0100] Figure 5 This is a flowchart of the iterative reconstruction process method described in an embodiment of the present invention;

[0101] Figure 6 This is a schematic diagram of the recognition model structure described in an embodiment of the present invention;

[0102] Figure 7 This is a schematic diagram illustrating image classification according to an embodiment of the present invention;

[0103] Figure 8 This is a schematic diagram of the structure of the surge arrester fault thermal imaging image recognition system combined with gaze prediction, as described in an embodiment of the present invention. Detailed Implementation

[0104] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0105] Example 1: Refer to Figure 1 One embodiment of the present invention provides a method for identifying surge arrester fault thermal imaging images by combining gaze prediction, the method comprising the following steps:

[0106] S1 downsamples the original thermal imaging image to a fixed size and outputs a predicted gaze point trajectory sequence after passing it through multiple convolutional and SpatialSoftmax layers. Based on the density of the gaze point trajectory sequence, key regions R_f and non-key regions R_b are extracted. In this embodiment, the key regions are areas where malfunctions may occur, and the non-key regions are other areas of the image that are not considered in the primary focus.

[0107] like Figure 2 As shown, the input image is first resized. Since the module only needs to perform a large-scale coarse prediction and does not require excessively high resolution, downsampling reduces the computational burden while preserving key semantic information. The resized image is then processed through a five-layer convolutional network to extract features, and a Spatial Softmax layer is applied at the end to generate a two-dimensional coordinate sequence, simulating the human gaze trajectory from global to local when observing an object. This sequence has a dual function: on the one hand, the coordinates at the end can directly locate the target area; on the other hand, by analyzing the coordinate distribution pattern, the size of the object can be distinguished—large objects, because they need to focus on the outline and details, exhibit a dispersed trajectory; small objects, because they focus on the central area, have a more concentrated trajectory.

[0108] Spatial Softmax is an attention mechanism widely used in image processing and computer vision. It treats the input feature map as a distribution in a two-dimensional space and calculates the weight of each location to perform a weighted sum of features at different locations.

[0109] The key points of the Spatial Softmax layer include: 1. Input Feature Map: The input to Spatial Softmax is typically a two-dimensional feature map, denoted as F, where F(i,j) represents the feature vector at the i-th row and j-th column of the feature map. 2. Weight Calculation: For each position F(i,j) in the feature map, Spatial Softmax calculates its weight by applying the Softmax function. 3. Weighted Summation: The feature vector at each position in the feature map is multiplied by the corresponding weight, and then the weighted feature vectors at all positions are summed to obtain the final output. 4. Attention Distribution: The Spatial Softmax mechanism provides an attention distribution, indicating which positions in the input feature map have higher importance. This can be used to weighted summarize information from different regions in image processing tasks. Therefore, the key to the Spatial Softmax mechanism lies in the weighted attention mechanism applied to features at different positions during weight calculation and weighted summation of the input feature map.

[0110] In this embodiment, the key region R_f and non-key region R_b are extracted based on the density of the gaze point trajectory sequence, specifically including:

[0111] First, the model formed by multiple convolutional and spatial softmax layers needs to be trained. The training set used is a collection of thermal imaging images labeled with gaze coordinates. Here, gaze coordinates are two-dimensional coordinate sequences formed by the human operator's eye gaze behavior based on the focused position. For example, in this embodiment, the focused position is a surge arrester. Based on the surge arrester, the gaze is focused and the surrounding position is moved, forming a set of coordinate sequences, which can be represented as: (x1, y1), (x2, y2), ..., (x... n ,y n Where n is the total number of times a human operator focuses their line of sight on the surge arrester.

[0112] The model is trained based on the training set formed above to output images with anchor boxes. The anchor boxes are the regions obtained by passing the training set through multiple convolutional and Spatial Softmax layers based on channels. The regions inside the anchor boxes are considered as key regions, and the regions outside are considered as non-key regions.

[0113] In this embodiment, the process of forming the anchor frame includes the following steps:

[0114] The images in the training set are sequentially input into 5 convolutional layers. The first convolutional layer includes a 2D convolutional layer (Conv2D) and a Rectified Linear Unit (ReLU). The specific dimensions of the 2D convolutional layer are: a kernel size of 3×3, 32 filters, a stride of 1, and padding of 1. After the first convolutional layer, there is a max pooling layer with a corresponding kernel size of 3×3.

[0115] The second convolutional layer includes a two-dimensional convolution Conv2D and a linear rectified unit ReLU. The specific size of the two-dimensional convolution Conv2D is a kernel size of 3×3 and the number of kernels is 64. After the second convolution, there is a max pooling layer with a corresponding convolution sum of 2×2.

[0116] The third convolutional layer includes: two-dimensional convolution Conv2D and linear rectified unit ReLU. The specific size of the two-dimensional convolution Conv2D is a kernel size of 3×3 and the number of convolutional kernels is 128.

[0117] The fourth convolutional layer includes: two-dimensional convolution Conv2D and linear rectified unit ReLU, wherein the specific size of the two-dimensional convolution Conv2D is 1×1 kernel size and the number of convolutional kernels is 64;

[0118] The fifth convolutional layer includes: two-dimensional convolution Conv2D and linear rectified unit ReLU, wherein the specific size of the two-dimensional convolution Conv2D is 1×1 kernel size and the number of convolutional kernels is 32;

[0119] After five convolutional layers, the convolutional values ​​of each position in the current image across multiple channels are obtained. These values ​​are then input into the Spatial Softmax layer for the corresponding channel. In this embodiment, if the number of channels is set to 20, each channel is connected to a Spatial Softmax layer. The Spatial Softmax layer transforms the convolutional values ​​of each position in its respective channel into a probability value, i.e., a weight value, and outputs the maximum weight value. Thus, the position corresponding to the maximum weight value is known, and the coordinates of that position can be obtained. Therefore, the coordinates corresponding to all channels are combined to obtain a coordinate sequence. In this embodiment, 20 channels correspond to 20 coordinates, which can be represented as: (x1', y1'), (x'2, y'2), ..., (x'...). 20 ,y' 20 ).

[0120] Using the coordinate sequence, clustering or fitting algorithms are employed to determine the degree of clustering or dispersion. If the sequence is determined to be relatively clustered, a smaller anchor box is used to label it. If clustering indicates that the sequence is relatively dispersed, a larger anchor box is used, i.e., a medium or large anchor box. The specific choice between medium and large anchor boxes depends on the degree of dispersion. This embodiment does not set the quantization values ​​for clustering or dispersion after clustering or fitting. These values ​​can be set based on human experience or trained using a neural network to obtain the optimal quantization values ​​for clustering and dispersion. For example, the distance between two locations can be calculated, and the average distance between each pair of locations can be calculated. If the average distance is greater than a first threshold, the sequence is considered dispersed; if it is less than or equal to the first threshold but greater than a second threshold, it is considered relatively dispersed; and if it is less than or equal to the second threshold, it is considered clustered. The first threshold value must be greater than the second threshold value. However, the anchor box size must encompass all positions corresponding to the coordinate sequence. Of course, this embodiment can use other effective methods to determine the degree of clustering or dispersion, as long as the corresponding anchor boxes are obtained.

[0121] Secondly, after multiple iterations, a trained model is obtained, enabling anchor box prediction of images. This embodiment does not describe in detail the specific loss function used in training or the number of iterations. Furthermore, since other models are involved later, this embodiment can either train the parameters of the overall model formed by all models to obtain the optimal result for prediction, or train the parameters of each module individually to ensure each module is highly accurate before performing the concatenation operation. Because this training process is a fairly conventional method, it will not be described in detail in this embodiment.

[0122] In this embodiment, the anchor frame settings are based on the structural characteristics of the specific target surge arrester and the positional relationships between components, such as the fixed relative positions of the surge arrester and insulator. Three types of anchor frames with fixed aspect ratios—large, medium, and small—are preset. The anchor frame size is selected and matched using the method described above, thereby cropping out the target candidate region, filtering out redundant background information, and forming structured prior knowledge. This design reduces the computational complexity of subsequent precise identification through pre-screening and enhances the interpretability of classification using biomimetic trajectories, achieving a balance between efficiency and accuracy.

[0123] Specifically, in the prediction stage, in this embodiment, the original thermal imaging image W×H×C is input. First, the image is downsampled to a fixed size of 640*640*3, then processed through 5 convolutional layers and a Spatial Softmax layer. Spatial Softmax is applied to each channel to obtain K attention distribution maps, and the coordinates of the region with the highest weight are output. Each channel outputs one coordinate, and the coordinates from different channels form a sequence, which in turn outputs the predicted gaze point trajectory sequence T={(x1',y1'),(x'2,y'2),...,(x'2',y ... 20 ,y' 20 This process involves using a clustering algorithm to generate an attention region mask, extracting the critical region R_f and the non-critical region R_b. The critical region contains key structural information about the target surge arrester. In this embodiment, based on the density of the last 20 coordinates of the trajectory sequence, the size of R_f, i.e., the anchor frame, can be divided into three types: 100×240×3, 120×300×3, or 180×400×3. The non-critical region is the remaining part of the original image.

[0124] Therefore, this embodiment first introduces a trajectory prediction network to simulate human eye gaze behavior and quickly locate key regions in an image that may contain target objects. Based on historical gaze trajectories and image content, this network dynamically predicts the gaze focus, thereby significantly reducing the processing area required for subsequent computation. Differentiated feature processing paths are designed for key and non-key regions to maximize the model's adaptability to information density.

[0125] S2 compresses the key region R_f into a low-resolution, high-semantic feature F_f using an encoder. This feature map is then modeled in spatial order using a swin transformer model and subsequently decoded and reconstructed into the compressed and recombined target region R_f'.

[0126] This step designs an image compression and reconstruction process for localization regions containing important discriminative features. Based on an encoder-decoder structure and a loop optimization mechanism, this step can efficiently compress image size while reconstructing semantically complete feature representations, thereby improving the overall processing efficiency and storage utilization of the system.

[0127] Specifically, the encoder part of this embodiment no longer adopts the traditional convolution and residual stacking structure, but uses an improved Separate cross-graph autoencoder design to optimize the performance of structural modeling.

[0128] like Figure 3As shown, the improved split cross graph autoencoder of this embodiment includes two parts: an encoder and a mapping. The encoder uses a multi-layer graph convolutional network (GCN), which specifically includes network layers and pooling layers. The network layers include four layers. The first layer includes a two-dimensional convolution, a batch normalization layer, and a Rectified Linear Unit (ReLU). The two-dimensional convolution has a kernel size of 3, a stride of 2, and a padding of 1. The specific structures of the second and third layers are the same as those of the first layer, and will not be described again here. The last layer consists of two residual blocks. The pooling layer is a graph pooling structure.

[0129] Based on the encoder described above, the key regions of the current input are patched and feature extracted to obtain graph node features X and adjacency matrix A. X represents the feature nodes in the key region image, and A represents the degree of association between nodes. Feature extraction is then performed using a multi-layer graph convolutional network (GCN) based on X and A. Dimensionality reduction is then achieved through graph pooling, gradually compressing the data into a low-dimensional graph representation Z∈R. N×d This diagram not only preserves the structural information between regions but also integrates multi-scale semantics, where N×d is the spatial dimension, N is the length, and d is the width. The output at this point is denoted as the feature map F_f.

[0130] Based on the above, unlike traditional autoencoders that only output a single path, the S-CAutoencoder of this application projects the feature Z onto two semantic embedding spaces M = f through two independent mapping branches after the encoder output. M (Z)∈R N×dm N = f N (Z)∈R N×dn Where N×dm is the dimension of space M and N×dn is the dimension of space N, the embedding space is essentially a high-dimensional expected vector set of image nodes, which is used to construct a bi-branch embedding structure. The embedding vectors of the two spaces are reconstructed using a cross-correlation kernel, the calculation formula of which is:

[0131]

[0132] The correlation matrix describes the reconstructive structural relationships between image regions. This mechanism is more expressive than commonly used autocorrelation methods, especially when dealing with symmetrical structures, acyclic island regions, and potential directional relationships in images, exhibiting stronger robustness and flexibility. Since the surge arrester in this embodiment exhibits a symmetrical structure in some images, this method is well-suited for this scenario.

[0133] Subsequently, the structural correlation matrix As a structure-guided bias term, it is incorporated into the reorganization module of the Swin Transformer to enhance its structure-aware ability in sliding window attention. For a detailed explanation of the Swin Transformer model, please refer to the paper "Swin Transformer: Hierarchical Vision Transformer using ShiftedWindows".

[0134] Specifically, the Swin Transformer serves as the central computational unit in this module for the structural reconstruction process. It models spatial relationships within the image region compressed by the graph encoder, restores local consistency between regions, and completes / reconstructs incomplete image patches. Its role is to enhance semantic information, not to perform pixel-level image restoration. The specific implementation method is as follows:

[0135] Add a structure-guided bias term to the encoder's output feature Z. As input to Swin, the corresponding attention calculation formula is:

[0136]

[0137] Here, β is a parameter that can be adjusted during training to control the position. The matrix weights in the attention score, where d is the query / key dimension, and Q, K, and V are obtained by linearly transforming the feature Z:

[0138] Q = ZW Q K = ZW K V=ZW V Among them, W Q W K and W V These are the weights used during linear transformation.

[0139] The Swin Transformer employs a staggered window partitioning strategy in adjacent layers, allowing features from neighboring regions that originally belonged to different windows in the previous layer to be fused into the same window in the next layer to calculate attention values. This breaks the receptive field limitation between windows, promotes cross-regional feature interaction and fusion, and ultimately yields the enhanced feature Z'.

[0140] In this embodiment, the decoder is a mirror image of the encoder, used to reconstruct the output features Z'. It uses a transposed convolution (Deconv×3 layers) to restore the image size, with each layer consisting of a two-dimensional deconvolution (ConvTranspose2D) and a ReLU function. The dimensions of the two-dimensional deconvolution are:

[0141] The convolution kernel is 3, the stride is 2, the padding is 1, and the output_padding is 1. In this embodiment, the output_padding is the number of zeros padded at the beginning and end of each dimension. That is, during convolution, input data with different shapes can produce the same shape with the same kernel function; during deconvolution, the same input can produce multiple outputs with different shapes with the same kernel function, but there can only be one output result. Therefore, the output shape must be constrained, and finally the image R_f' of the reconstructed key region is obtained.

[0142] Therefore, in this step, the input image R_f is compressed into a low-resolution, high-semantic feature map F_f by the encoder. The feature map F_f models the spatial order through the swin transformer model, and then decodes and reconstructs it into a feature map R_f'. The output R_f' is the compressed and reconstructed target region image, and the size of R_f' is 1 / 4 of R_f.

[0143] In addition, this embodiment deeply integrates the iterative mechanism with the downstream recognition task. The unreconstructed portion (residual) after each round of reconstruction is binarized and then iteratively reconstructed, generating an image of the unreconstructed portion that is summed with the image generated in the previous round to achieve optimization. The reconstruction loss is then incorporated into the joint loss function. In practical applications, the number of iterations can be adjusted to dynamically control the computational load and reconstruction compression rate.

[0144] Specifically, such as Figure 5 As shown, the residual is calculated between the image R_f' of the key region after the first reconstruction and the original key region image R_f. The residual values ​​of different regions are then binarized. The binarization result includes 1 and 0, where 1 represents a value with significant image differences in that region, indicating that the first reconstruction result is not ideal, and 0 represents a value with minimal image differences in that region, indicating that the reconstruction effect is good. Therefore, only the regions with values ​​of 1 are reconstructed, that is, the above steps of encoding and decoding are repeated to obtain the result of the second reconstruction. This result is then accumulated with the image from the previous round and reconstructed again until the set number of iterations is reached or the iteration ends based on indicators such as the compression ratio after reconstruction. The final feature map is denoted as R_f. n , where n is the number of iterations.

[0145] S3 inputs the non-critical region R_b into the variational autoencoder to obtain the low-dimensional latent feature vector Z_b.

[0146] This embodiment targets non-critical regions, considering that they primarily contain background environmental information. An image dimensionality reduction module based on a variational autoencoder (VAE) is designed to extract low-dimensional latent feature representations. This module reduces computational resource consumption while preserving contextual information that aids scene understanding, avoiding the complete discarding or over-weighting of these features during the overall recognition process.

[0147] The specific method includes: image portions in non-critical regions typically do not contain critical fault information, but they do contain rich environmental context. To avoid these regions occupying too high a computational weight in the model input, while retaining their potential auxiliary recognition role, this embodiment uses a low-dimensional representation extraction mechanism based on Variational Autoencoder (VAE).

[0148] In this example, structurally, the VAE encoder employs a multi-layer convolutional network to extract latent spatial features of local regions and generates a continuous latent variable distribution through mean and variance reparameterization, achieving a compressed representation of regional features. The decoder is optionally used for reconstruction evaluation to ensure compression quality. The training objective of this module combines reconstruction error and KL divergence to balance compression ratio and information retention. By fusing low-dimensional latent variables with high-dimensional features from the target region and inputting them into the subsequent recognition network, the model improves overall information utilization while suppressing the negative impact of background interference on classification performance, achieving a structurally reasonable attention allocation and resource control.

[0149] Its structure is as follows Figure 4 As shown, in this module, the encoder uses two layers of 2D convolutional Conv2D and ReLU layers. The specific dimensions of the first layer of 2D convolutional Conv2D are: kernel size 3, number of channels 32, stride size 2. The specific dimensions of the second layer of 2D convolutional Conv2D are: kernel size 3, number of channels 64, stride size 2. Therefore, the number of channels increases continuously. After the dimensionality reduction function Flatten, the mean μ and variance vector σ are output through two fully connected layers (FC). Then, standard Gaussian sampling and KL divergence are used for adjustment. Specifically:

[0150] Sampling a location x from a Gaussian distribution (μ, σ) is equivalent to adding the mean μ of the Gaussian distribution to a factor of σ, which is the noise from a standard normal distribution.

[0151] x = μ + σθ, where θ ~ N(0,1). This makes the sampling behavior in the Gaussian distribution differentiable, and the entire training can be carried out through backpropagation.

[0152] Finally, the image is restored through deconvolution by the decoder. The specific structure of the decoder includes: first, a fully connected (FC) layer is used to perform a reshape operation, and then the reshaped result is input into the first deconvolutional layer CorvTranspose2D and the ReLU layer. The first deconvolutional layer CorvTranspose2D has 64 channels. Then, it is input into the second deconvolutional layer CorvTranspose2D and the ReLU layer. The second deconvolutional layer CorvTranspose2D has 32 channels. Finally, it is input into a two-dimensional convolutional layer and the sigmoid activation function. The two-dimensional convolutional layer has 3 channels. Finally, the low-dimensional semantic information Z_b is output.

[0153] In other words, the region dimensionality reduction process in this embodiment involves inputting R_b, using a variational autoencoder (VAE) to extract low-dimensional latent feature vectors, preserving environmental semantics to aid classification without significantly increasing computational cost, and finally outputting a vector Z_b, which, along with R_f... n Together, they are used as auxiliary features and then entered into the classification and recognition module.

[0154] S4 will compress and reassemble the target region R_f n The low-dimensional latent feature vector Z_b is concatenated with the image-structure joint feature representation, which is then input into the EffiCroprViT model. The EffiCroprViT model extracts features from the input feature vector and performs dynamic token pruning by combining low-rank spatial channel mixing operations. The dynamically pruned set is then combined with the activation density matrix for prototype matching. After prototype matching, each token is matched to the most suitable semantic prototype, and each prototype is uniformly activated. The model is trained according to the set loss function to obtain a prototype library. By calculating the response confidence between the subsequent images to be identified and the prototype library, interpretable classification and high-confidence recognition of the fault state of the surge arrester can be achieved.

[0155] The EffiCroprViT model differs from traditional point-based prototype generation models. It uses a parameter-free clustering method to automatically divide semantic components in the feature space. Each prototype is constructed from local image patches, exhibiting significant semantic complementarity and region distinguishability, effectively solving problems such as prototype redundancy and response duplication.

[0156] The EffiCroprViT model generates a high-resolution response map by calculating the similarity distribution between image patches and multiple prototypes. It then performs weighted aggregation based on the maximum response of each prototype class to construct a class conditional score distribution, ultimately completing classification through Bayesian inference. Simultaneously, this module provides natural prototype visualization and salient region heatmap output, intuitively demonstrating the semantic components and spatial location information relied upon by the model during inference, exhibiting strong interpretability and traceability.

[0157] Obviously, this model also requires a training process. This embodiment will not explain the related concepts of model training and recognition, as these are concepts that are obviously known to those skilled in the art. This embodiment describes the training and recognition process in conjunction with the structure of the model, specifically:

[0158] The specific structure of the model is as follows: Figure 6 As shown, the input image is first divided into N image patches, and the embedding features of each patch are extracted. This embodiment does not limit the extraction of embedding features, as long as the feature extraction requirements are met, such as convolutional pooling. In addition, to reduce computational cost, the model moves the channel mixing operation of image features to a low-rank space. Traditional models perform mixing of all input channels at each pixel location, which is computationally expensive. This model utilizes a projection matrix W... proj ∈R C×d The high-dimensional channel information of the input feature map is mapped to a low-dimensional space, and channel mixing is performed in this low-dimensional space. In this embodiment, this is referred to as low-rank channel mixing, and its specific implementation method is as follows:

[0159] Let the input feature map after feature extraction be X∈R H×W×C First, using the projection matrix W proj ∈R C×d Transform the feature map into X′∈R H×W×d In this low-dimensional space, the computational cost of channel mixing is significantly reduced, and then the features Y∈R are mapped back to the high-dimensional space. H×W×C Then, for the feature map Y∈R H×W×C Dynamic pruning is performed, where d is the number of channels less than C, H is the length of the feature map, W is the width of the feature map, and C is the number of channels.

[0160] Channel blending is an operation performed after feature extraction. It interacts and fuses information from different channels (feature dimensions), enabling the model to capture the correlation between channels. This embodiment does not impose restrictions on specific channel blending operations. Common channel blending operations can be used, such as weighted fusion of all input channels at each spatial location in 1×1 convolution.

[0161] Since key anomalous regions in infrared images typically occupy only a small portion of the image, EffiCroprViT introduces a dynamic token pruning mechanism based on attention scoring, known as Token Cropping, to reduce computational costs and enhance the expressive power of regions of interest. Through a cross-attention network, an attention score is assigned to each token, expressed as:

[0162]

[0163] Where q is the feature vector to be queried, d is the vector dimension corresponding to q, and T i Let i be the i-th token in Y, and let score be the i-th token in Y. i This represents the confidence level of the i-th token. A higher output score indicates a more important token. A subset is formed by retaining the K highest-scoring, high-confidence tokens. Therefore, this step achieves spatial focusing on regions that may contain prototype structures without introducing prior bounding boxes.

[0164] In addition, this embodiment also constructs an activation density matrix B∈R y×M This matrix represents the frequency with which each feature of each category is matched, where y is the number of categories and M is a subset formed based on the K high-confidence tokens in this embodiment. The dimension of the feature vector set obtained by combining relevant clustering algorithms.

[0165] In this embodiment, during the training of the EffiCroprViT model, the initial values ​​of the elements of the activation density matrix are set to zero. Each time the model is trained, the number of times a certain feature is identified in the current category is recorded. In this embodiment, a certain feature is the feature in the semantic prototype vector. If a match is successful with the feature, the current element value is incremented by one; otherwise, it remains unchanged. For example, if the current training process includes two categories, normal operation surge arresters and faulty surge arresters, and the features in the semantic prototype vector may include features such as local highlighting and blurred boundaries, during the training process, if the feature corresponding to the faulty surge arrester is blurred boundaries, the corresponding element value is incremented by one. Therefore, the corresponding element value in the activation density matrix will increase as the number of times the feature prototype is matched increases during the training process.

[0166] Secondly, in this embodiment, in order to ensure that each feature is learned evenly, during the training process, it is necessary to ensure that the variance of the elements in the matrix tends to zero. The feature that is matched the least is derived from the lowest density point of the matrix, so that the model focuses on recognizing this feature, thereby increasing the activation density.

[0167] Finally, in this embodiment, the variance of this matrix is ​​incorporated into the loss function to control the variance during training.

[0168] This mechanism ensures that each prototype feature is uniformly activated during continuous training iterations, and the model eventually learns a category-level prototype library that includes the current category and its corresponding prototype features until the iteration ends, thereby improving learning stability and semantic complementarity.

[0169] The above describes the model training process. The corresponding loss function is designed as follows in this embodiment:

[0170] In the design of the EffiCroprViT module, a joint loss function system under a multimodal prototype generation framework was constructed to simultaneously optimize image reconstruction quality, prototype expressiveness, and category discrimination performance. This loss function integrates the following seven key sub-items:

[0171] The Reconstruction Loss L1 uses the L2 distance to measure the pixel differences between the image generated by the Diffusion model and the original input image, in order to maintain the structural consistency of the prototype generation.

[0172] Perceptual Loss (L2) introduces distance evaluation in the feature space of a pre-trained visual network, improving the semantic fidelity and detail perception of the generated image.

[0173] The Prototype Classification Loss (L3) is a class discrimination loss guided by the Bayesian posterior probability distribution, ensuring that each sample can be accurately matched to its corresponding Gaussian prototype cluster.

[0174] Prototype Mining Loss L4 encourages the model to learn differential features in non-salient image regions through a prototype expansion mechanism guided by sub-salient regions, thereby improving prototype diversity and fine-grained expressive ability.

[0175] The Alignment & Contrastive Loss (L5) includes contrastive learning loss and prototype alignment loss, which enhance the clustering of similar samples and the consistency of prototype expression, and strengthen the distinction boundary between classes.

[0176] The Adversarial Loss (L6) introduces an adversarial learning mechanism, which uses a discriminator to evaluate the distinction between generated images and real samples, further enhancing the realism and discriminative driving ability of the prototype image.

[0177] The Activate Density Variance Loss (L7) is a parameter introduced to ensure that the prototype library learns each feature prototype evenly during the construction process. In this embodiment, the variance value of the corresponding similarity density matrix is ​​added to the end of other loss functions.

[0178] The loss function described above is then normalized, as shown below:

[0179]

[0180] Where i∈[1,7], ε is a numerically stable term, and normalization can prevent a certain term from dominating the training process and promote collaborative learning of each subtask under the same optimization scale. mean is the mean value.

[0181] Therefore, the final joint loss is defined as:

[0182]

[0183] This joint optimization strategy enables the EffiCroprViT module to achieve a synergistic improvement in image generation quality and classification performance during training, taking into account both interpretability and task effectiveness, and adapting to the multimodal scene understanding needs in complex power inspection environments.

[0184] After training to establish a prototype library of objects to be recognized, during the recognition phase of the EffiCroprViT model, the system receives salient image region features from the front-end reconstruction module and fuses them with the compressed features of the corresponding background region to form a joint image-structure feature representation. The recognition process of this model is as follows:

[0185] Using the prototype library built during training and the image to be detected x, a similarity response matrix is ​​obtained. The image to be detected is encoded into N token feature vectors, and the prototype library includes D prototype feature vectors. Therefore, the elements of the similarity response matrix are represented as follows:

[0186]

[0187] Among them, T i For the image to be detected, P is encoded into the i-th token feature vector among several token features. j Let be the j-th prototype feature vector in the prototype library, where i∈N, j∈D. Then, based on the calculated element values, the corresponding similarity response matrix is ​​obtained, which reflects the probability that the prototype feature belongs to a certain category.

[0188] The posterior probability of the current category is calculated based on the obtained similarity response matrix. This posterior probability reflects the matching probability between the image to be detected and the corresponding category. Specifically:

[0189] By combining the prior importance weights of each prototype, the response values ​​at all locations are summed to construct the overall class conditional likelihood:

[0190]

[0191] Where, ω y,j Using the weight values, the posterior probability of the image x to be detected corresponding to the category y is finally obtained through the Bayesian inference formula:

[0192]

[0193] The final multimodal category posterior probability is calculated and used to predict the classification result.

[0194] Specifically, if p(y|x) is greater than the set threshold, it is determined that the image to be detected matches the category; otherwise, it is determined that it does not match.

[0195] like Figure 7 As shown, the left image is a picture of the surge arrester during normal operation, and the right image is a picture of the surge arrester during fault operation. It is obvious that the color of the picture during normal operation is more uniform, while the picture during fault operation will produce light spots due to local overheating, and the lower half of the picture is brighter.

[0196] Therefore, the model designed in this embodiment can automatically locate fault regions without relying on bounding box regression, significantly improving inference efficiency and model interpretability, and is particularly suitable for infrared image fault identification and small object detection in complex backgrounds. Through this innovative design, the EffiCroprViT model provides higher computational efficiency and stronger adaptability, solving the problems of excessive computational overhead and insufficient performance in traditional methods.

[0197] Example 2: Refer to Figure 8 As shown, the present invention also provides a surge arrester fault thermal imaging image recognition system combining gaze prediction, comprising:

[0198] The gaze prediction module is used to downsample the original thermal imaging image to a fixed size, and output a predicted gaze trajectory sequence after passing it through multiple convolutional and Spatial Softmax layers in sequence. Key regions and non-key regions are extracted based on the density of the gaze trajectory sequence.

[0199] The compression processing module is used to compress the key region into low-resolution high semantic features through an encoder. The low-resolution high semantic features are modeled into spatial order by a swin transformer model and then decoded and reconstructed into the compressed and recombined target region.

[0200] The image dimensionality reduction module is used to input non-critical regions into the variational autoencoder to obtain low-dimensional latent feature vectors;

[0201] The image recognition and localization module is used to concatenate the compressed and recombined target region and the low-dimensional latent feature vector, denoted as the image-structure joint feature representation. The image-structure joint feature representation is then input into the EffiCroprViT model. The EffiCroprViT model extracts features from the input feature vector and performs dynamic token pruning by combining low-rank spatial channel mixing operations. The dynamically pruned set is then combined with the activation density matrix for prototype matching. After prototype matching, each token is matched to the most suitable semantic prototype. The model is trained according to the set loss function to establish a prototype library of objects to be identified. By matching the image to be detected with the prototype library, the final lightning arrester fault classification result is obtained.

[0202] The other technical features of the surge arrester fault thermal imaging image recognition system combined with gaze prediction described in this embodiment are similar to those of the surge arrester fault thermal imaging image recognition method combined with gaze prediction, and will not be repeated here.

[0203] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0204] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0205] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0206] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0207] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0208] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0209] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0210] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0211] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0212] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for identifying surge arrester faults using thermal imaging images combined with gaze prediction, characterized in that, The method includes: The original thermal imaging image is downsampled to a fixed size and then sequentially processed through multiple convolutional layers and... Spatial Softmax The layer outputs a predicted gaze point trajectory sequence, and key and non-key regions are extracted based on the density of the gaze point trajectory sequence. The key region is compressed into low-resolution, high-semantic features by an encoder. These low-resolution, high-semantic features are then processed... swin transformer The model establishes the spatial order, and then decodes and reconstructs it into a compressed and recombined target region. Non-critical regions are input into a variational autoencoder to obtain low-dimensional latent feature vectors; The compressed and recombined target region and low-dimensional latent feature vector are concatenated to form an image-structure joint feature representation. This image-structure joint feature representation is then input into the EffiCroprViT model. The EffiCroprViT model extracts features from the input feature vector and performs dynamic token pruning using a low-rank spatial channel mixing operation. The dynamically pruned set is then combined with the activation density matrix for prototype matching. After prototype matching, each token is matched to the most suitable semantic prototype. The model is trained according to the set loss function to establish a prototype library of objects to be identified. By matching the image to be detected with the prototype library, the final lightning arrester fault classification result is obtained. The prototype activation density matrix is ​​used to record the frequency of token matching with semantic prototypes during training. The rows of this matrix correspond to the fault category, and the columns correspond to the semantic prototypes under the same fault category. The matrix elements represent the number of times a semantic prototype is selected by a token or the normalized frequency during the current training batch or cumulative training process. The low-rank spatial channel mixing operation refers to... The process involves mapping the feature matrix from a high-dimensional channel space to a low-rank channel space, completing inter-channel information exchange in the low-rank channel space, and then mapping it back to the original high-dimensional channel space.

2. The arrester fault thermal imaging image recognition method combining gaze prediction according to claim 1, characterized in that, The process involves multiple convolutions and... Spatial Softmax The layer outputs a predicted gaze point trajectory sequence, and key and non-key regions are extracted based on the density of the gaze point trajectory sequence, including: For multiple convolutions and SpatialSoftmax The model formed by the layers is trained using a set of thermal images labeled with gaze coordinates. The gaze coordinates are two-dimensional coordinate sequences formed by human operators based on the focus position and gaze behavior, and based on gaze migration. The model is trained based on the training set formed above to output images with anchor boxes, where the anchor boxes are generated after multiple convolutions and... SpatialSoftmax The region is formed by the channel-based coordinate sequence obtained from the layer, and the region inside the anchor frame is regarded as the critical region, while the region outside it is regarded as the non-critical region.

3. The arrester fault thermal imaging image recognition method combined with gaze prediction according to claim 2, characterized in that, The process of training the model based on the training set formed above to output an image with anchor boxes includes: Images from the training set are sequentially fed into five convolutional layers. After five convolutional layers, the convolutional values ​​at each position of the current image across multiple channels are obtained. The corresponding values ​​are then input into the corresponding channels. SpatialSoftmax In the layer, SpatialSoftmax Each layer transforms the convolution values ​​of each position in the image of its respective channel into a weight value and outputs the maximum weight value, thus revealing the position corresponding to the maximum weight value and obtaining the coordinates of that position. Therefore, the set of coordinates corresponding to all channels together yields a channel-based coordinate sequence. Using channel-based coordinate sequences, clustering or fitting algorithms are employed to determine the degree of clustering or dispersion of the corresponding sequences. If the sequence is determined to be clustered, it is marked with a small anchor box. If the sequence is determined to be relatively dispersed, it is marked with a medium anchor box. Otherwise, if the sequence is determined to be quite dispersed, it is marked with a large anchor box. The dimensions of the small, medium, and large anchor boxes are obtained based on a preset fixed aspect ratio.

4. The arrester fault thermal imaging image recognition method combined with gaze prediction according to claim 1, characterized in that, The key region is compressed into low-resolution, high-semantic features by an encoder, and these low-resolution, high-semantic features are then processed... swin transformer The model establishes the spatial order of the model, which is then decoded and reconstructed into a compressed and recombined target region, including: The encoder includes a multi-layer graph convolutional network model and mapping, and performs key region processing on the current input based on the aforementioned multi-layer graph convolutional network. patch The process involves partitioning and feature extraction to obtain graph node features X and an adjacency matrix A. X represents the feature nodes in the key region image, and A represents the degree of association between nodes. Based on X and A, a multi-layer graph convolutional network is used to perform feature extraction and dimensionality reduction, progressively compressing the image into a low-dimensional graph, represented as the feature map. ,in, For the dimension of space; The feature map Z is projected onto two semantic embedding spaces through two independent mapping branches. , , The dimension of the semantic embedding space M is represented. N represents the dimension of the semantic embedding space, which is essentially a set of high-dimensional expected vectors of image nodes, and thus a bi-branch embedding structure is constructed. The embedding vectors of the two spaces are reconstructed using a cross-correlation kernel to obtain the structural correlation matrix, which describes the reconstructive structural relationships between image regions. The calculation formula is expressed as follows: ; Subsequently, a structure-guided bias term is added to the encoder's output feature Z. Integration Swin Transformer In the reorganization module, the corresponding attention calculation formula is: ; in, This is a parameter that can adjust itself during training to control position. The weights of the matrix in the attention score, For the query / key dimension, and among them By respectively through features A linear transformation yields: ,in, Weights used during linear transformation; Swin Transformer A staggered window partitioning strategy is used in adjacent layers, so that the features of neighboring regions that originally belonged to different windows in the previous layer can be fused into the same window for calculation in the next layer. Attn The value of this value is used to break the receptive field limitations between windows, promote cross-regional feature interaction and fusion, and thus obtain features with enhanced correlation. ; The decoder is a mirror image of the encoder, used to process the output features. Reconstruction is performed to obtain the key region feature map after reconstruction. .

5. The arrester fault thermal imaging image recognition method combined with gaze prediction according to claim 4, characterized in that, The key region is compressed into low-resolution, high-semantic features by an encoder, and these low-resolution, high-semantic features are then processed... swin transformer After the model models the spatial order and then decodes and reconstructs the compressed and reassembled target region, it also includes: Feature map of key areas after the first reconstruction The residuals of the original key region image are calculated, and the residual values ​​of different regions are binarized. The binarization result includes 1 and 0, where 1 represents a value with significant image differences in that region, meaning the first reconstruction result is not ideal, and 0 represents a value with minimal image differences in that region, meaning the reconstruction effect is good. Therefore, only the regions with values ​​of 1 are reconstructed, that is, the above steps of encoding, mapping, and decoding are repeated to obtain the result of the second reconstruction. This result is accumulated with the image from the previous round and reconstructed again until the set number of iterations is reached or the iteration ends based on the compression ratio after reconstruction. The final feature map is denoted as... , This represents the number of iterations.

6. The arrester fault thermal imaging image recognition method combining gaze prediction according to claim 1, characterized in that, The EffiCroprViT model includes a training phase and a recognition phase, wherein... Training phase: The image-structure joint feature representation is divided into image patches, and the embedding features of each image patch are extracted. The extracted features are then mixed in low-rank channels and mapped back to high-dimensional space. Dynamic pruning is then applied, and high confidence is obtained based on the constructed activation density matrix and the dynamic pruning. Token The subset is matched with prototype features to build a prototype library. The construction of the prototype library includes: the corresponding elements in the activation density matrix increase as the number of times the feature prototype is matched increases during the training process, and a joint loss function is constructed based on the variance of the activation density matrix. During the continuous iterative training process, each prototype feature is uniformly activated until the iteration ends, and finally a category-level prototype library is obtained, which includes categories and corresponding prototype features. Recognition phase: Utilizing the prototype library built during training and the images to be detected The similarity response matrix is ​​obtained, and the posterior probability of the current category is calculated based on the obtained similarity response matrix. This posterior probability reflects the matching probability between the image to be detected and the corresponding category.

7. The arrester fault thermal imaging image recognition method combined with gaze prediction according to claim 6, characterized in that, The process of performing low-rank channel blending on the extracted features and then mapping them back to the high-dimensional space, followed by dynamic pruning, includes: Let the input feature map after feature extraction be denoted as . First, using the projection matrix Transform the feature map In this low-dimensional space, channel blending is performed, and then the features are mapped back to the high-dimensional space. Then the feature map Dynamic pruning is performed, among which... H is the number of channels less than C, W is the length of the feature map, and C is the width of the feature map. The channel mixing operation is an operation after feature extraction, which interacts and fuses information from different channels, enabling the model to capture the correlation between channels. pass cross-attention The network assigns a credit score to each token, represented as follows: ; in, It is the feature vector that needs to be queried. for The corresponding vector dimension, For the first in Y indivual Token , Indicates the first indivual Token The higher the output score for the corresponding confidence level, the better the confidence level. Token The more important it is, the higher the score will be retained. K High confidence Token Forming a subset .

8. The arrester fault thermal imaging image recognition method combining gaze prediction according to claim 7, characterized in that, The high confidence level obtained based on the constructed activation density matrix and dynamic pruning Token The subset is used to match prototype features, thereby constructing a prototype library, including: The activation density matrix This matrix represents the frequency with which the prototype feature corresponding to each category is matched, where, y It is the number of categories. M It is based on K High confidence Token Forming a subset The dimension of the feature vector set obtained by combining clustering algorithms; During training, the initial values ​​of the elements of the activation density matrix are set to zero. Each time the model is trained, the number of times the current category recognizes a certain prototype feature is recorded. If the match with the feature is successful, the current element value is incremented by one; otherwise, it remains unchanged. During training, the variance of the elements in the matrix is ​​always kept close to zero. The variance of the matrix is ​​introduced into the joint loss function to control the variance during training, thereby obtaining the prototype library.

9. The arrester fault thermal imaging image recognition method combining gaze prediction according to claim 8, characterized in that, The joint loss function includes: Reconstruction loss It uses L2 distance to measure the pixel difference between the image generated by the Diffusion model and the original input image, in order to maintain the structural consistency of the prototype generation; Perceived loss It introduces a pre-trained visual network for distance evaluation in the feature space, which improves the semantic fidelity and detail perception of the generated images. Prototype classification loss Its class discrimination loss, guided by the Bayesian posterior probability distribution, ensures that each sample can be accurately matched to its corresponding Gaussian prototype cluster. Prototype mining loss It encourages the model to learn differential features in non-salient image regions through a prototype expansion mechanism guided by sub-salient regions, thereby improving prototype diversity and fine-grained expressive ability. Auxiliary measurement loss It includes contrastive learning loss and prototype alignment loss, which enhance the clustering of similar samples and the consistency of prototype expression, and strengthen the distinction boundary between classes; Determine the loss It introduces an adversarial learning mechanism, which uses a discriminator to evaluate the distinction between generated images and real samples, thereby further improving the realism and discriminative driving ability of the prototype image; Activation density variance loss The parameter index introduced is to enable the prototype library to learn each feature prototype evenly during the construction process. That is, the variance value of the corresponding similarity density matrix is ​​added to the end of other loss functions. The joint loss function described above is then normalized, as shown below: ; in, , It is a numerically stable term. Normalization can prevent any one term from dominating the training process and promote collaborative learning among subtasks on the same optimization scale. Calculated as the mean; Therefore, the final joint loss function is defined as: .

10. The arrester fault thermal imaging image recognition method combining gaze prediction according to claim 9, characterized in that, The prototype library and the image to be detected are built during the training process. x Obtain the similarity response matrix, and calculate the posterior probability of the current category based on the obtained similarity response matrix, including: The image to be detected is encoded into N. Token The feature vectors, and the prototype library includes D prototype feature vectors, therefore, the elements of the similarity response matrix are represented as follows: ; in, The image to be detected is encoded into several Token The first of the features i indivual Token Feature vector For the prototype library j There are prototype feature vectors, where... , Thus, based on the calculated element values, the corresponding similarity response matrix is ​​obtained, which reflects the probability that the prototype feature belongs to a certain category; The posterior probability is calculated by combining the prior importance weights of each prototype feature to construct the overall class conditional likelihood. ; in, Using these weights, the final image to be detected is obtained through Bayesian inference formulas. x right y Posterior probabilities corresponding to the categories: ; Thus, the classification result is predicted, including: if If the value is greater than the set threshold, the image is considered to match the category; otherwise, it is considered not to match.

11. A surge arrester fault thermal imaging image recognition system combining gaze prediction, characterized in that, include: The gaze prediction module is used to downsample the original thermal imaging image to a fixed size and then sequentially pass it through multiple convolutional layers and... Spatial Softmax The layer outputs a predicted gaze point trajectory sequence, and key and non-key regions are extracted based on the density of the gaze point trajectory sequence. The compression processing module is used to compress the key region into low-resolution, high-semantic features using an encoder. These low-resolution, high-semantic features are then processed... swin transformer The model establishes the spatial order, and then decodes and reconstructs it into a compressed and recombined target region. The image dimensionality reduction module is used to input non-critical regions into the variational autoencoder to obtain low-dimensional latent feature vectors; The image recognition and localization module is used to concatenate the compressed and recombined target region and the low-dimensional latent feature vector, denoted as the image-structure joint feature representation, and input the image-structure joint feature representation into the EffiCroprViT model. The EffiCroprViT model extracts features from the input feature vector and performs dynamic feature extraction by combining a low-rank spatial channel mixing operation. Token Pruning involves combining the dynamically pruned set with the activation density matrix for prototype matching. After prototype matching, each... Token A suitable semantic prototype is matched, and the model is trained according to the set loss function to establish a prototype library of objects to be identified. By matching the image to be detected with the prototype library, the final classification result of the surge arrester fault is obtained. The prototype activation density matrix is ​​a matrix used to record the matching frequency of tokens and semantic prototypes during training. The rows of the matrix correspond to the fault category, and the columns correspond to the semantic prototypes under the same fault category. The matrix elements represent the number of times a certain semantic prototype is selected by the token or the normalized frequency during the current training batch or cumulative training process. The low-rank space channel mixing operation refers to the process of mapping the token feature matrix from the high-dimensional channel space to the low-rank channel space, completing the inter-channel information interaction in the low-rank channel space, and then mapping it back to the original high-dimensional channel space.

Images