Electronic equipment thermal anomaly detection method and device, equipment and storage medium

Abstract

The invention provides an electronic equipment thermal anomaly detection method and device, equipment and a storage medium. Relates to the technical field of image analysis. The method comprises the following steps: constructing a thermal infrared image data set of the electronic equipment; carrying out denoising processing on the thermal infrared image by using Gaussian filtering, carrying out normalization on the denoised thermal infrared image based on linear normalization, and carrying out data augmentation processing to obtain a preprocessed data set; constructing a thermal anomaly detection model of the electronic equipment; pre-training the electronic equipment thermal anomaly detection model by using a transfer learning method to obtain a pre-trained electronic equipment thermal anomaly detection model; and performing secondary training on the pre-trained electronic equipment thermal anomaly detection model by using the pre-processed data set, and detecting the test set by using the secondarily trained electronic equipment thermal anomaly detection model to obtain a thermal anomaly position of the electronic equipment. The misjudgment rate of thermal anomaly detection of the electronic equipment can be reduced, and the recognition efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and in particular to a method, device, equipment and storage medium for detecting thermal anomalies in electronic equipment. Background Art

[0002] At present, information leakage has become one of the major hidden dangers threatening corporate secrets and personal privacy. Among them, the use of personal belongings to implant stealing devices for illegal information theft is becoming more and more common. These secret devices are usually disguised or embedded in daily items such as mobile phones, laptops, car keys, power banks, and even glasses and watches. They are often small in size, strong in disguise, low in energy consumption, and hidden in operation, making them difficult to detect with the naked eye or ordinary security inspection methods. Such devices can not only collect sensitive information, but also leak data in real time through wireless transmission, further increasing security risks.

[0003] Although traditional security inspection methods (such as metal detectors, X-ray scanning, etc.) can detect conventional threats to a certain extent, their detection capabilities are limited for non-metallic electronic components or micro embedded devices. At the same time, with the development of information technology and miniaturization technology, the concealment and technical complexity of espionage devices have significantly increased, which further challenges the limits of existing detection technology. In highly sensitive environments such as military enterprises, scientific research institutions, and large corporate conference venues, this type of risk is particularly prominent and more advanced detection methods are urgently needed to deal with it. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for detecting thermal anomalies of electronic devices, so as to reduce the misjudgment rate of thermal anomaly detection of electronic devices and improve recognition efficiency.

[0005] In a first aspect, the present application provides a method for detecting thermal anomalies in electronic equipment, comprising:

[0006] Constructing a thermal infrared image data set of an electronic device; wherein the thermal infrared image data set of the electronic device includes a plurality of thermal infrared images, the thermal infrared images include normal images or thermal anomaly images, the thermal anomaly images have a mark, and the mark is used to mark the location of the thermal anomaly;

[0007] De-noising the thermal infrared image by using Gaussian filtering, normalizing the de-noised thermal infrared image based on linear normalization, and performing data augmentation processing to obtain a pre-processed thermal infrared image, wherein a plurality of pre-processed infrared images constitute a pre-processed data set;

[0008] Build a thermal anomaly detection model for electronic equipment;

[0009] Pre-training the electronic device thermal anomaly detection model using a transfer learning method to obtain a pre-trained electronic device thermal anomaly detection model;

[0010] The pre-processed data set is used to perform secondary training on the pre-trained electronic device thermal anomaly detection model, and the secondary trained electronic device thermal anomaly detection model is used to detect the test set to obtain the thermal anomaly position of the electronic device.

[0011] In a possible design, the thermal infrared image is denoised using Gaussian filtering, the denoised thermal infrared image is normalized based on linear normalization, and data augmentation is performed to obtain a preprocessed thermal infrared image, including:

[0012] The thermal infrared image dataset of electronic devices is represented as Where: N is the number of images in the dataset; H×W is the resolution of each image;

[0013] The Gaussian kernel is generated by the following formula:

[0014]

[0015] in:

[0016] (x,y) is the pixel position in the kernel;

[0017] σ is the standard deviation of the Gaussian function, which is used to control the smoothness of the filter;

[0018] G(x,y) is the Gaussian kernel;

[0019] e is a natural constant;

[0020] The Gaussian kernel is convolved with the image using the following formula to obtain the denoised thermal infrared image:

[0021]

[0022] in:

[0023] I filtered (x, y) is the denoised thermal infrared image;

[0024] k is the radius of the Gaussian kernel;

[0025] i is the horizontal pixel position in the convolution kernel;

[0026] j is the vertical pixel position in the convolution kernel;

[0027] G(i,j) is the convolution kernel used for the convolution operation;

[0028] I(x+i,y+j) is the thermal infrared image before denoising;

[0029] The denoised thermal infrared image is normalized using the following formula to linearly map the pixel values ​​to the range [0,1]:

[0030] in:

[0031] I normalized (x, y) is the normalized thermal infrared image;

[0032] I(x,y) is the denoised thermal infrared image;

[0033] I max and I min are the maximum and minimum values ​​of pixels in the denoised thermal infrared image, respectively.

[0034] In a possible design, the electronic device thermal anomaly detection model includes a network structure and an attention mechanism module, wherein the network structure includes a backbone network, a feature pyramid, and a detection head; the attention mechanism module includes a fuzzy adaptive channel attention module and a spatial attention module;

[0035] The backbone network is a network that extracts the STAGE0 to STAGE4 layers in the ResNet50 network. The input image will first be feature extracted in the backbone network. The extracted features can be called feature layers, which are feature sets of the input image. In the backbone network, four feature layers can be obtained for the next step of network construction.

[0036] The feature pyramid is used to construct a multi-scale feature pyramid based on four feature layers. The multi-scale feature pyramid includes feature maps of multiple scales. Feature maps of different scales are gradually fused through upsampling to achieve fusion of deep features and shallow features to obtain a fused feature map.

[0037] The detection head includes a classifier and a regressor. The data processing steps of the classifier are: adjusting the fused feature map to a set size, and obtaining an intermediate result after the first convolution operation and the second convolution operation, stacking the intermediate results and then classifying them through a fully connected layer, and outputting a prediction result; the first convolution operation is to perform 4 3×3 convolution operations, and a 2×2 maximum pooling operation is performed after every two convolution operations, and the second convolution operation is to perform 9 3×3 convolution operations, and a 2×2 maximum pooling operation is performed after every three convolution operations.

[0038] In one possible design, the data processing steps of the fuzzy adaptive channel attention module include:

[0039] Perform global average pooling and global maximum pooling operations on the input single-layer feature map to extract the corresponding weight vector;

[0040] The obtained weight vector is fuzzified, and the feature representation after global average pooling and global maximum pooling is input into the shared fully connected layer for further feature transformation; in this process, the system multiplies the two processed feature representations by the previously obtained weight vector element by element, and adds the results to generate the weight coefficient of each channel in the input feature map; the weight coefficient of each channel is applied to the original input feature map to achieve channel-level feature reweighting, thereby enhancing the model's attention to important features;

[0041] The method of performing fuzzy operation on the obtained weight vector includes:

[0042] Fuzzy logic is introduced between the maximum pooling and average pooling results. The fuzzification operation is based on the feature strength of the pooling output, the fuzzification weight, and then the result is defuzzified using the centroid method; the fuzzy rule is defined as:

[0043] w fmax =fuzzify(MaxPool(Z j ))

[0044] w favg =fuzzify(AvgPool(Z j ))

[0045] in:

[0046] w fmax is the weight of the maximum pooled eigenvalue after blurring;

[0047] w favg is the weight of the average pooled feature value after blurring;

[0048] Fuzzify is to perform fuzzification operations;

[0049] MaxPool is to perform the maximum pooling operation;

[0050] AvgPool performs average pooling operation;

[0051] Z j is the input feature map;

[0052] Rule 1: If the local feature is significant, increase w fmax The weight of

[0053] Rule 2: If the global feature is stable, increase w favg The weight of .

[0054] The fuzzy adaptive channel attention module includes a fuzzy adaptive weight module, and the adaptive weight operation in the fuzzy adaptive weight module is as follows:

[0055]

[0056] in:

[0057] w max is the weight used to weight the largest eigenvalue;

[0058] FC m1 is a fully connected layer of dimension m1;

[0059] ReLU is the activation function;

[0060] FC m0 (w fmax ) is a fully connected layer of dimension m0;

[0061] w avg is the weight used to weight the average eigenvalue;

[0062] FC a1 is a fully connected layer of dimension a1;

[0063] FC a0 (w favg ) is a fully connected layer of dimension a0.

[0064] In one possible design, the data processing process of the spatial attention module is:

[0065] The maximum value and average value of each spatial position in the channel dimension of the input feature map are calculated to generate two spatial feature maps;

[0066] The two feature maps are concatenated in the channel dimension and feature fused through a convolutional layer with a channel number of 1 to generate a single-channel spatial weight map;

[0067] Apply the Sigmoid activation function to the weight map of a single channel, normalize it to the range of [0,1], and obtain the weight coefficient of each spatial position;

[0068] The weight coefficient of each spatial position is multiplied element by element with the original input feature map, thereby realizing adaptive weighting of the feature map in the spatial dimension and highlighting the feature information of important areas.

[0069] In a possible design, a method of performing secondary training on the pre-trained electronic device thermal anomaly detection model using the pre-processed data set includes:

[0070] Dividing the preprocessed data set into a training set and a test set in proportion;

[0071] Based on the pre-trained electronic equipment thermal anomaly detection model, the training set is used for secondary training;

[0072] The test set is input into the second-trained electronic device thermal anomaly detection model for detection to obtain the thermal anomaly location of the electronic device.

[0073] In a second aspect, the present application provides a device for detecting thermal anomalies in electronic equipment, the device comprising:

[0074] A data set construction module is configured to construct a thermal infrared image data set of an electronic device; wherein the thermal infrared image data set of the electronic device includes a plurality of thermal infrared images, the thermal infrared images include normal images or thermal abnormality images, the thermal abnormality images have a mark, and the mark is used to mark the location of the thermal abnormality;

[0075] A data preprocessing module is configured to perform denoising on the thermal infrared image using Gaussian filtering, normalize the denoised thermal infrared image based on linear normalization, and perform data augmentation processing to obtain a preprocessed thermal infrared image, wherein a plurality of preprocessed infrared images constitute a preprocessed data set;

[0076] A detection model building module, configured to build a thermal anomaly detection model for electronic equipment;

[0077] A transfer learning module is configured to pre-train the electronic device thermal anomaly detection model using a transfer learning method to obtain a pre-trained electronic device thermal anomaly detection model;

[0078] The secondary training module is configured to perform secondary training on the pre-trained electronic device thermal anomaly detection model using the pre-processed data set, and use the secondary trained electronic device thermal anomaly detection model to detect the test set to obtain the thermal anomaly position of the electronic device.

[0079] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the electronic device thermal anomaly detection method as described in the first aspect and various possible designs of the first aspect.

[0080] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method for detecting thermal anomalies in an electronic device as described in the first aspect and various possible designs of the first aspect is implemented.

[0081] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the electronic device thermal anomaly detection method as described in the first aspect and various possible designs of the first aspect.

[0082] The electronic device thermal anomaly detection method, device, equipment and storage medium provided by the present application have at least the following beneficial effects:

[0083] This application first removes noise through preprocessing of video data or image data, which is beneficial to the development of subsequent algorithms, and uses rotation, scaling, cropping, brightness change and other methods to augment the data set. Secondly, deep transfer learning can use the features learned by the pre-trained model on a large public data set to transfer these features to the thermal anomaly detection task; through transfer learning, only a small amount of labeled thermal data is needed to complete model fine-tuning, which greatly reduces the demand for large-scale labeled data and reduces data collection and labeling costs. Afterwards, the extracted features can be optimized using an attention mechanism module with a fuzzy adaptive module, which can significantly improve the performance of the model in complex situations and enhance the robustness of the electronic equipment thermal anomaly detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0085] Figure 1 A flowchart of a method for detecting thermal anomalies in electronic equipment provided in an embodiment of the present application;

[0086] Figure 2 A schematic diagram of the structure of a thermal anomaly detection model for an electronic device provided in an embodiment of the present application;

[0087] Figure 3 A schematic diagram of a fuzzy adaptive channel attention module and a spatial attention module provided in an embodiment of the present application; wherein, (a), a fuzzy adaptive channel attention module; (b), a spatial attention module;

[0088] Figure 4 A schematic diagram of a TUnetM module provided in an embodiment of the present application;

[0089] Figure 5 A schematic diagram of the structure of a thermal anomaly detection device for an electronic device provided in an embodiment of the present application.

[0090] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0091] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0092] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0093] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0094] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0095] The present invention provides a method for detecting thermal anomalies in electronic equipment. Figure 1 A flowchart of a method for detecting thermal anomalies of electronic equipment provided in an embodiment of the present application. The method for detecting thermal anomalies of electronic equipment includes steps S10 to S50.

[0096] S10: Construct a thermal infrared image dataset of an electronic device; wherein the thermal infrared image dataset of the electronic device includes a plurality of thermal infrared images, the thermal infrared images include normal images or thermal anomaly images, the thermal anomaly images have marks, and the marks are used to mark the location of the thermal anomaly.

[0097] Step S10 is the data preparation stage. As an example only, this embodiment first establishes a data set of thermal infrared images of electronic devices, converts the image resolution to 224*224, and uses the LabelImg annotation tool to annotate the collected thermal infrared images. The main annotation rule is to annotate the entire image as a normal image or an image containing thermal anomalies; for images containing thermal anomalies, a rectangular box needs to be used in the image to mark the location of the thermal anomaly.

[0098] S20: denoising the thermal infrared image by using Gaussian filtering, normalizing the denoised thermal infrared image based on linear normalization, and performing data augmentation processing to obtain a preprocessed thermal infrared image. Multiple preprocessed infrared images constitute a preprocessed data set.

[0099] In some embodiments, step S20 includes the following steps S201 and S202.

[0100] Step S201: using a Gaussian filtering algorithm to denoise the annotated thermal infrared image.

[0101] Suppose the dataset of annotated thermal infrared images is in:

[0102] N is the number of images in the dataset;

[0103] H×W is the resolution of each image.

[0104] Definition of Gaussian kernel: Gaussian kernel is a two-dimensional convolution kernel whose values ​​are generated by Gaussian function:

[0105]

[0106] in:

[0107] (x,y) is the pixel position in the kernel;

[0108] σ is the standard deviation of the Gaussian function, which is used to control the smoothness of the filter;

[0109] G(x,y) is the Gaussian kernel;

[0110] e is a natural constant.

[0111] Convolution operation: Convolve the Gaussian kernel with the image to obtain the denoised image:

[0112]

[0113] in:

[0114] I filtered (x, y) is the denoised thermal infrared image;

[0115] k is the radius of the Gaussian kernel;

[0116] i is the horizontal pixel position in the convolution kernel;

[0117] j is the vertical pixel position in the convolution kernel;

[0118] G(i,j) is the convolution kernel used for the convolution operation;

[0119] I(x+i,y+j) is the thermal infrared image before denoising.

[0120] The thermal infrared image data set obtained after Gaussian filtering and denoising is:

[0121] Step S202: using a linear normalization method to normalize the denoised thermal infrared image.

[0122] I normalized (x, y) is the normalized thermal infrared image;

[0123] I(x,y) is the denoised thermal infrared image;

[0124] I max and I min are the maximum and minimum values ​​of pixels in the denoised thermal infrared image, respectively.

[0125] S30: Construct a thermal anomaly detection model for electronic equipment.

[0126] In some embodiments, the electronic device thermal anomaly detection model includes a network structure and an attention mechanism module, the network structure includes a backbone network, a feature pyramid and a detection head; the attention mechanism module includes a fuzzy adaptive channel attention module and a spatial attention module.

[0127] Specifically, if Figure 2As shown in the figure, the network structure of Yolo is divided into three parts: Backbone (backbone network), FPN (feature pyramid) and Yolo Head (detection head); Backbone is the backbone feature extraction network, which is derived from the STAGE0 to STAGE4 layers in the ResNet50 network. The input image will first be feature extracted in the Backbone. The extracted features can be called feature layers, which are the feature sets of the input image; in the backbone part, four feature layers can be obtained to build the next network, and these four feature layers are called effective feature layers; FPN can be called Yolo's enhanced feature extraction network. Based on Yolo's detection framework and the network design idea of ​​FPN, the corresponding feature maps are extracted from the STAGE1, STAGE2, STAGE3, and STAGE4 layers of the ResNet50 backbone network, respectively, to construct a multi-scale feature pyramid, and the feature maps of different scales are fused through upsampling and downsampling to obtain the fused feature map, which is convolved and input into the TUnetM module, as shown in FIG. Figure 4 As shown in the figure, three groups of feature maps of different scales are obtained, and these feature maps are fused to achieve the fusion of deep features and shallow features. The purpose of feature fusion is to combine feature information of different scales; Yolo Head is Yolo's classifier and regressor, and the classification part is the fully connected layer. The specific steps of the classifier are to adjust the image size to (224, 224, 3), perform 4 3×3 convolutions on the image, perform a 2×2 maximum pooling operation after each two convolutions, perform 9 3×3 convolutions, and perform a 2×2 maximum pooling operation after each three convolutions. The output result of the above operations is (7, 7, 512), and the results are stacked and classified through the fully connected layer, and the final output is the relevant prediction.

[0128] like Figure 3As shown in the figure, the attention mechanism module is divided into two parts, namely the Fuzzy Adaptive Channel Attention Module (FACAM) and the Spatial Attention Module (SAM); the Fuzzy Adaptive Channel Attention Module is a lightweight and efficient attention mechanism, and its core function is to achieve the coordinated extraction of local and global feature information by dynamically adjusting the weights of different channels in the feature map. This module combines the two operations of maximum pooling and average pooling, which are used to capture local activation values ​​and global trend information respectively. Maximum pooling can extract detailed features and highlight local salient areas, while average pooling focuses on reflecting the overall feature distribution and characterizing global context information. The two features are weighted and dynamically fused by the multi-layer perceptron to generate the final feature vector, which is used to update the channel weights of the feature map. This mechanism can effectively highlight key areas (such as thermal anomaly areas, etc.) while suppressing redundant information, thereby significantly improving the adaptability and robustness of the model in complex scenarios; in addition, the spatial attention module further strengthens the spatial distribution information of significant features such as thermal anomalies of electronic devices by assigning weights to each spatial position of the feature map. The attention mechanism module combines the fuzzy adaptive channel attention mechanism with the spatial attention mechanism to achieve the network's adaptive attention to key areas. Specifically, the module performs fuzzy adaptive channel attention processing and spatial attention processing on the input feature layer, thereby optimizing feature representation in both channel and spatial dimensions and improving the model's ability to capture target features.

[0129] The fuzzy adaptive channel attention mechanism mainly consists of two key steps. First, the mechanism performs global average pooling (GAP) and global maximum pooling (GMP) operations on the input single-layer feature map to extract the corresponding weight vector. Subsequently, the obtained weight vector is optimized by the fuzzy adaptive algorithm, and the feature representation after global average pooling and global maximum pooling is input into the shared fully connected layer for further feature transformation. In this process, the system multiplies the two processed feature representations by the previously obtained weight vector element by element, and adds the results to generate the weight coefficients for each channel in the input feature map. Finally, these weight coefficients are applied to the original input feature map to achieve channel-level feature reweighting, thereby enhancing the model's attention to important features.

[0130] Fuzzy logic is introduced between the maximum pooling and average pooling results. The fuzzification operation can fuzzify the weights according to the feature strength of the pooling output, and then the centroid method is used to defuzzify the results. The fuzzy rule is defined as:

[0131] w fmax =fuzzify(MaxPool(Z j ))

[0132] w favg =fuzzify(AvgPool(Z j ))

[0133] in:

[0134] w fmax is the weight of the maximum pooled eigenvalue after blurring;

[0135] w favg is the weight of the average pooled feature value after blurring;

[0136] Fuzzify is to perform fuzzification operations;

[0137] MaxPool is to perform the maximum pooling operation;

[0138] AvgPool performs average pooling operation;

[0139] Z j is the input feature map;

[0140] Rule 1: If the local feature is significant, increase w fmax The weight of

[0141] Rule 2: If the global feature is stable, increase w favg The weight of .

[0142] For the fuzzy adaptive weight module in the fuzzy adaptive channel attention module, the adaptive weight operation is as follows:

[0143]

[0144] in:

[0145] w max is the weight used to weight the largest eigenvalue;

[0146] FC m1 is a fully connected layer of dimension m1;

[0147] ReLU is the activation function;

[0148] FC m0 (w fmax ) is a fully connected layer of dimension m0;

[0149] w avg is the weight used to weight the average eigenvalue;

[0150] FC a1 is a fully connected layer of dimension a1;

[0151] FC a0 (w favg ) is a fully connected layer of dimension a0.

[0152] The fuzzy adaptive weight module consists of two fully connected layers.

[0153] S40: Pre-training the electronic device thermal anomaly detection model by using a transfer learning method to obtain a pre-trained electronic device thermal anomaly detection model.

[0154] In some embodiments, the method of pre-training the electronic device thermal anomaly detection model using the transfer learning method is: loading the electronic device thermal anomaly detection model in step S30, freezing some convolutional layers, and training the model using the ImageNet dataset to obtain the pre-trained electronic device thermal anomaly detection model.

[0155] S50: performing secondary training on the pre-trained electronic device thermal anomaly detection model using the pre-processed data set, and performing detection on the test set using the secondary trained electronic device thermal anomaly detection model to obtain the thermal anomaly position of the electronic device.

[0156] In some embodiments, step S50 includes the following steps S501 to S503.

[0157] S501: Randomly select 80% of the samples from the electronic device thermal infrared image dataset as a training set, and the remaining 20% ​​as a test set.

[0158] S502: Perform secondary training using the training set based on the pre-trained electronic device thermal anomaly detection model.

[0159] S503: Input the test set into the electronic device thermal anomaly detection model that has been trained twice to perform detection, and obtain the thermal anomaly location of the electronic device.

[0160] The present application also provides a device for detecting thermal anomalies in electronic equipment. Figure 5 As shown, the electronic equipment thermal anomaly detection device comprises:

[0161] The data set construction module 501 is configured to construct a thermal infrared image data set of an electronic device; wherein the thermal infrared image data set of the electronic device includes a plurality of thermal infrared images, wherein the thermal infrared images include normal images or thermal abnormality images, wherein the thermal abnormality images have a mark, and the mark is used to mark the location of the thermal abnormality;

[0162] The data preprocessing module 502 is configured to perform denoising on the thermal infrared image using Gaussian filtering, normalize the denoised thermal infrared image based on linear normalization, and perform data augmentation processing to obtain a preprocessed thermal infrared image. Multiple preprocessed infrared images constitute a preprocessed data set.

[0163] The detection model building module 503 is configured to build a thermal anomaly detection model for electronic equipment;

[0164] The transfer learning module 504 is configured to pre-train the electronic device thermal anomaly detection model by using a transfer learning method to obtain a pre-trained electronic device thermal anomaly detection model;

[0165] The secondary training module 505 is configured to perform secondary training on the pre-trained electronic device thermal anomaly detection model using the pre-processed data set, and detect the test set using the secondary trained electronic device thermal anomaly detection model to obtain the electronic device thermal anomaly location.

[0166] In some embodiments, the data preprocessing module is further configured to:

[0167] The thermal infrared image dataset of electronic devices is represented as Where: N is the number of images in the dataset; H×W is the resolution of each image;

[0168] The Gaussian kernel is generated by the following formula:

[0169]

[0170] in:

[0171] (x,y) is the pixel position in the kernel;

[0172] σ is the standard deviation of the Gaussian function, which is used to control the smoothness of the filter;

[0173] G(x,y) is the Gaussian kernel;

[0174] e is a natural constant;

[0175] The Gaussian kernel is convolved with the image using the following formula to obtain the denoised thermal infrared image:

[0176]

[0177] in:

[0178] I filtered (x, y) is the denoised thermal infrared image;

[0179] k is the radius of the Gaussian kernel;

[0180] i is the horizontal pixel position in the convolution kernel;

[0181] j is the vertical pixel position in the convolution kernel;

[0182] G(i,j) is the convolution kernel used for the convolution operation;

[0183] I(x+i,y+j) is the thermal infrared image before denoising;

[0184] The denoised thermal infrared image is normalized using the following formula to linearly map the pixel values ​​to the range [0,1]:

[0185] in:

[0186] I normalized (x, y) is the normalized thermal infrared image;

[0187] I(x,y) is the denoised thermal infrared image;

[0188] I max and I min are the maximum and minimum values ​​of pixels in the denoised thermal infrared image, respectively.

[0189] In some embodiments, the electronic device thermal anomaly detection model includes a network structure and an attention mechanism module, wherein the network structure includes a backbone network, a feature pyramid, and a detection head; the attention mechanism module includes a fuzzy adaptive channel attention module and a spatial attention module;

[0190] The backbone network is a network that extracts the STAGE0 to STAGE4 layers in the ResNet50 network. The input image will first be feature extracted in the backbone network. The extracted features can be called feature layers, which are feature sets of the input image. In the backbone network, four feature layers can be obtained for the next step of network construction.

[0191] The feature pyramid is used to construct a multi-scale feature pyramid based on four feature layers. The multi-scale feature pyramid includes feature maps of multiple scales. Feature maps of different scales are gradually fused through upsampling to achieve fusion of deep features and shallow features to obtain a fused feature map.

[0192] The detection head includes a classifier and a regressor. The data processing steps of the classifier are: adjusting the fused feature map to a set size, and obtaining an intermediate result after the first convolution operation and the second convolution operation, stacking the intermediate results and then classifying them through a fully connected layer, and outputting a prediction result; the first convolution operation is to perform 4 3×3 convolution operations, and a 2×2 maximum pooling operation is performed after every two convolution operations, and the second convolution operation is to perform 9 3×3 convolution operations, and a 2×2 maximum pooling operation is performed after every three convolution operations.

[0193] In some embodiments, the data processing steps of the fuzzy adaptive channel attention module include:

[0194] Perform global average pooling and global maximum pooling operations on the input single-layer feature map to extract the corresponding weight vector;

[0195] The obtained weight vector is fuzzified, and the feature representation after global average pooling and global maximum pooling is input into the shared fully connected layer for further feature transformation; in this process, the system multiplies the two processed feature representations by the previously obtained weight vector element by element, and adds the results to generate the weight coefficient of each channel in the input feature map; the weight coefficient of each channel is applied to the original input feature map to achieve channel-level feature reweighting, thereby enhancing the model's attention to important features;

[0196] The method of performing fuzzy operation on the obtained weight vector includes:

[0197] Fuzzy logic is introduced between the maximum pooling and average pooling results. The fuzzification operation is based on the feature strength of the pooling output, the fuzzification weight, and then the result is defuzzified using the centroid method; the fuzzy rule is defined as:

[0198] w fmax =fuzzify(MaxPool(Z j ))

[0199] w favg =fuzzify(AvgPool(Z j ))

[0200] in:

[0201] w fmax is the weight of the maximum pooled eigenvalue after blurring;

[0202] w favg is the weight of the average pooled feature value after blurring;

[0203] Fuzzify is to perform fuzzification operations;

[0204] MaxPool is to perform the maximum pooling operation;

[0205] AvgPool performs average pooling operation;

[0206] Z j is the input feature map;

[0207] Rule 1: If the local feature is significant, increase w fmax The weight of

[0208] Rule 2: If the global feature is stable, increase w favg The weight of

[0209] The fuzzy adaptive channel attention module includes a fuzzy adaptive weight module, and the adaptive weight operation in the fuzzy adaptive weight module is as follows:

[0210]

[0211] in:

[0212] w max is the weight used to weight the largest eigenvalue;

[0213] FC m1 is a fully connected layer of dimension m1;

[0214] ReLU is the activation function;

[0215] FC m0 (w fmax ) is a fully connected layer of dimension m0;

[0216] w avg is the weight used to weight the average eigenvalue;

[0217] FC a1 is a fully connected layer of dimension a1;

[0218] FC a0 (w favg ) is a fully connected layer of dimension a0.

[0219] In some embodiments, the data processing process of the spatial attention module is:

[0220] The maximum value and average value of each spatial position in the channel dimension of the input feature map are calculated to generate two spatial feature maps;

[0221] The two feature maps are concatenated in the channel dimension and feature fused through a convolutional layer with a channel number of 1 to generate a single-channel spatial weight map;

[0222] Apply the Sigmoid activation function to the weight map of a single channel, normalize it to the range of [0,1], and obtain the weight coefficient of each spatial position;

[0223] The weight coefficient of each spatial position is multiplied element by element with the original input feature map, thereby realizing adaptive weighting of the feature map in the spatial dimension and highlighting the feature information of important areas.

[0224] In some embodiments, the secondary training module is further configured to:

[0225] Dividing the preprocessed data set into a training set and a test set in proportion;

[0226] Based on the pre-trained electronic equipment thermal anomaly detection model, the training set is used for secondary training;

[0227] The test set is input into the second-trained electronic device thermal anomaly detection model for detection to obtain the thermal anomaly location of the electronic device.

[0228] An embodiment of the present application provides an electronic device, which may include: a processor and a memory, wherein the processor and the memory may communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0229] The processor executes the computer execution instructions stored in the memory, so that the processor executes the scheme in the above embodiment. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0230] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The system bus can be divided into an address bus, a data bus, a control bus, etc. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0231] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0232] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the electronic device thermal anomaly detection method of the above embodiment.

[0233] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the electronic device thermal anomaly detection method in the above embodiment can be implemented.

[0234] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0235] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0236] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0237] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0238] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0239] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0240] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0241] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0242] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0243] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0244] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting thermal anomaly of electronic equipment, characterized in that: The method comprises: Constructing a thermal infrared image data set of an electronic device; wherein the thermal infrared image data set of the electronic device includes a plurality of thermal infrared images, the thermal infrared images include normal images or thermal anomaly images, the thermal anomaly images have a mark, and the mark is used to mark the location of the thermal anomaly; De-noising the thermal infrared image by using Gaussian filtering, normalizing the de-noised thermal infrared image based on linear normalization, and performing data augmentation processing to obtain a pre-processed thermal infrared image, wherein a plurality of pre-processed infrared images constitute a pre-processed data set; Build a thermal anomaly detection model for electronic equipment; Pre-training the electronic device thermal anomaly detection model using a transfer learning method to obtain a pre-trained electronic device thermal anomaly detection model; The pre-processed data set is used to perform secondary training on the pre-trained electronic device thermal anomaly detection model, and the secondary trained electronic device thermal anomaly detection model is used to detect the test set to obtain the thermal anomaly position of the electronic device.

2. The electronic device thermal anomaly detection method according to claim 1, characterized in that: The thermal infrared image is denoised by using Gaussian filtering, the denoised thermal infrared image is normalized based on linear normalization, and data augmentation is performed to obtain a preprocessed thermal infrared image, including: The thermal infrared image dataset of electronic equipment is represented as Where: N is the number of images in the dataset; H×W is the resolution of each image; The Gaussian kernel is generated by the following formula: in: (x,y) is the pixel position in the kernel; σ is the standard deviation of the Gaussian function, which is used to control the smoothness of the filter; G(x,y) is the Gaussian kernel; e is a natural constant; The Gaussian kernel is convolved with the image using the following formula to obtain the denoised thermal infrared image: in: I filtered (x, y) is the denoised thermal infrared image; k is the radius of the Gaussian kernel; i is the horizontal pixel position in the convolution kernel; j is the vertical pixel position in the convolution kernel; G(i,j) is the convolution kernel used for the convolution operation; I(x+i,y+j) is the thermal infrared image before denoising; The denoised thermal infrared image is normalized using the following formula to linearly map the pixel values ​​to the range [0,1]: in: I normalized (x, y) is the normalized thermal infrared image; I(x,y) is the denoised thermal infrared image; I max and I min are the maximum and minimum values ​​of pixels in the denoised thermal infrared image, respectively.

3. The electronic device thermal anomaly detection method according to claim 1, characterized in that: The electronic equipment thermal anomaly detection model includes a network structure and an attention mechanism module, wherein the network structure includes a backbone network, a feature pyramid and a detection head; the attention mechanism module includes a fuzzy adaptive channel attention module and a spatial attention module; The backbone network is a network that extracts the STAGE0 to STAGE4 layers in the ResNet50 network. The input image will first be feature extracted in the backbone network. The extracted features can be called feature layers, which are feature sets of the input image. In the backbone network, four feature layers can be obtained for the next step of network construction. The feature pyramid is used to construct a multi-scale feature pyramid based on four feature layers. The multi-scale feature pyramid includes feature maps of multiple scales. Feature maps of different scales are gradually fused through upsampling to achieve fusion of deep features and shallow features to obtain a fused feature map. The detection head includes a classifier and a regressor. The data processing steps of the classifier are: adjusting the fused feature map to a set size, and obtaining an intermediate result after the first convolution operation and the second convolution operation, stacking the intermediate results and then classifying them through a fully connected layer, and outputting a prediction result; the first convolution operation is to perform 4 3×3 convolution operations, and a 2×2 maximum pooling operation is performed after every two convolution operations, and the second convolution operation is to perform 9 3×3 convolution operations, and a 2×2 maximum pooling operation is performed after every three convolution operations.

4. The electronic device thermal anomaly detection method according to claim 3, characterized in that: The data processing steps of the fuzzy adaptive channel attention module include: Perform global average pooling and global maximum pooling operations on the input single-layer feature map to extract the corresponding weight vector; The obtained weight vector is fuzzified, and the feature representation after global average pooling and global maximum pooling is input into the shared fully connected layer for further feature transformation; in this process, the system multiplies the two processed feature representations by the previously obtained weight vector element by element, and adds the results to generate the weight coefficient of each channel in the input feature map; the weight coefficient of each channel is applied to the original input feature map to achieve channel-level feature reweighting, thereby enhancing the model's attention to important features; The method of performing fuzzy operation on the obtained weight vector includes: Fuzzy logic is introduced between the maximum pooling and average pooling results. The fuzzification operation is based on the feature strength of the pooling output, the fuzzification weight, and then the result is defuzzified using the centroid method; the fuzzy rule is defined as: In fmax =fuzzify(MaxPool(Z j )) In favg =fuzzify(AvgPool(Z j )) in: w fmax is the weight of the maximum pooled eigenvalue after blurring; w favg is the weight of the average pooled feature value after blurring; Fuzzify is to perform fuzzification operations; MaxPool is to perform the maximum pooling operation; AvgPool performs average pooling operation; Z j is the input feature map; Rule 1: If the local feature is significant, increase w fmax The weight of Rule 2: If the global feature is stable, increase w favg The weight of The fuzzy adaptive channel attention module includes a fuzzy adaptive weight module, and the adaptive weight operation in the fuzzy adaptive weight module is as follows: in: w max is the weight used to weight the largest eigenvalue; FC m1 is a fully connected layer of dimension m1; ReLU is the activation function; FC m0 (w fmax ) is a fully connected layer of dimension m0; w avg is the weight used to weight the average eigenvalue; FC a1 is a fully connected layer of dimension a1; FC a0 (w favg ) is a fully connected layer of dimension a0.

5. The electronic device thermal anomaly detection method according to claim 3, characterized in that: The data processing process of the spatial attention module is: The maximum value and average value of each spatial position in the channel dimension of the input feature map are calculated to generate two spatial feature maps; The two feature maps are concatenated in the channel dimension and feature fused through a convolutional layer with a channel number of 1 to generate a single-channel spatial weight map; Apply the Sigmoid activation function to the weight map of a single channel, normalize it to the range of [0,1], and obtain the weight coefficient of each spatial position; The weight coefficient of each spatial position is multiplied element by element with the original input feature map, thereby realizing adaptive weighting of the feature map in the spatial dimension and highlighting the feature information of important areas.

6. The electronic device thermal anomaly detection method according to claim 1, characterized in that: The method of performing secondary training on the pre-trained electronic device thermal anomaly detection model using the pre-processed data set includes: Dividing the preprocessed data set into a training set and a test set in proportion; Based on the pre-trained electronic equipment thermal anomaly detection model, the training set is used for secondary training; The test set is input into the second-trained electronic device thermal anomaly detection model for detection to obtain the thermal anomaly location of the electronic device.

7. An electronic equipment thermal anomaly detection device, characterized in that: The device comprises: A data set construction module is configured to construct a thermal infrared image data set of an electronic device; wherein the thermal infrared image data set of the electronic device includes a plurality of thermal infrared images, the thermal infrared images include normal images or thermal abnormality images, the thermal abnormality images have a mark, and the mark is used to mark the location of the thermal abnormality; A data preprocessing module is configured to perform denoising on the thermal infrared image using Gaussian filtering, normalize the denoised thermal infrared image based on linear normalization, and perform data augmentation processing to obtain a preprocessed thermal infrared image, wherein a plurality of preprocessed infrared images constitute a preprocessed data set; A detection model building module, configured to build a thermal anomaly detection model for electronic equipment; A transfer learning module is configured to pre-train the electronic device thermal anomaly detection model using a transfer learning method to obtain a pre-trained electronic device thermal anomaly detection model; The secondary training module is configured to perform secondary training on the pre-trained electronic device thermal anomaly detection model using the pre-processed data set, and use the secondary trained electronic device thermal anomaly detection model to detect the test set to obtain the thermal anomaly position of the electronic device.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the electronic device thermal anomaly detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the electronic device thermal anomaly detection method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the electronic device thermal anomaly detection method according to any one of claims 1 to 6 is implemented.

Images