Training method and device of infrared image detection model and infrared image detection method

Abstract

The invention discloses a training method and device of an infrared image detection model and an infrared image detection method, which can be used in the field of artificial intelligence, and the method comprises the steps: firstly, obtaining a training picture containing an infrared image; then, based on the training picture, through a pre-constructed infrared image detection model, outputting a defect detection result; then, calculating a first loss function based on a bipartite graph matching result of a defect detection result and a defect label corresponding to the training picture; thirdly, calculating the sum of the first loss function and the multi-layer perception loss item to obtain a loss function; and finally, performing iterative training on the infrared image detection model based on the loss function. Therefore, the infrared image detection model is trained by adopting the composite loss function based on the first loss function obtained based on the bipartite graph matching result and the multi-layer perception loss item, and the trained infrared image detection model can realize more accurate detection on small targets in the infrared image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method and device for an infrared image detection model and an infrared image detection method. Background Art

[0002] In photovoltaic power plants, defects in photovoltaic panels can lead to energy loss and reduced system efficiency. As the scale of photovoltaic power plants continues to expand, how to detect energy loss areas caused by defects such as cracks, hot spots, blockages, and corrosion in photovoltaic panels has become increasingly important.

[0003] Drone infrared inspection technology can be used to quickly inspect large areas of photovoltaic panels at high altitudes. It has the advantages of being efficient and non-destructive, and has been widely used in defect detection of photovoltaic panels. Infrared thermal imaging technology can detect temperature distribution by measuring the infrared radiation of an object. After the infrared image is captured by drone infrared inspection technology, a deep learning model is needed to identify abnormal areas on the photovoltaic panel. However, due to the small number of label files that photovoltaic power stations can provide, the current deep learning model has low detection accuracy for small targets such as hot spots and diode failures, and it is difficult to detect defects in small targets.

[0004] Therefore, how to achieve more accurate detection of small targets in infrared images has become a problem that needs to be solved. Summary of the invention

[0005] Based on the above problems, the present application provides a training method and device for an infrared image detection model and an infrared image detection method, which can achieve relatively accurate detection of small targets in infrared images.

[0006] The embodiments of the present application disclose the following technical solutions: In a first aspect, an embodiment of the present application provides a method for training an infrared image detection model, the method comprising: Get training pictures containing infrared images; Based on the training image, a defect detection result is output through a pre-built infrared image detection model; the number of prediction boxes generated by the infrared image detection model is greater than the real target; the category label in the defect detection result is selected from the real category to which the real target belongs and the empty category representing the absence of the real target; Calculating a first loss function based on a bipartite graph matching result between the defect detection result and the defect label corresponding to the training image; Calculating the sum of the first loss function and a multi-layer perceptual loss term to obtain a loss function; the multi-layer perceptual loss term includes the sum of weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model; The infrared image detection model is iteratively trained based on the loss function.

[0007] Optionally, the infrared image detection model includes a convolutional neural network, a spectral domain Transformer encoder, a multi-scale Transformer decoder and a fully connected layer; The convolutional neural network is used to extract feature markers of the training image to obtain a feature map; The spectral domain Transformer encoder is used to encode the feature map to obtain encoded features; The multi-scale Transformer decoder is used to obtain a pixel-by-pixel defect prediction result based on the input encoding features; The fully connected layer is used to output the defect detection result of the training image based on the defect prediction result.

[0008] Optionally, the depths of the spectral domain Transformer encoder and the multi-scale Transformer decoder are both 6; the spectral domain Transformer encoder and the multi-scale Transformer decoder are symmetrical structures; The last layer of the spectral domain Transformer encoder inputs the encoded features into decoding modules of various depths in the multi-scale Transformer decoder.

[0009] Optionally, the spectral domain Transformer encoder based on the self-attention mechanism is specifically used for: The feature map is mapped from the spatial domain to the spectral domain to obtain a coding feature; the coding feature includes a global dependency relationship between different feature markers.

[0010] Optionally, the multi-scale Transformer decoder includes a multi-scale feature extraction module and a multi-layer regression prediction module; the multi-scale Transformer decoder is specifically used to: Extracting multi-scale features from the coding features through the multi-scale feature extraction module based on a multi-head attention mechanism, a layer normalization operation, and a full connection operation; fusing the multi-scale features through a convolution layer with a convolution kernel size of 1 to obtain a fused feature; Through the multi-layer regression prediction module, a defect prediction result is obtained based on the fusion features.

[0011] Optionally, the number of layers of the multi-scale Transformer decoder is 4, and the spatial resolution levels corresponding to different layers are 1, 1 / 2, 1 / 4 and 1 / 8 respectively.

[0012] Optionally, before acquiring the training picture containing the infrared image, the method further includes: Acquire an initial picture including an infrared image; the initial picture is collected by an infrared camera; Based on the distortion coefficient of the infrared camera and the camera internal parameter matrix, the initial image is transformed into a training image.

[0013] Optionally, the number of prediction boxes generated by the infrared image detection model is 50.

[0014] In a second aspect, an embodiment of the present application provides a training device for an infrared image detection model, the device comprising: an acquisition module, an output module, a first calculation module, a second calculation module and a training module; The acquisition module is used to acquire a training picture containing an infrared image; The output module is used to output defect detection results based on the training image through a pre-built infrared image detection model; the number of prediction boxes generated by the infrared image detection model is greater than the real target; the defect detection results include the real category to which the real target belongs and the empty category representing the absence of the real target; The first calculation module is used to calculate a first loss function based on a bipartite graph matching result of the defect detection result and a defect label corresponding to the training image; The second calculation module is used to calculate the sum of the first loss function and a multi-layer perceptual loss term to obtain a loss function; the multi-layer perceptual loss term includes the sum of weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model; The training module is used to iteratively train the infrared image detection model based on the loss function.

[0015] In a third aspect, an embodiment of the present application provides an infrared image detection method, the method comprising: Acquire the image to be inspected including the infrared image; Based on the image to be inspected, an infrared image detection model trained by the training method of the infrared image detection model described in any embodiment of the first aspect outputs a defect detection result.

[0016] Compared with the prior art, this application has the following beneficial effects: The embodiment of the present application provides a training method for an infrared image detection model, in which, first, a training picture containing an infrared image is obtained; then, based on the training picture, a defect detection result is output through a pre-built infrared image detection model; the number of prediction boxes generated by the infrared image detection model is greater than the real target; the category label in the defect detection result is selected from the real category to which the real target belongs and the empty category representing the absence of the real target; then, based on the bipartite graph matching result of the defect detection result and the defect label corresponding to the training picture, a first loss function is calculated; then, the sum of the first loss function and the multi-layer perceptual loss term is calculated to obtain a loss function; the multi-layer perceptual loss term includes the sum of the weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model; finally, the infrared image detection model is iteratively trained based on the loss function. Thus, based on the first loss function obtained from the bipartite graph matching result and the multi-layer perceptual loss term, a composite loss function is used to train the infrared image detection model, and the trained infrared image detection model can achieve more accurate detection of small targets in infrared images. Among them, through the first loss function adjusted based on the bipartite graph matching result, the difference between the predicted box and the real target can be minimized, which helps to improve the detection accuracy of the infrared image detection model for small target objects; and through the multi-layer perceptual loss term, the feature extraction ability and detail restoration of the infrared image detection model can be effectively improved. By weighting each layer, it ensures that the influence of different levels of perceptual losses on the final result is reasonably balanced, thereby retaining high-level information while enhancing the ability to capture low-level details, and better constraining small real targets and predicted targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flow chart of a training method for an infrared image detection model provided in an embodiment of the present application; Figure 2 A schematic diagram of radial distortion provided in an embodiment of the present application; Figure 3 A schematic diagram of the cause of tangential distortion provided in an embodiment of the present application; Figure 4 A schematic diagram of an imaging projection provided in an embodiment of the present application; Figure 5 An architecture diagram of an infrared image detection model provided in an embodiment of the present application; Figure 6 A structural diagram of a spectral domain Transformer encoder provided in an embodiment of the present application; Figure 7 A structural diagram of a multi-scale Transformer decoder provided in an embodiment of the present application; Figure 8 A schematic diagram of a multi-level codec connection mechanism provided in an embodiment of the present application; Fig. 9 A schematic diagram of a training device for an infrared image detection model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The training method, device and infrared image detection method of an infrared image detection model provided in the present application can be used in the field of artificial intelligence. The above is only an example and does not limit the application field of the training method, device and infrared image detection method of an infrared image detection model provided in the present application.

[0020] The terms "first", "second", "third" and "fourth" etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0021] In the embodiments of the present application, the words "as an example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "as an example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the words "as an example" or "for example" is intended to present the relevant concepts in a specific way.

[0022] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] See also Figure 1 , which is a flow chart of a training method for an infrared image detection model provided in an embodiment of the present application, the method comprising: S101: Acquire a training picture containing an infrared image.

[0025] As an example, a training image can be obtained based on an initial image containing an infrared image taken by an infrared camera such as an infrared thermal imager carried by a device such as a drone; the training image carries a defect label, and the training image can be divided into a training set, a test set, and a validation set to train the infrared image detection model. As an example, the infrared image can be an infrared image of a photovoltaic panel taken from a high altitude, and the defect label can include but is not limited to the defect area and defect category of the photovoltaic panel in the infrared image.

[0026] Optionally, after obtaining the initial picture containing the infrared image, the coordinate transformation of the initial picture can be performed based on the distortion coefficient of the infrared camera and the camera internal parameter matrix, so as to obtain a distortion-corrected training picture. The data format of the initial picture is JPG, and the color palette of the infrared thermal imager (infrared camera) defaults to black-red mode, which can be adjusted to iron-red mode according to actual needs.

[0027] Specifically, on the one hand, due to the lens of the infrared camera carried by the drone, radial and tangential distortion will appear around the initial image, which needs to be corrected by the distortion coefficient. The distortion coefficients [k1, k2, p1, p2] include radial distortion coefficients k1 and k2, and tangential distortion coefficients p1 and p2.

[0028] Among them, radial distortion is caused by the irregularity of the lens shape and the modeling method, resulting in different focal lengths in different areas of the lens. The degree of deflection of light will change at a distance from the center of the lens, and may be greater, forming a pincushion distortion, such as Figure 2 As shown in (a), or the deflection is smaller, forming a barrel distortion, such as Figure 2 As shown in (b), the barrel distortion coefficient k1>0, and the pincushion distortion coefficient k1<0. The tangential distortion comes from the assembly process of the infrared camera. It is caused by the defects in the lens manufacturing that make the lens itself non-parallel to the image plane (camera sensor), as shown in Figure 3 shown.

[0029] On the other hand, the camera intrinsic matrix is ​​needed to establish the conversion relationship between world coordinates, camera coordinates and image coordinates. Figure 4, regard the infrared camera as a pinhole, and project the point P in the real world through the optical center O of the infrared camera onto the physical imaging plane to become point P'. Assume that the coordinates of point P in the real world (world coordinates) are [X, Y, Z], the coordinates of the imaged point P' (camera coordinates) are [X', Y', Z'], and the distance between the physical imaging plane and the optical center O is f (i.e., the focal length). The image coordinate system is introduced, and the pixel plane ouv is fixed on the physical imaging plane. The coordinates of P' in the image coordinate system (i.e., image coordinates) are recorded as [u,v]. The image coordinate system is usually defined in such a way that the origin o' is located in the upper left corner of the image, the u axis is parallel to the x axis to the right, the v axis is parallel to the y axis downward, and K is the camera intrinsic parameter matrix.

[0030] Therefore, the three-dimensional world coordinates can be converted into two-dimensional image coordinates through the coordinate conversion formula shown below:

[0031] As an example, the point in the world coordinate system can be first projected onto the physical imaging plane to obtain the coordinates of the point in the camera coordinate system; then, the point in the camera coordinate system can be projected onto the normalized plane, on which the Z-direction coordinate is normalized to 1 to obtain the normalized coordinates; then, the undistorted normalized coordinates of the point can be calculated based on the distortion coefficient and the distortion correction formula on the normalized coordinates; finally, the intrinsic parameter matrix K is used to map the point to the image coordinate system based on the corrected normalized coordinates, thereby converting the three-dimensional world coordinates into two-dimensional image coordinates to obtain an undistorted training image.

[0032] S102: Based on the training images, the defect detection results are output through a pre-built infrared image detection model.

[0033] Among them, the number of prediction boxes generated by the infrared image detection model is greater than the real target; the category labels in the defect detection results are selected from the real category to which the real target (Ground Truth) belongs and the empty category Φ representing the absence of real targets. The real target refers to the object or area that is exactly identified in the data.

[0034] Specifically, for an infrared image, assuming that the number of real targets contained therein is m, and the number of targets that can be detected by the preset infrared image detection model is N, then in the embodiment of the present application, N is greater than m, preferably N>>m, that is, the number of prediction boxes generated by the infrared image detection model is much greater than the number of real targets. As an example, the number of prediction boxes generated by the infrared image detection model can be 50.

[0035] In order to quickly match the positions of predicted targets and real targets, the embodiment of the present application constructs an empty category Φ that represents no real target and adds it to the target category. Then, Nm predicted targets that do not correspond to any real target can be paired with the empty category Φ, so that the pairing task of predicted targets and real targets can be regarded as a bipartite graph matching problem of two sets of equal capacity, simplifying the matching process and improving the efficiency of model training. Among them, the target category includes the real category and the empty category.

[0036] S103: Calculate a first loss function based on the bipartite graph matching result of the defect detection result and the defect label corresponding to the training image.

[0037] Specifically, for the defect detection result (prediction box W) and the defect label (true target U) corresponding to the training image, two node sets can be created to construct a bipartite graph. j , w j ), if the predicted box w j Covering the real target u j , then add an edge between the two nodes and assign a weight to the edge based on the intersection over union (IoU). In matching, if a predicted box overlaps with multiple true targets, the true target with the highest IoU value is selected for matching; if a true target is covered by multiple predicted boxes, the predicted box with the highest IoU value is selected; for redundant predicted boxes, by matching them with empty categories, the results of other correct matches can be avoided. During the training process, adjusting the first loss function based on the bipartite graph matching results can minimize the difference between the predicted box and the true target, which helps to improve the model's detection accuracy for small targets.

[0038] Specifically, the first loss function for: .

[0039] Among them, L iou is the intersection over union (IoU); Hyperparameters for controlling the IoU loss weight; is the L1 norm loss; To control the hyperparameter of the L1 norm loss weight, it can be set to 0.1; is the indicator function, when c i Returns 1 when not empty. i Returns 0 if empty; is the predicted probability distribution, where is a permutation function used to match the category labels in the defect detection results with the defect labels corresponding to the training images; is the bounding box loss function, which is used to measure the predicted box and the ground-truth bounding box b i The difference between.

[0040] S104: Calculate the sum of the first loss function and the multi-layer perception loss term to obtain a loss function.

[0041] Among them, the multi-layer perceptual loss term includes the sum of the weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model.

[0042] Specifically, the multi-layer perceptual loss term L percep for: .

[0043] in, It represents the perceptual loss corresponding to the first layer activation function in the infrared image detection model, and its weight is 1 / 8; It represents the perceptual loss corresponding to the second layer activation function in the infrared image detection model, and its weight is 1 / 4; It represents the perceptual loss corresponding to the activation function of the third layer in the infrared image detection model, and its weight is 1 / 2; Represents the perceptual loss corresponding to the fourth layer activation function in the infrared image detection model, and its weight is 1.

[0044] Loss function L total for: .

[0045] Therefore, the introduction of multi-layer perceptual loss terms in the loss function effectively improves the feature extraction capability and detail restoration of the infrared image detection model. By weighting each layer, it can ensure that the impact of different levels of perceptual loss on the final result is reasonably balanced, thereby enhancing the ability to capture low-level details while retaining high-level information. This multi-level perceptual loss fusion strategy can better constrain small targets and predicted targets, and achieve more accurate detection of small targets in infrared images.

[0046] S105: Iteratively train the infrared image detection model based on the loss function.

[0047] Specifically, after calculating the loss function, the internal parameters of the infrared image detection model can be adjusted based on the loss function to complete the training of the current iteration round. If the current iteration round is greater than or equal to the total iteration rounds or the accuracy is less than or equal to the preset accuracy, the training can be terminated. If the current iteration round is less than the total iteration rounds and the accuracy is greater than the preset accuracy, the next iteration round can be entered, and steps S101 to S104 are repeated to obtain the training pictures for the next iteration round. After calculating the loss function of the next iteration round, the infrared image detection model is trained based on the loss function of the next iteration round until the training end conditions are met (the total iteration rounds are reached, or the preset accuracy is reached).

[0048] As an example, the Adam optimizer can be used to iteratively train the infrared image detection model based on the loss function, and the learning rate can be set to 1×10 -4 .

[0049] Therefore, in the embodiment of the present application, a composite loss function is used to train the infrared image detection model based on the first loss function obtained from the bipartite graph matching result and the multi-layer perception loss term, and the trained infrared image detection model can achieve more accurate detection of small targets in infrared images. Among them, by adjusting the first loss function based on the bipartite graph matching result, the difference between the predicted frame and the real target can be minimized, which helps to improve the detection accuracy of the infrared image detection model for small target objects; and by using the multi-layer perception loss term, the feature extraction ability and detail restoration of the infrared image detection model can be effectively improved. By weighting layer by layer, it is ensured that the influence of different levels of perception loss on the final result is reasonably balanced, thereby retaining high-level information while enhancing the ability to capture low-level details, and better constraining small real targets and predicted targets.

[0050] See also Figure 5 , this figure is an architecture diagram of an infrared image detection model provided in an embodiment of the present application, and the infrared image detection model includes: Convolutional Neural Networks (CNN), spectral domain Transformer encoder (Transformer Encoder), multiscale Transformer decoder (Multiscale Transformer Decoder) and a fully connected layer (Feedforward Neural Network, FNN).

[0051] Among them, the convolutional neural network is used to extract the feature markers of the training images; the spectral domain Transformer encoder is used to encode the feature markers to obtain the encoded features; the multi-scale Transformer decoder is used to obtain the pixel-by-pixel defect prediction results based on the input encoded features; the fully connected layer is used to output the defect detection results of the training images based on the defect prediction results.

[0052] Specifically, the training image of the input infrared image detection model can be resized to a preset size, such as 640×512 pixels. Then, the feature token of the adjusted image is extracted through a restrictive convolutional neural network (Restrictive CNN) to output a feature map. The feature vector in the feature map is flattened and position embedding is added to retain spatial information. Then, the position-embedded feature vector (Embedded Tokens) is sent to the spectral domain Transformer encoder for encoding. The output of the spectral domain Transformer encoder is rearranged into a two-dimensional matrix form (Unflatten) and then enters the spectral domain processing module. The processed feature map is sent to the multi-scale Transformer decoder (Transformer The decoder outputs pixel-by-pixel defect prediction results, including prediction boxes and category labels. Finally, the fully connected layer (FNN) performs final processing to obtain defect detection results. In the defect detection results, each prediction box corresponds to a category label. The category labels may include but are not limited to labels such as classbox and no object.

[0053] Assume that X represents the photovoltaic infrared image after distortion correction, represents the output image with k classified defect detection boxes. The detection model aims to generate an output image with k classified defect detection boxes , and as close as possible to the real defect classification image Y, its workflow is defined as follows: .

[0054] CNN(.) stands for Restrictive CNN feature extractor, which specifically limits the size of the convolution kernel of the convolution layer to ensure the accuracy of detecting small targets. Encoder(.) stands for spectral domain Transformer encoder, Decoder(.) stands for multi-scale Transformer decoder, and FFN(.) stands for the classifier as a standard fully connected layer. represents the feature map extracted by the encoder, Represents the features captured by the feature extractor.

[0055] Among them, the spectral domain Transformer encoder contains a multi-head attention mechanism (Multi-Head Attention), a normalization layer (Norm) and a fully connected layer (MLP).

[0056] See also Figure 6 , this figure is a structural diagram of a spectral domain Transformer encoder provided in an embodiment of the present application. The feature map of the input spectral domain Transformer encoder is first encoded by an encoder module; then, the encoded feature map is sent to a series of linear transformation modules and spectral domain change modules, and linear Transformer and spectral Transformer are performed respectively. The input is modeled through Fourier operation, and the input image is mapped from the spatial domain to the spectral domain, the association between different positions in the image is established, and the global dependency between different feature markers is captured; the data after the linear Transformer and spectral Transformer are sent to the decoder for final feature reconstruction and output generation, thereby realizing the conversion of the input image into a high-dimensional feature representation, so as to facilitate target recognition, positioning and classification through these feature representations.

[0057] In the embodiment of the present application, the core calculation formula for the spectral domain transformation performed by the spectral domain Transformer encoder is: .

[0058] Where v represents data input, F represents Fourier operation, and F -1 represents the inverse Fourier operation, K represents the kernel function, and · represents matrix multiplication.

[0059] The spectral domain Transformer encoder provided in the embodiment of the present application adopts the general form of the attention mechanism for exhaustiveness. Except for the details of position encoding, the mechanism follows the traditional Transformer. Multi-head attention is the concatenation of M single heads, followed by linear projection (L), which is mathematically expressed as: ; Among them, attn represents the attention mechanism; T represents the attention head; Represents the output of the Mth attention head; the [] operation indicates concatenation in the channel dimension.

[0060] Optionally, residual connections can be used to help the spectral domain Transformer encoder better learn long-term dependencies; dropout can be used to prevent overfitting; layer normalization can be used to ensure that the input of each layer has a similar distribution to stabilize and accelerate the training process. As a result, the final output can be: .

[0061] in, ; represents the final output, layernorm refers to the layer normalization operation; dropout represents the dropout operation; L is the linear projection matrix.

[0062] See also Figure 7 , this figure is a structural diagram of a multi-scale Transformer decoder provided in an embodiment of the present application. The multi-scale Transformer decoder is an architecture that enhances the decoder capabilities of the Transformer model, including a multi-scale feature extraction module and a multi-layer regression prediction module. It can process information at different scales, better capture multi-level information in the input image, and effectively identify and locate small targets in the image.

[0063] Specifically, in the multi-scale feature extraction module, Backbone is a pre-trained convolutional neural network (CNN) used to extract basic features of the image, which may include convolutional neural networks such as ResNet or VGG; the feature map extracted by Backbone is divided into multiple small patches (patches), each patch is flattened and converted into a vector form, and multi-scale features are extracted; the global token is added as an additional token to the segmented patch token sequence, and the patch with the global token is converted into a higher-dimensional representation Embedded Patches through an embedding layer, and then fused with the local features extracted by Backbone to obtain the fused features, which are output to the multi-layer regression prediction module. Among them, Transformer Layer 1 to Transformer Layer N represent multiple Transformer layers connected in series. Each Transformer layer contains a multi-head attention mechanism (Multi-HeadAttention), a normalization layer (Norm) and a feedforward neural network (MLP). The multi-scale feature extraction module extracts multi-scale features from the encoding features provided by the spectral domain Transformer encoder based on the multi-head attention mechanism, layer normalization operation and full connection operation.

[0064] The process of multi-scale feature extraction is defined as follows: ; ; .

[0065] Among them, MSA represents the multi-head attention mechanism, LN represents the layer normalization operation; MLP represents the fully connected operation; It represents the fusion of different features. The Fusion operation is a convolution layer with a convolution kernel size of 1.

[0066] In the multi-layer regression prediction module, the features at different levels in the Level Classify classification fusion feature are used. Level 1, Level 2, ..., Level M are the feature representations of multiple levels. Finally, the features at different levels are integrated by weighted summation (WeightSum) to generate the final prediction result.

[0067] The calculation process of multi-level regression prediction is defined as follows: .

[0068] On the basis of extracting fusion features from multi-scale features, the formula Perform softmax weight scaling, and then, through the formula , perform feature mapping; finally, through the formula A weighted sum is performed to obtain the final feature representation, which is the defect prediction result. The feature representation is decoded into a pixel-level infrared image through a fully connected layer.

[0069] See also Figure 8 , this figure is a schematic diagram of a multi-level encoding and decoding connection mechanism provided by an embodiment of the present application. Taking the depth of the spectral domain Transformer encoder and the multi-scale Transformer decoder as 6 as an example, the spectral domain Transformer encoder and the multi-scale Transformer decoder are set to a symmetrical structure, wherein the last layer of the spectral domain Transformer encoder inputs the encoded features into the decoding modules of 6 different levels (depths) in the multi-scale Transformer decoder, and performs channel splicing operations. In this way, the problem of important feature loss of spatial features in the encoding and decoding process can be reduced, and the spatial features in the image can be more accurately retained, thereby facilitating more accurate identification of numerous small targets in infrared images.

[0070] Optionally, the infrared image detection model provided in the embodiment of the present application has 256 input feature channels.

[0071] Optionally, in the infrared image detection model provided in an embodiment of the present application, the Fourier module depth of the spectral domain Transformer encoder is 4, and the number of layers of the multi-scale Transformer decoder is 4, representing four different levels of spatial resolution of 1, 1 / 2, 1 / 4 and 1 / 8 respectively.

[0072] See also Fig. 9 , which is a schematic diagram of a training device for an infrared image detection model provided in an embodiment of the present application, the device includes: an acquisition module 901, an output module 902, a first calculation module 903, a second calculation module 904 and a training module 905; An acquisition module 901 is used to acquire a training picture including an infrared image; Output module 902, used to output defect detection results based on training images through a pre-built infrared image detection model; the number of prediction boxes generated by the infrared image detection model is greater than the real target; the defect detection results include the real category to which the real target belongs and the empty category representing the absence of the real target; A first calculation module 903 is used to calculate a first loss function based on a bipartite graph matching result of a defect detection result and a defect label corresponding to a training image; The second calculation module 904 is used to calculate the sum of the first loss function and the multi-layer perceptual loss term to obtain a loss function; the multi-layer perceptual loss term includes the sum of weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model; The training module 905 is used to iteratively train the infrared image detection model based on the loss function.

[0073] Therefore, in the embodiment of the present application, a composite loss function is used to train the infrared image detection model based on the first loss function obtained from the bipartite graph matching result and the multi-layer perception loss term, and the trained infrared image detection model can achieve more accurate detection of small targets in infrared images. Among them, by adjusting the first loss function based on the bipartite graph matching result, the difference between the predicted frame and the real target can be minimized, which helps to improve the detection accuracy of the infrared image detection model for small target objects; and by using the multi-layer perception loss term, the feature extraction ability and detail restoration of the infrared image detection model can be effectively improved. By weighting layer by layer, it is ensured that the influence of different levels of perception loss on the final result is reasonably balanced, thereby retaining high-level information while enhancing the ability to capture low-level details, and better constraining small real targets and predicted targets.

[0074] Optionally, another training device for an infrared image detection model provided in an embodiment of the present application also includes a preprocessing module for obtaining an initial picture containing an infrared image; the initial picture is collected by an infrared camera; based on the distortion coefficient of the infrared camera and the camera intrinsic parameter matrix, the initial picture is coordinate transformed to obtain a training picture.

[0075] In addition, the present application also provides an infrared image detection method, the method comprising: S1: Acquire a picture to be inspected including an infrared image.

[0076] Optionally, the image to be inspected may be obtained by correcting the distortion of an initial image captured by an infrared camera.

[0077] S2: Based on the image to be inspected, an infrared image detection model trained by the infrared image detection model training method described in any of the above embodiments is used to output defect detection results.

[0078] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0079] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method for an infrared image detection model, characterized in that: The method comprises: Get training pictures containing infrared images; Based on the training image, a defect detection result is output through a pre-built infrared image detection model; the number of prediction boxes generated by the infrared image detection model is greater than the real target; the category label in the defect detection result is selected from the real category to which the real target belongs and the empty category representing the absence of the real target; Calculating a first loss function based on a bipartite graph matching result between the defect detection result and the defect label corresponding to the training image; Calculating the sum of the first loss function and a multi-layer perceptual loss term to obtain a loss function; the multi-layer perceptual loss term includes the sum of weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model; The infrared image detection model is iteratively trained based on the loss function.

2. The method according to claim 1, characterized in that The infrared image detection model includes a convolutional neural network, a spectral domain Transformer encoder, a multi-scale Transformer decoder and a fully connected layer; The convolutional neural network is used to extract feature markers of the training image to obtain a feature map; The spectral domain Transformer encoder is used to encode the feature map to obtain encoded features; The multi-scale Transformer decoder is used to obtain a pixel-by-pixel defect prediction result based on the input encoding features; The fully connected layer is used to output the defect detection result of the training image based on the defect prediction result.

3. The method according to claim 2, characterized in that The depth of the spectral domain Transformer encoder and the multi-scale Transformer decoder are both 6; the spectral domain Transformer encoder and the multi-scale Transformer decoder are symmetrical structures; The last layer of the spectral domain Transformer encoder inputs the encoded features into decoding modules of various depths in the multi-scale Transformer decoder.

4. The method according to claim 2, characterized in that: The spectral domain Transformer encoder based on the self-attention mechanism is specifically used for: The feature map is mapped from the spatial domain to the spectral domain to obtain a coding feature; the coding feature includes a global dependency relationship between different feature markers.

5. The method according to claim 2, characterized in that: The multi-scale Transformer decoder includes a multi-scale feature extraction module and a multi-layer regression prediction module; the multi-scale Transformer decoder is specifically used to: Extracting multi-scale features from the coding features through the multi-scale feature extraction module based on a multi-head attention mechanism, a layer normalization operation, and a full connection operation; fusing the multi-scale features through a convolution layer with a convolution kernel size of 1 to obtain a fused feature; Through the multi-layer regression prediction module, a defect prediction result is obtained based on the fusion features.

6. The method according to claim 2, characterized in that The multi-scale Transformer decoder has 4 layers, and the spatial resolution levels corresponding to different layers are 1, 1 / 2, 1 / 4 and 1 / 8 respectively.

7. The method according to claim 1, characterized in that Before acquiring the training picture containing the infrared image, the method further includes: Acquire an initial picture including an infrared image; the initial picture is collected by an infrared camera; Based on the distortion coefficient of the infrared camera and the camera internal parameter matrix, the initial image is transformed into a training image.

8. The method according to claim 1, characterized in that The number of prediction boxes generated by the infrared image detection model is 50.

9. A training device for an infrared image detection model, characterized in that: The device comprises: an acquisition module, an output module, a first calculation module, a second calculation module and a training module; The acquisition module is used to acquire a training picture containing an infrared image; The output module is used to output defect detection results based on the training image through a pre-built infrared image detection model; the number of prediction boxes generated by the infrared image detection model is greater than the real target; the defect detection results include the real category to which the real target belongs and the empty category representing the absence of the real target; The first calculation module is used to calculate a first loss function based on a bipartite graph matching result of the defect detection result and a defect label corresponding to the training image; The second calculation module is used to calculate the sum of the first loss function and a multi-layer perceptual loss term to obtain a loss function; the multi-layer perceptual loss term includes the sum of weighted perceptual losses corresponding to the multi-layer activation functions in the infrared image detection model; The training module is used to iteratively train the infrared image detection model based on the loss function.

10. An infrared image detection method, characterized in that: The method comprises: Acquire the image to be inspected including the infrared image; Based on the image to be inspected, an infrared image detection model trained by the training method of an infrared image detection model according to any one of claims 1 to 8 outputs a defect detection result.

Images