Target defect detection method and system based on dual-light fusion

Abstract

The invention relates to a target defect detection method and system based on dual-light fusion. The method comprises the following steps: constructing a comprehensive loss function; performing iterative training on the target defect detection model and the dual-light fusion model by using the target detection loss and the comprehensive loss, and correspondingly outputting the dual-light fusion model and the target defect detection model; carrying out image registration; outputting a dual-light fusion image through the dual-light fusion model; and inputting the dual-light fusion image into a target defect detection model, and outputting a target defect detection result. According to the method, the comprehensive loss function containing the defect detection loss is constructed, so that the dual-light fusion model not only can pay attention to the visual quality of the fusion image, but also can pay attention to the features which are crucial to target defect detection, and therefore, joint training and synchronous optimization of the two models can be realized; and mutual learning and adaptation of the two models in the training process are ensured, so that the overall performance of the models is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target defect detection method and system based on dual-light fusion. Background Art

[0002] In the prior art, a single type of image data (such as visible light images or infrared images) is usually used to detect targets (such as power equipment, etc.), and target defect detection is further performed based on the target detection results. However, due to the single source of image data, the accuracy and precision of the detection results are not high, and the detection method has limited usage scenarios.

[0003] Based on this, a method has been developed to fuse visible light images and infrared images through a dual-light fusion model to improve target detection accuracy. However, this dual-light fusion model and target defect detection model are trained separately based on different loss functions, which makes it impossible to take into account both the dual-light fusion effect and the defect detection effect at the same time. Summary of the invention

[0004] The purpose of the present invention is to provide a target defect detection method and system based on dual-light fusion, which constructs a comprehensive loss function that includes defect detection loss, so that the dual-light fusion model can not only focus on the visual quality of the fused image, but also focus on features that are crucial to target defect detection, so as to realize joint training and synchronous optimization of the two models, and ensure that the two models can learn and adapt to each other during the training process, thereby greatly improving the overall performance of the model.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] On the one hand, a target defect detection method based on dual-light fusion is provided, which comprises the following steps:

[0007] Constructing the comprehensive loss function L total , and L total =Dual light fusion loss + defect detection loss;

[0008] Iterative training is performed on the target defect detection model and the dual-light fusion model using the target detection loss and the comprehensive loss. After the iterative training is completed, the dual-light fusion model and the target defect detection model are output accordingly.

[0009] Acquire an original visible light image and an original infrared image of the same scene, and perform image registration to obtain a registered visible light image and a registered infrared image accordingly;

[0010] Inputting the registered visible light image and the registered infrared image into the trained output dual-light fusion model, and outputting the dual-light fusion image through the dual-light fusion model;

[0011] The dual-light fusion image is input into the trained target defect detection model, and the target defect detection result is output.

[0012] Preferably, the dual light fusion loss = αL recon +βL perc ; L recon , L perc They are reconstruction loss and perceptual loss respectively; α and β are weight parameters.

[0013] Preferably, the defect detection loss = γ(L cls +L loc +L conf );L cls , L loc , L conf They are classification loss, positioning loss and confidence loss respectively; γ is the weight parameter.

[0014] Preferably, the target detection loss includes the classification loss L cls , positioning loss L loc , confidence loss L conf One or more of the .

[0015] Preferably, the reconstruction loss is defined as:

[0016]

[0017] Among them, L recon represents the reconstruction loss; represents a dual light fusion image; Indicates infrared image; represents a visible light image; represents the reference image; N represents the number of samples.

[0018] Preferably, the perceptual loss is defined as:

[0019]

[0020] Among them, L recon Indicates perceived loss; Represents the j-th layer feature extraction of the image; M represents the number of feature layers; N represents the number of samples in a training batch.

[0021] Preferably, the classification loss is defined as:

[0022]

[0023] Among them, L cls represents the classification loss; They represent the true label and predicted probability of the cth category of the ith sample respectively; C represents the number of categories.

[0024] Preferably, the positioning loss is defined as:

[0025]

[0026] Among them, L loc represents the positioning loss; b i , They represent the true bounding box and predicted bounding box of the i-th sample respectively.

[0027] Preferably, the confidence loss is defined as:

[0028]

[0029] Among them, L conf represents the confidence loss; y i Represents the true confidence label of the i-th sample; Indicates the confidence of the prediction.

[0030] On the other hand, a target defect detection system is also provided, which includes:

[0031] Comprehensive loss construction module, which is used to construct a comprehensive loss function;

[0032] A target defect detection model training module is used to optimize the model parameter setting of the target defect detection model by using the target detection loss, so as to perform multiple iterative training on the target defect detection model, and output the target defect detection model after the iterative training is completed;

[0033] A dual-light fusion model training module, which is used to optimize the model parameter settings of the dual-light fusion model by using comprehensive loss, so as to perform multiple iterative training on the dual-light fusion model, and output the dual-light fusion model after the iterative training is completed;

[0034] An image registration module is used to perform image registration on an original visible light image and an original infrared image of the same scene to obtain a registered visible light image and a registered infrared image accordingly;

[0035] A dual-light fusion module, which is used to fuse the registered visible light image and the registered infrared image through the dual-light fusion model output by the dual-light fusion model training module to output a dual-light fusion image;

[0036] The target defect detection module is used to perform defect detection on the target in the dual-light fusion image output by the dual-light fusion module through the target defect detection model output by the target defect detection model training module, and output the target defect detection result.

[0037] In summary, the present invention has the following beneficial effects compared with the prior art:

[0038] The present invention constructs a comprehensive loss function including defect detection loss, so that the dual-light fusion model can not only focus on the visual quality of the fused image, but also on the features that are crucial to target defect detection. After the comprehensive loss function is back-propagated, the model parameters can be updated to achieve joint training and synchronous optimization of the two models, ensuring that the two models can learn and adapt to each other during the training process, thereby greatly improving the overall performance of the model.

[0039] Furthermore, the present invention also specifically defines reconstruction loss, perception loss, classification loss, positioning loss, and confidence loss, respectively, so as to improve the training effect of the dual-light fusion model and the target defect detection model and improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of the steps of the target defect detection method in the present invention;

[0041] Figure 2 The original visible light image, the original infrared image, and the target defect detection result image output by the target defect detection method of the present invention;

[0042] Figure 3 It is a schematic diagram of the structure of the target defect detection system in the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] Example 1

[0045] like Figure 1 The present embodiment provides a target defect detection method based on dual-light fusion, which includes the following steps:

[0046] S1. Construct a comprehensive loss function, and the comprehensive loss function is obtained by formula (1):

[0047] L total = Dual-light fusion loss + defect detection loss (1)

[0048] Among them, L total Represents the comprehensive loss; the dual light fusion loss = αL recon +βL perc ; The defect detection loss = γ(Lcls +L loc +L conf );L recon , L perc They are reconstruction loss (Reconstruction Loss) and perceptual loss (Perceptual Loss); L cls , L loc , L conf They are classification loss, localization loss, and confidence loss; α, β, and γ are all weight parameters used to balance the contribution of various losses to the comprehensive loss. For example, in this embodiment, the value ranges of α, β, and γ can all be [0, 1], and α+β+γ=1.

[0049] S2. By using the back propagation algorithm, the target defect detection model is optimized by using the target detection loss, and the dual-light fusion model is optimized by using the comprehensive loss, so as to perform multiple iterative training on the target defect detection model and the dual-light fusion model respectively. After the iterative training is completed and the losses of the dual-light fusion model and the target defect detection model are converged, the dual-light fusion model and the target defect detection model are outputted accordingly;

[0050] In this embodiment, the target defect detection model includes a YOLO model (specifically, a YOLOv8 model), the target includes one or more of power equipment and personnel operating the power equipment, and the power equipment includes one or more of an oil storage cabinet, a cooler, a submersible pump, a relay, a metal expander, an insulator, a connection fitting, a voltage grading ring, a lightning arrester, a hydraulic gauge, an oil temperature gauge, a conductive joint, a conductive lead, etc.; the target defect includes one or more of oil leakage, deformation, breakage, looseness, rust, surface contamination, abnormal reading, abnormal personnel behavior (including not wearing work clothes, not wearing a safety helmet, smoking), etc.;

[0051] The target detection loss includes the classification loss L cls , positioning loss L loc , confidence loss L conf One or more of the following;

[0052] S3, obtaining an original visible light image and an original infrared image of the same scene, and performing image registration to obtain a registered visible light image and a registered infrared image correspondingly; wherein the original visible light image and the original infrared image can be obtained based on corresponding imaging devices, respectively, the imaging devices include visible light cameras and infrared cameras, and the imaging devices can be mounted on unmanned equipment, the unmanned equipment includes drones, unmanned vehicles, etc.;

[0053] Furthermore, image registration includes the following steps:

[0054] The original visible light image and the original infrared image are respectively subjected to image preprocessing to obtain a preprocessed visible light image and a preprocessed infrared image; since the imaging device is affected by factors such as environmental weather, light, and imaging angle when acquiring an image, resulting in poor spatial consistency of the obtained image pixels, the image needs to be subjected to corresponding image preprocessing before image registration and fusion. In this embodiment, the preprocessing includes one or more of: image denoising, image scaling, edge enhancement, etc.;

[0055] Performing image registration on the preprocessed visible light image and the preprocessed infrared image to obtain a registered visible light image and a registered infrared image accordingly. In this embodiment, the image registration method is a prior art and will not be described in detail.

[0056] S4, inputting the registered visible light image and the registered infrared image into the dual-light fusion model output after training in step S2, and outputting the dual-light fusion image through the dual-light fusion model;

[0057] Specifically, outputting a dual-light fusion image through a dual-light fusion model includes the following steps:

[0058] Image feature extraction is performed on the registered visible light image and the registered infrared image respectively to obtain visible light image features and infrared image features respectively; in this embodiment, commonly used feature extraction networks such as VGG, ResNet, and Transformer can be used to extract image features;

[0059] The visible light image features and the infrared image features are fused to obtain a dual-light fusion image; in this embodiment, the image feature fusion can be realized based on a feature fusion network module, and the feature fusion network module is composed of four convolution layers with a convolution kernel size of 3×3 and one convolution layer with a convolution kernel size of 1×1;

[0060] S5. Input the dual-light fusion image into the target defect detection model output after training in step S2, and output the target defect detection result.

[0061] In the prior art, the training of the dual-light fusion model and the target defect detection model are usually carried out separately, and they are iteratively trained through independent loss functions respectively, resulting in that the two can only optimize their own parameter settings and results (that is, only optimize the fusion effect, or only optimize the defect detection accuracy), but cannot take into account both the dual-light fusion effect and the target defect detection accuracy at the same time.

[0062] In this embodiment, by constructing a comprehensive loss function that includes defect detection loss, the dual-light fusion model can not only focus on the visual quality of the fused image, but also focus on features that are critical to target defect detection, avoiding ignoring detailed information that is helpful for target defect detection; at the same time, the comprehensive loss function obtained after fusion weighted calculation can be back-propagated to update the model parameters, so as to achieve joint training and synchronous optimization of the two models, ensuring that the two models can learn and adapt to each other during the training process, thereby greatly improving the overall performance of the model.

[0063] Specifically, Figure 2 As shown, parts (a) and (b) are the original visible light image and the original infrared image of the same scene, respectively, wherein the details of the insulator in the box are not clear. After the original images are aligned and input into the dual-light fusion model in this embodiment for fusion, the fused image of part (c) is output, and in the fused image, the details of the insulator (i.e., the parts in the box) can be clearly displayed.

[0064] Embodiment 2:

[0065] The difference between this embodiment and embodiment 1 is that the reconstruction loss is defined as:

[0066]

[0067] Among them, L recon Represents the reconstruction loss represents a dual light fusion image; Indicates infrared image; represents a visible light image; represents the reference image. In this embodiment, the visible light image as the reference image; N represents the number of samples;

[0068] Since the output dual-light fusion image may lose important texture information compared to the visible light original image after dual-light image fusion, the reconstruction loss is used in this embodiment to reduce the texture information loss generated by the dual-light fusion model in the process of constructing the fused image, so as to reduce the difference between the dual-light fusion image and the high-resolution reference image; at the same time, the mean absolute error (MAE) can also be used as the reconstruction loss function in this embodiment.

[0069] The perceptual loss is defined as:

[0070]

[0071] Among them, L recon Indicates perceived loss; Represents the image (i.e., dual-light fusion image or reference image )’s j-th layer feature extraction; M represents the number of feature layers; N represents the number of samples in a training batch;

[0072] Perceptual loss is used to measure the difference between the fused image and the reference image in the feature space, which is helpful to enhance the detail information of the output. Usually, a pre-trained convolutional neural network (such as VGG) is used to extract features and calculate the error in the feature space.

[0073] Furthermore, the classification loss is defined as:

[0074]

[0075] Among them, L cls represents the classification loss; They represent the true label and predicted probability of the cth category of the ith sample respectively; C represents the number of categories; thus, the difference between the predicted target defect category and the true category can be measured by the above classification loss.

[0076] The positioning loss is defined as:

[0077]

[0078] Among them, L loc represents the positioning loss; b i , They respectively represent the true bounding box and the predicted bounding box of the i-th sample. Therefore, the difference between the predicted bounding box position and the true position can be measured by the above positioning loss. It should be noted that, in this embodiment, IoU loss can also be used as the positioning loss function.

[0079] The confidence loss is defined as:

[0080]

[0081] Among them, L conf represents the confidence loss; y i represents the true confidence label of the i-th sample (1 means defective, 0 means no defect); Represents the confidence of the prediction; thus, the difference between the confidence of whether the predicted bounding box contains a defect and the actual situation can be measured by the above confidence loss.

[0082] Embodiment 3:

[0083] This embodiment provides a target defect detection system for implementing the target defect detection method described in Embodiment 1 or 2, such as Figure 3 As shown, the target defect detection system includes:

[0084] A comprehensive loss construction module 1, which is used to construct the comprehensive loss function;

[0085] A target defect detection model training module 2 is used to optimize the model parameter setting of the target defect detection model by using the target detection loss, so as to perform multiple iterative training on the target defect detection model, and output the target defect detection model after the iterative training is completed;

[0086] A dual-light fusion model training module 3 is used to optimize the model parameter settings of the dual-light fusion model by using comprehensive loss, so as to perform multiple iterative training on the dual-light fusion model, and output the dual-light fusion model after the iterative training is completed;

[0087] An image registration module 4 is used to perform image registration on the original visible light image and the original infrared image of the same scene to obtain a registered visible light image and a registered infrared image correspondingly, and the registration process is the same as step S3;

[0088] A dual-light fusion module 5 is used to fuse the registered visible light image and the registered infrared image through the dual-light fusion model output by the dual-light fusion model training module 3 to output a dual-light fusion image;

[0089] The target defect detection module 6 is used to perform defect detection on the target in the dual-light fusion image output by the dual-light fusion module 5 through the target defect detection model output by the target defect detection model training module 2, and output the target defect detection result.

[0090] In summary, the present invention constructs a comprehensive loss function including defect detection loss, so that the dual-light fusion model can not only focus on the visual quality of the fused image, but also focus on the features that are crucial to target defect detection, avoiding ignoring the detailed information that is helpful for target defect detection; at the same time, the comprehensive loss function obtained after fusion weighted calculation can be back-propagated to update the model parameters, so as to realize the joint training and synchronous optimization of the two models, ensuring that the two models can learn and adapt to each other during the training process, thereby greatly improving the overall performance of the model;

[0091] Furthermore, the present invention also specifically defines reconstruction loss, perception loss, classification loss, positioning loss, and confidence loss, respectively, so as to improve the training effect of the dual-light fusion model and the target defect detection model and improve the detection accuracy.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A target defect detection method based on dual-light fusion, characterized in that: The steps include: Constructing the comprehensive loss function L total , and L total =Dual light fusion loss + defect detection loss; Iterative training is performed on the target defect detection model and the dual-light fusion model using the target detection loss and the comprehensive loss. After the iterative training is completed, the dual-light fusion model and the target defect detection model are output accordingly. Acquire an original visible light image and an original infrared image of the same scene, and perform image registration to obtain a registered visible light image and a registered infrared image accordingly; Inputting the registered visible light image and the registered infrared image into the trained output dual-light fusion model, and outputting the dual-light fusion image through the dual-light fusion model; The dual-light fusion image is input into the trained target defect detection model, and the target defect detection result is output.

2. The target defect detection method according to claim 1, characterized in that: The dual light fusion loss = αL recon +βL perc ; L recon , L perc They are reconstruction loss and perceptual loss respectively; α and β are weight parameters.

3. The target defect detection method according to claim 1, characterized in that: The defect detection loss = γ(L cls +L loc +L conf );L cls , L loc , L conf They are classification loss, positioning loss and confidence loss respectively; γ is the weight parameter.

4. The target defect detection method according to claim 3, characterized in that: The target detection loss includes the classification loss L cls , positioning loss L loc , confidence loss L conf One or more of the .

5. The target defect detection method according to claim 3, characterized in that: The reconstruction loss is defined as: Among them, L recon represents the reconstruction loss; represents a dual light fusion image; Indicates infrared image; represents a visible light image; represents the reference image; N represents the number of samples.

6. The target defect detection method according to claim 3, characterized in that: The perceptual loss is defined as: Among them, L recon Indicates perceived loss; Represents the j-th layer feature extraction of the image; M represents the number of feature layers; N represents the number of samples in a training batch.

7. The target defect detection method according to claim 4, characterized in that: The classification loss is defined as: Among them, L cls represents the classification loss; They represent the true label and predicted probability of the cth category of the ith sample respectively; C represents the number of categories.

8. The target defect detection method according to claim 4, characterized in that: The positioning loss is defined as: Among them, L loc represents the positioning loss; b i , They represent the true bounding box and predicted bounding box of the i-th sample respectively.

9. The target defect detection method according to claim 4, characterized in that: The confidence loss is defined as: Among them, L conf represents the confidence loss; y i Represents the true confidence label of the i-th sample; Indicates the confidence of the prediction.

10. A target defect detection system, characterized in that: include: Comprehensive loss construction module, which is used to construct a comprehensive loss function; A target defect detection model training module is used to optimize the model parameter setting of the target defect detection model by using the target detection loss, so as to perform multiple iterative training on the target defect detection model, and output the target defect detection model after the iterative training is completed; A dual-light fusion model training module, which is used to optimize the model parameter settings of the dual-light fusion model by using comprehensive loss, so as to perform multiple iterative training on the dual-light fusion model, and output the dual-light fusion model after the iterative training is completed; An image registration module is used to perform image registration on an original visible light image and an original infrared image of the same scene to obtain a registered visible light image and a registered infrared image accordingly; A dual-light fusion module, which is used to fuse the registered visible light image and the registered infrared image through the dual-light fusion model output by the dual-light fusion model training module to output a dual-light fusion image; The target defect detection module is used to perform defect detection on the target in the dual-light fusion image output by the dual-light fusion module through the target defect detection model output by the target defect detection model training module, and output the target defect detection result.

Images