Defect detection method, device, computer equipment and computer readable storage medium

By using an image downsampling model and feature combination layer based on the self-attention mechanism in image defect detection, the image captured by the drone is solved, and the accuracy and accuracy of defect detection are improved.

CN118608521BActive Publication Date: 2024-11-26SHENZHEN SMARTMORE TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411074096.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-26
Estimated Expiration
2044-08-07

AI Technical Summary

Technical Problem

When existing image defect detection technology performs defect detection on larger images, preprocessing that directly reduces image resolution cannot effectively retain image detail information, resulting in low detection accuracy, especially in images taken by drones. It is difficult to accurately detect small defects.

Method used

Using an image downsampling model based on the self-attention mechanism, downsampling the image to be detected through at least two downsampling layers to obtain features of different sizes, and combining these features through feature combination layers to enhance attention within the global scope of the image. Force information analysis and multi-dimensional feature inference, and finally, the combined image features are input into the target defect detection model for processing to improve detection accuracy.

Benefits of technology

By retaining the detailed feature information of the image, the accuracy of defect detection of images captured by the drone is significantly improved, and small defects in the image can be detected more accurately, reducing random noise in the detection results, and enhancing the target defect detection model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118608521B_ABST
    Figure CN118608521B_ABST
Patent Text Reader

Abstract

The present application relates to a defect detection method, device, computer equipment and computer-readable storage medium. The method includes: obtaining an image to be detected, a trained image downsampling model and a trained target defect detection model; the image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism; the image to be detected is input into the image downsampling model, and the image to be detected is downsampled through at least two downsampling layers to obtain at least two image features of different sizes to be combined; at least two image features to be combined are combined through the feature combination layer to obtain a combined image feature; the combined image feature is input into the target defect detection model for processing, and the defect detection result corresponding to the image to be detected is output. The present application can improve the accuracy of detecting defects in images taken by drones.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a defect detection method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] With the development of computer technology, image defect detection technology is applied to more and more fields. When existing image defect detection technology is used to detect defects in large-sized images (such as images taken by drone inspection), it generally pre-processes the large-sized images by directly reducing the resolution. However, the pre-processing of directly reducing the image resolution cannot well retain more detailed information of the image, so when detecting small-sized defects, the detected defects can only be located in the approximate area, resulting in low accuracy of image defect detection. Summary of the invention

[0003] Based on this, it is necessary to provide a defect detection method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems, which can improve the accuracy of detecting defects in images taken by drones.

[0004] In a first aspect, the present application provides a defect detection method, comprising:

[0005] Obtaining an image to be detected, a trained image downsampling model, and a trained target defect detection model; the image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism;

[0006] Inputting the image to be detected into the image downsampling model, downsampling the image to be detected through at least two downsampling layers to obtain at least two image features to be combined in different sizes; combining the at least two image features to be combined through the feature combination layer to obtain a combined image feature;

[0007] The combined image features are input into the target defect detection model for processing, and the defect detection results corresponding to the image to be detected are output.

[0008] In a second aspect, the present application provides a defect detection device, comprising:

[0009] An acquisition module, used to acquire an image to be detected, a trained image downsampling model, and a trained target defect detection model; the image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism;

[0010] The first processing module is used to input the image to be detected into the image downsampling model, perform downsampling processing on the image to be detected through at least two downsampling layers to obtain at least two image features to be combined in different sizes; and combine the at least two image features to be combined through the feature combination layer to obtain a combined image feature;

[0011] The second processing module is used to input the combined image features into the target defect detection model for processing, and output the defect detection result corresponding to the image to be detected.

[0012] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0014] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program implements the steps in the above method when executed by a processor.

[0015] The above-mentioned defect detection method, apparatus, computer device, computer-readable storage medium and computer program product, by inputting the acquired image to be detected into a trained image downsampling model including at least two downsampling layers and a feature combination layer constructed based on the self-attention mechanism, downsampling the image to be detected through the at least two downsampling layers to obtain at least two image features to be combined of different sizes, and then combining the at least two image features to be combined through the feature combination layer to obtain the combined image feature, is conducive to analyzing the attention information of the image to be detected in a global range according to the downsampling layer constructed based on the self-attention mechanism, and according to the self-attention mechanism The downsampling layer constructed by the system realizes the inference of multi-size features of the image to be detected, adds additional features of the image to be detected that are higher or lower than the normal value, and better retains more detail feature information in the image to be detected; finally, the combined image features are input into the target defect detection model for processing to obtain the defect detection result corresponding to the image to be detected, so that on the basis of retaining more detail feature information when processing the image to be detected by the image downsampling model, further deep defect detection analysis is performed based on the target defect detection model, which is conducive to the target defect detection model to more accurately detect and analyze the defects of the image to be detected, thereby greatly improving the accuracy of detecting image defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An application environment diagram of a defect detection method provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of a defect detection method provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a process of image defect detection provided in an embodiment of the present application;

[0019] Figure 4 A structural block diagram of a defect detection device provided in an embodiment of the present application;

[0020] Figure 5 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0021] Figure 6 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0022] Figure 7 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The defect detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0025] like Figure 2 As shown, the embodiment of the present application provides a defect detection method, which is applied to Figure 1 The terminal 102 or the server 104 in the example is used for explanation. It is understandable that the computer device may include at least one of the terminal and the server. The method comprises the following steps:

[0026] S200, obtaining an image to be detected, a trained image downsampling model, and a trained target defect detection model; the image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism.

[0027] The image to be detected is obtained by taking pictures with a camera mounted on a drone; it may be an image of a power transmission line taken with a camera mounted on a drone, and the image includes but is not limited to key electrical equipment such as poles, insulators, conductors, line corridors and hardware; the camera's shooting resolution may be 5000×5000 pixels or others. The image downsampling model refers to a model used to downsample the image to be detected to obtain feature maps of multiple sizes, and may be an image downsampling model formed by at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism. The target defect detection model refers to a model used to detect defects in an image, and may be but is not limited to a YOLOv5 neural network model. The downsampling layer is used to downsample image features, and may be a multi-head self-attention mechanism network, which is used to focus on feature information of different representation subspaces in the image to be detected, which can enhance the ability of the image downsampling model to capture different types of relationships and dependencies. The feature combination layer is used to splice and combine image features of different sizes, that is, to splice feature matrices of image features of different sizes.

[0028] For example, since the images taken by drone inspections are large in size, if the image resolution is directly reduced to obtain an image size suitable for detection by the defect detection model, more detail information in the image will be lost. In order to better capture the local feature information of the images taken by the drone at extremely long viewing distances and the global feature information at extremely close viewing distances, an image downsampling model can be added before the image is detected based on the defect detection model. Before using the image downsampling model and the defect detection model to detect defects in an image, the image downsampling model and the defect detection model are first trained. The image downsampling model and the defect detection model can be trained in an end-to-end manner, for example: obtaining a set of images to be trained, training the image downsampling model based on the set of images to be trained, obtaining combined image features corresponding to each image to be trained obtained by analyzing the set of images to be trained by the image downsampling model during the training process, wherein the combined image features include image features of multiple sizes corresponding to the image to be trained; using the combined image features as input to the defect detection model, training the defect detection model, and obtaining a trained defect detection model and a trained image downsampling model; screening image downsampling models and defect detection models that meet preset model conditions from the trained image downsampling model and the trained defect detection model, using the screened image downsampling model as a trained image downsampling model for downsampling the image to be detected, and using the screened defect detection model as a target defect detection model for defect detection of the image to be detected. Using an end-to-end approach in the training process of the image downsampling model and the defect detection model can reduce the complexity of model training, avoid the impact of training bias caused by training each model separately on the overall image defect detection results, and help improve the performance of each model in detecting the same task.

[0029] S202, input the image to be detected into the image downsampling model, downsample the image to be detected through at least two downsampling layers to obtain at least two image features to be combined in different sizes; combine the at least two image features to be combined through the feature combination layer to obtain a combined image feature.

[0030] Among them, the image features to be combined refer to the image features to be combined; each downsampling layer has a corresponding image feature to be combined, and the feature map size of the image features to be combined corresponding to each downsampling layer is different. The combined image features refer to the image features obtained by combining the image features to be combined corresponding to each downsampling layer; it includes image features of various sizes corresponding to the image to be detected.

[0031] Exemplarily, in order to obtain feature maps of various sizes of the image to be detected so as to improve the detection accuracy of the subsequent target defect detection model on the detailed feature information in the image to be detected, the image downsampling model may include at least two downsampling layers constructed based on the self-attention mechanism, and each downsampling layer is connected in series, that is, the output of the current downsampling layer is used as the input of the next downsampling layer, and the image to be detected is input into the image downsampling model for downsampling processing to obtain the image features to be combined corresponding to each downsampling layer, and the image features to be combined corresponding to each downsampling layer are combined to obtain the combined image features, so as to provide data for the subsequent target defect detection model to perform more detailed defect detection.

[0032] S204: Input the combined image features into the target defect detection model for processing, and output the defect detection result corresponding to the image to be detected.

[0033] Among them, the defect detection result refers to the result obtained by detecting the defects in the image to be detected, including the category of each defect predicted by the feature analysis of images of different sizes and the corresponding position information of the defects in the image to be detected.

[0034] For example, the combined image features output by the trained image downsampling model retain more detailed feature information of the image to be detected, and the combined image features can be input into the target defect detection model for processing, and the defect detection result corresponding to the image to be detected can be output. In order to reduce the unknown random noise in the defect detection result, the defect detection result can be denoised, wherein the denoising operation can use the Monte Carlo method to obtain the denoised target defect detection result.

[0035] It can be seen that in the embodiment of the present application, the acquired image to be detected is input into a trained image downsampling model including at least two downsampling layers and a feature combination layer constructed based on the self-attention mechanism, the image to be detected is downsampled by at least two downsampling layers to obtain at least two image features of different sizes to be combined, and then at least two image features to be combined are combined by the feature combination layer to obtain a combined image feature, which is conducive to analyzing the attention information of the image to be detected in a global range according to the downsampling layer constructed based on the self-attention mechanism, and realizing the reasoning of multi-size features of the image to be detected according to the downsampling layer constructed based on the self-attention mechanism, adding additional features of the image to be detected that are higher and lower than the normal value, and better retaining more detail feature information in the image to be detected; finally, the combined image feature is input into the target defect detection model for processing to obtain the defect detection result corresponding to the image to be detected, so that on the basis of the image downsampling model processing the image to be detected to retain more detail feature information, a deep defect detection analysis is further performed based on the target defect detection model, which is conducive to the target defect detection model to more accurately detect and analyze the defects of the image to be detected, thereby better improving the accuracy of detecting image defects.

[0036] In some embodiments, each downsampling layer is connected in series; in S202, the image to be detected is downsampled by at least two downsampling layers to obtain at least two image features of different sizes to be combined, including:

[0037] S300, performing downsampling processing on an initial downsampling layer among at least two downsampling layers to obtain image features to be combined corresponding to the initial downsampling layer, and using the initial downsampling layer as a current downsampling layer.

[0038] S302: Input the image features to be combined corresponding to the current downsampling layer into the subsequent downsampling layer corresponding to the current downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the subsequent downsampling layer.

[0039] S304, taking the next down-sampling layer as the current down-sampling layer, cyclically executing the step of inputting the image features to be combined corresponding to the current down-sampling layer into the next down-sampling layer corresponding to the current down-sampling layer for down-sampling processing, and obtaining the image features to be combined corresponding to the next down-sampling layer; taking the next down-sampling layer as the current down-sampling layer, until obtaining the image features to be combined corresponding to each down-sampling layer.

[0040] The initial downsampling layer refers to the downsampling layer at the first position of at least two downsampling layers, and the subsequent downsampling layer refers to the downsampling layer next to the current downsampling layer.

[0041] Exemplarily, in order to avoid a large amount of loss of image detail feature information caused by directly reducing the image resolution and to retain the detail feature information in the image to be detected as much as possible, the image to be detected can be input into an image downsampling model including at least two downsampling layers constructed based on the self-attention mechanism for downsampling processing, and each downsampling layer is connected in series. The image to be detected can be first input into the initial downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the initial downsampling layer, and the initial downsampling layer is used as the current downsampling layer. The image features to be combined corresponding to the current downsampling layer are input into the corresponding subsequent downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the subsequent downsampling layer, and then the subsequent downsampling layer is used as the current downsampling layer, and so on, until the image features to be combined corresponding to each downsampling layer are obtained. Since the downsampling layers are connected in series, the sizes of the feature maps represented by the image features to be combined corresponding to each downsampling layer are different, that is, the image features to be combined corresponding to each downsampling layer pay attention to different feature information in the image to be detected, which is conducive to better retaining more detail feature information of the image to be detected.

[0042] To more intuitively understand the processing process of the image downsampling model on the image to be detected, we can combine Figure 3 As can be understood from the flowchart shown in (2), each downsampling layer is connected end-to-end, and the image features to be combined output by each downsampling layer are finally used as the input of the feature combination layer, so that the feature combination layer can combine the image features corresponding to the feature maps of different sizes of the image to be detected, thereby providing more abundant feature information for the subsequent target defect detection model to perform defect detection on the image to be detected.

[0043] For the image features to be combined corresponding to the feature maps of different sizes output by each downsampling layer, it can be understood as follows: Assume that the input feature size is C×H×W, that is, H pixels in height, W pixels in width and C channels in total, and the downsampling ratio of the downsampling layer built based on the self-attention mechanism is 2 (that is, the width and height of the output feature map are both half of the input feature map). Each layer of the image downsampling model divides the input features into 2×2 grids in the spatial dimension, totaling After that, the features of the 4 pixels contained in each 2×2 grid are fused and passed to the downsampling layer, and the attention information is calculated globally so that each grid outputs the features of 1 pixel, which are merged into a high Pixels, width The feature map of the pixel, then the next downsampling layer generates a high Pixels, width feature map of a pixel, and so on.

[0044] It can be seen that in this embodiment, the image to be detected is input into an image downsampling model including at least two downsampling layers constructed based on the self-attention mechanism, and each downsampling layer is connected in series to perform downsampling processing, so that each downsampling layer has corresponding image features to be combined. Since the downsampling layers are connected in series, the size of the feature map representing the image features to be combined corresponding to each downsampling layer is different, which is beneficial to avoid directly adjusting the resolution of the image to be detected to obtain images of different sizes and causing the loss of detail feature information, and better retain more detail feature information in the image to be detected, thereby improving the accuracy of image defect detection.

[0045] In some embodiments, at S202, at least two to-be-combined image features are combined through a feature combination layer to obtain a combined image feature, including:

[0046] The image features to be combined corresponding to each downsampling layer are input into the feature combination layer for processing, and the combined image features are output.

[0047] Exemplarily, in order to gather image features of feature maps of different sizes as the input of the target defect detection model, the image features to be combined corresponding to each downsampling layer can be spliced ​​and combined based on the feature combination layer to obtain the combined image features.

[0048] It can be seen that in this embodiment, the image features to be combined corresponding to each downsampling layer are spliced ​​and combined through the feature combination layer to obtain the combined image features, which provides more detailed and rich feature information for the subsequent target defect detection model to perform defect detection on the image to be detected, and is beneficial to improving the accuracy of the target defect detection model in detecting image defects.

[0049] In some embodiments, the target defect detection model includes a feature processing network and a detection head processing network, the feature processing network includes a feature extractor and a feature fusion device, and the detection head processing network includes a defect classifier and a defect locator; step S204 includes:

[0050] S400, performing feature extraction processing on the combined image features through a feature extractor to obtain extracted image features; performing feature fusion processing on the extracted image features through a feature fusion device to obtain fused image features.

[0051] S402, classify the fused image features through a defect classifier to obtain defect classification information; and locate the fused image features through a defect locator to obtain defect position information.

[0052] S404: Use the defect classification information and the defect location information as the defect detection result corresponding to the image to be detected.

[0053] Among them, the feature processing network refers to a neural network that performs deep neural network processing on the combined image features. The feature extractor refers to a neural network that performs deep feature extraction on the image features. The feature fusion device refers to a neural network that performs fusion processing on the image features obtained by deep extraction; fusion includes model algorithm logic processing. The detection head processing network refers to a neural network that analyzes the defects in the image to be detected based on the feature processing information. The defect classifier refers to the defect classification detection head, which is a neural network used to distinguish the defect categories. The defect locator refers to the defect positioning detection head, which is a neural network used to analyze the location of the defect in the image to be detected.

[0054] Extracted image features refer to image features obtained by performing deep feature extraction on combined image features. Fusion image features refer to image features obtained by performing deep fusion analysis on feature image features obtained by deep extraction. Defect classification information refers to the type information of defects in the image to be detected. Defect location information refers to the location information of defects in the image to be detected.

[0055] Exemplarily, the target defect detection model includes a feature processing network and a detection head processing network. The target defect detection model can be used to detect defects in an image to be detected. In order to improve the accuracy of the target defect detection model in detecting defects, combined image features containing image features of multiple different sizes can be input into the target defect detection model for processing. Based on the feature extractor in the feature processing network in the target defect detection model, further deep feature extraction is performed on the input combined image features to obtain extracted image features after deep feature extraction; then, based on the feature fusion device in the feature processing network, deep feature fusion analysis is performed on the extracted image features to obtain fused image features; finally, based on the defect classifier in the detection head processing network in the target defect detection model, the fused image features are analyzed and processed to obtain defect classification information, and based on The defect locator in the detection head processing network analyzes and processes the fused image features to obtain defect location information, defect classification information and defect location information, that is, the defect detection result of the image to be detected. The defect detection result includes the defect detection result predicted by analyzing the same defect in the image to be detected by image features of different sizes. For example, there is an unknown defect A in the image to be detected, and the combined image features input to the target defect detection model include image features of three images to be detected with sizes a, b, and c. Then, the defect detection result output by the target defect detection model includes defect detection result a1 of defect A analyzed by image features corresponding to size a, defect detection result b1 of defect A analyzed by image features corresponding to size b, and defect detection result c1 of defect A analyzed by image features corresponding to size c.

[0056] It can be seen that in this embodiment, by inputting the combined image features containing image features of various sizes into the target defect detection model for deep neural network processing, the target defect detection model is able to more accurately analyze small defects in the image to be detected on the basis of the combined image features containing more detailed feature information of the image features to be detected, thereby greatly improving the accuracy of the target defect detection model in detecting image defects.

[0057] In some embodiments, the defect detection method further includes:

[0058] S500, obtaining initial parameters of a detection head processing network, adjusting the initial parameters into multiple groups of adjustment parameters within a preset adjustment range, and using the initial parameters and the multiple groups of adjustment parameters as a detection head parameter set, wherein the detection head parameter set includes multiple groups of detection head parameters.

[0059] S502, taking each group of detection head parameters in the detection head parameter set as parameters of the detection head processing network, and processing the fused image features based on the detection head processing network corresponding to each group of detection head parameters to obtain defect detection results corresponding to each group of detection head parameters.

[0060] S504, fusing the defect detection results corresponding to each group of detection head parameters to obtain the target defect detection results corresponding to the image to be detected.

[0061] Among them, the initial parameters refer to the parameters of the initial setting of the target defect detection model detection head processing network. The preset adjustment range refers to the range within which the detection head parameters are allowed to be fine-tuned. The detection head parameter set refers to the set of parameters set by the detection head processing network, and each group of detection head parameters includes the parameters of the defect classifier and the parameters of the defect locator. Fusion includes but is not limited to the operation of averaging calculation, and the fusion method of the defect detection results corresponding to each group of detection head parameters can be set according to actual conditions, such as weighted averaging. The target defect detection result refers to the defect detection result after denoising, including the target defect detection results of each defect obtained by analyzing the features of images of different sizes.

[0062] Exemplarily, since low-confidence defect detection results can be considered as high-quality defect detection results with unknown random noise superimposed on them, in order to reduce false alarms caused by low-confidence defect detection results output by the model for boundary samples, the defect detection results can be denoised. Specifically, the initial parameters of the detection head processing network in the target defect detection model can be randomly adjusted within a preset adjustment range. That is, within the preset adjustment range, the initial parameters of the detection head processing network are randomly fine-tuned into multiple groups of adjustment parameters, and the initial parameters and the multiple groups of adjustment parameters are used as a detection head parameter set. The detection head parameter set is a collection of multiple groups of detection head parameters. The parameters of the detection head processing network in the target defect detection model are respectively set to each group of detection head parameters in the detection head parameter set. The fused image features output by the feature processing network are analyzed based on the detection head processing network corresponding to each group of detection head parameters to obtain the defect detection results corresponding to each group of detection head parameters. Then, the defect detection results corresponding to each group of detection head parameters can be averaged, and the average result can be used as the target defect detection result.

[0063] It can be seen that in the present embodiment, by setting multiple groups of adjustment parameters within a preset range to obtain a detection head parameter set, the fused image features output by the feature processing network are analyzed based on the detection head processing networks corresponding to each group of detection head parameters in the detection head parameter set to obtain the defect detection results corresponding to each group of detection head parameters, and then the defect detection results corresponding to each group of detection head parameters are fused to obtain the target defect detection results, the parameters of the detection head processing network are fine-tuned and perturbed, and the defect detection results corresponding to different detection head parameters are fused, which is beneficial to remove random noise in the defect detection results and improve the accuracy of the defect detection results.

[0064] In some embodiments, the image to be inspected is obtained by taking pictures of a camera carried by a drone during inspection. The size of the image taken by the drone inspection is relatively large. Generally, before the image is input into the defect detection model for inspection, the image is pre-processed by directly reducing the image resolution. Such pre-processing of the image is prone to losing many details of the image, so that the defect detection model can only detect defects roughly at a certain tower or in a certain gear when detecting defects of smaller size (such as rust or improper installation of small metal fittings such as bolts and nuts). The processing of the image taken by the drone by the image downsampling model of this application is conducive to better retaining more detailed information of the image, thereby better solving the problem of low accuracy of defect detection of drone-taken images.

[0065] In some embodiments, the Figure 3 , taking the example of using drones to inspect transmission lines for defects such as broken wires, detached insulators, and foreign objects hanging on wires, among which, Figure 3(1) is a schematic diagram of the process of defect detection on the inspection image. Figure 3 (2) is the image downsampling model for the detected image (i.e. Figure 3 The invention relates to a processing process of the inspection image of the transmission line taken by the drone as the inspection image, and an image downsampling model is constructed based on at least two downsampling layers and a feature combination layer constructed based on the self-attention mechanism, and the trained YOLOv5 model is used as the target defect detection model; the inspection image to be detected is input into the image downsampling model, and the attention information of the inspection image to be detected is analyzed globally based on the downsampling layers connected in series in the image downsampling model, and the image features to be combined with different sizes corresponding to the downsampling layers are obtained, and the image features to be combined with the downsampling layers are combined based on the feature combination layer in the image downsampling model to obtain a combined image feature that retains more detailed feature information. Since the size of the feature map represented by the image features to be combined output by each downsampling layer is different, the combined image feature contains feature information of feature maps of different sizes (such as Figure 3 (as shown in (1)).

[0066] Furthermore, the combined image features are input into the target defect detection model, and the combined image features are further subjected to deep feature extraction based on the feature extractor in the target defect detection model. The extracted image features obtained by the deep extraction are input into the feature fusion device for deep analysis and processing, and the fused image features obtained by the fusion analysis are input into the detection head processing network. The defect classifier in the detection head processing network analyzes the defect category (i.e. Figure 3 The defect classifier in (1) in the detection head processing network analyzes the position information of the defect in the image to be detected (i.e. Figure 3 The defect locator in (1) is obtained by analyzing the image features of different sizes. In order to remove the random noise in the defect detection results, the parameters of the detection head processing network in the target defect detection model can be randomly fine-tuned to obtain the defect detection results corresponding to different detection head parameters. Then, the defect detection results corresponding to different detection head parameters are averaged, and the average result is used as the target defect detection result ( Figure 3The target defect detection result in (1) shows that the final defect detection result also includes the defect detection results predicted by the image feature analysis of different sizes, that is, for the same defect in the image to be detected, the defect detection results predicted by the image feature analysis of different sizes, the corresponding target defect detection result includes the defect detection results predicted by the image features of different sizes for the same defect), realizes the denoising operation of the defect detection result, and completes the defect detection of the image to be detected. In the process of defect detection of the image to be detected, firstly, according to the image downsampling constructed by at least two downsampling layers and feature combination layers constructed based on the self-attention mechanism, multi-size feature inference is performed on the image to be detected, which is conducive to retaining more detailed feature information in the image to be detected, adding additional features with sizes that are higher and lower than the normal value, which is conducive to the target defect detection model capturing local features at extremely far viewing distances and capturing global features at extremely close viewing distances, so that the target defect detection model has better detection ability for drone inspection images with abnormal viewing distances. On this basis, the defect detection results are denoised to reduce the false alarms of the target defect detection model, thereby improving the accuracy of defect detection in drone images.

[0067] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0068] Based on the same inventive concept, the embodiment of the present application also provides a defect detection device. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more defect detection device embodiments provided below can refer to the limitations on the defect detection method above, and will not be repeated here.

[0069] like Figure 4 As shown, the embodiment of the present application provides a defect detection device 400, including:

[0070] An acquisition module 402 is used to acquire an image to be detected, a trained image downsampling model, and a trained target defect detection model; the image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism;

[0071] The first processing module 404 is used to input the image to be detected into the image downsampling model, perform downsampling processing on the image to be detected through at least two downsampling layers to obtain at least two image features to be combined in different sizes; and combine the at least two image features to be combined through the feature combination layer to obtain a combined image feature;

[0072] The second processing module 406 is used to input the combined image features into the target defect detection model for processing, and output the defect detection result corresponding to the image to be detected.

[0073] In some embodiments, the downsampling layers are connected in series; in downsampling the image to be detected through at least two downsampling layers to obtain at least two image features to be combined of different sizes, the first processing module 404 is specifically used to:

[0074] Performing downsampling processing on an initial downsampling layer in at least two downsampling layers to obtain image features to be combined corresponding to the initial downsampling layer, and using the initial downsampling layer as the current downsampling layer;

[0075] Input the image features to be combined corresponding to the current downsampling layer into the subsequent downsampling layer corresponding to the current downsampling layer for downsampling processing, so as to obtain the image features to be combined corresponding to the subsequent downsampling layer;

[0076] The subsequent down-sampling layer is used as the current down-sampling layer, and the image features to be combined corresponding to the current down-sampling layer are input into the subsequent down-sampling layer corresponding to the current down-sampling layer for down-sampling processing in a loop to obtain the image features to be combined corresponding to the subsequent down-sampling layer; the subsequent down-sampling layer is used as the current down-sampling layer until the image features to be combined corresponding to each down-sampling layer are obtained.

[0077] In some embodiments, in terms of combining at least two to-be-combined image features through the feature combination layer to obtain a combined image feature, the first processing module 404 is specifically configured to:

[0078] The image features to be combined corresponding to each downsampling layer are input into the feature combination layer for processing, and the combined image features are output.

[0079] In some embodiments, the target defect detection model includes a feature processing network and a detection head processing network, the feature processing network includes a feature extractor and a feature fusion device, and the detection head processing network includes a defect classifier and a defect locator; in terms of inputting the combined image features into the target defect detection model for processing and outputting the defect detection result corresponding to the image to be detected, the second processing module 406 is specifically used to:

[0080] The combined image features are subjected to feature extraction processing by a feature extractor to obtain extracted image features; the extracted image features are subjected to feature fusion processing by a feature fusion device to obtain fused image features;

[0081] The defect classifier is used to classify the fused image features to obtain defect classification information; the defect locator is used to locate the fused image features to obtain defect location information;

[0082] The defect classification information and defect location information are used as the defect detection result corresponding to the image to be detected.

[0083] In some embodiments, the second processing module 406 is further configured to:

[0084] Acquire initial parameters of the detection head processing network, adjust the initial parameters into multiple groups of adjustment parameters within a preset adjustment range, and use the initial parameters and the multiple groups of adjustment parameters as a detection head parameter set, wherein the detection head parameter set includes multiple groups of detection head parameters;

[0085] Each group of detection head parameters in the detection head parameter set is used as the parameters of the detection head processing network. For the detection head processing network corresponding to each group of detection head parameters, the fused image features are processed based on the detection head processing network to obtain the defect detection results corresponding to each group of detection head parameters.

[0086] The defect detection results corresponding to each set of detection head parameters are fused to obtain the target defect detection results corresponding to the image to be detected.

[0087] In some embodiments, the image to be detected is obtained by photographing a camera carried by a drone.

[0088] Each module in the above defect detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0089] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to defect detection. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned defect detection method are implemented.

[0090] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned defect detection method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0091] Those skilled in the art will understand that Figure 5 or Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0092] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0093] In some embodiments, Figure 7 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0094] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0097] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A defect detection method, characterized in that: include: Obtain an image to be detected, a trained image downsampling model, and a trained target defect detection model; The image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism; each of the downsampling layers is connected in series; Inputting the image to be detected into the image downsampling model, performing downsampling processing through an initial downsampling layer among the at least two downsampling layers, obtaining image features to be combined corresponding to the initial downsampling layer, and using the initial downsampling layer as the current downsampling layer; Inputting the image features to be combined corresponding to the current downsampling layer into the subsequent downsampling layer corresponding to the current downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the subsequent downsampling layer; The following steps are used as the current downsampling layer, and the steps of inputting the image features to be combined corresponding to the current downsampling layer into the following downsampling layer corresponding to the current downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the following downsampling layer are executed cyclically; the following steps are used as the current downsampling layer until the image features to be combined corresponding to each downsampling layer are obtained; and the at least two image features to be combined are combined through the feature combination layer to obtain a combined image feature. The combined image features are input into the target defect detection model for processing, and the defect detection result corresponding to the image to be detected is output.

2. The method according to claim 1, characterized in that The step of combining the at least two image features to be combined by the feature combination layer to obtain a combined image feature includes: The image features to be combined corresponding to each downsampling layer are input into the feature combination layer for processing, and the combined image features are output.

3. The method according to claim 1, characterized in that The target defect detection model includes a feature processing network and a detection head processing network, the feature processing network includes a feature extractor and a feature fusion device, and the detection head processing network includes a defect classifier and a defect locator; The step of inputting the combined image features into the target defect detection model for processing and outputting the defect detection result corresponding to the image to be detected includes: The feature extractor performs feature extraction processing on the combined image features to obtain extracted image features; the feature fusion device performs feature fusion processing on the extracted image features to obtain fused image features; The defect classifier performs classification processing on the fused image features to obtain defect classification information; the defect locator performs defect locating processing on the fused image features to obtain defect position information; The defect classification information and the defect location information are used as the defect detection result corresponding to the image to be detected.

4. The method according to claim 3, characterized in that The method further comprises: Acquire initial parameters of the detection head processing network, adjust the initial parameters into multiple groups of adjustment parameters within a preset adjustment range, and use the initial parameters and the multiple groups of adjustment parameters as a detection head parameter set, wherein the detection head parameter set includes multiple groups of detection head parameters; Respectively taking each group of detection head parameters in the detection head parameter set as parameters of the detection head processing network, and for the detection head processing network corresponding to each group of the detection head parameters, processing the fused image features based on the detection head processing network to obtain defect detection results corresponding to each group of the detection head parameters; The defect detection results corresponding to each group of the detection head parameters are fused to obtain the target defect detection results corresponding to the image to be detected.

5. The method according to claim 1, characterized in that The image to be detected is obtained by photographing a camera carried by a drone.

6. A defect detection device, characterized in that: include: An acquisition module is used to acquire an image to be detected, a trained image downsampling model, and a trained target defect detection model; The image downsampling model includes at least two downsampling layers and a feature combination layer constructed based on a self-attention mechanism; each of the downsampling layers is connected in series; A first processing module, configured to input the image to be detected into the image downsampling model, perform downsampling processing through an initial downsampling layer among the at least two downsampling layers, obtain image features to be combined corresponding to the initial downsampling layer, and use the initial downsampling layer as a current downsampling layer; Inputting the image features to be combined corresponding to the current downsampling layer into the subsequent downsampling layer corresponding to the current downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the subsequent downsampling layer; The following steps are used as the current downsampling layer, and the steps of inputting the image features to be combined corresponding to the current downsampling layer into the following downsampling layer corresponding to the current downsampling layer for downsampling processing to obtain the image features to be combined corresponding to the following downsampling layer are executed cyclically; the following steps are used as the current downsampling layer until the image features to be combined corresponding to each downsampling layer are obtained; and the at least two image features to be combined are combined through the feature combination layer to obtain a combined image feature. The second processing module is used to input the combined image features into the target defect detection model for processing, and output the defect detection result corresponding to the image to be detected.

7. The device according to claim 6, characterized in that The target defect detection model includes a feature processing network and a detection head processing network, the feature processing network includes a feature extractor and a feature fusion device, and the detection head processing network includes a defect classifier and a defect locator; the second processing module is also used to: perform feature extraction processing on the combined image features through the feature extractor to obtain extracted image features; perform feature fusion processing on the extracted image features through the feature fusion device to obtain fused image features; perform classification processing on the fused image features through the defect classifier to obtain defect classification information; perform defect location processing on the fused image features through the defect locator to obtain defect location information; and use the defect classification information and the defect location information as the defect detection result corresponding to the image to be detected.

8. The device according to claim 7, characterized in that The second processing module is also used for: Acquire initial parameters of the detection head processing network, adjust the initial parameters into multiple groups of adjustment parameters within a preset adjustment range, and use the initial parameters and the multiple groups of adjustment parameters as a detection head parameter set, wherein the detection head parameter set includes multiple groups of detection head parameters; Respectively taking each group of detection head parameters in the detection head parameter set as parameters of the detection head processing network, and for the detection head processing network corresponding to each group of the detection head parameters, processing the fused image features based on the detection head processing network to obtain defect detection results corresponding to each group of the detection head parameters; The defect detection results corresponding to each group of the detection head parameters are fused to obtain the target defect detection results corresponding to the image to be detected.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Product surface defect detection method and system based on positive sample reference

    CN111768388A

  • Image semantic segmentation model training method, image semantic segmentation method and related device

    CN113361536A

  • PCB surface defect detection method and device, computer equipment and storage medium

    CN117557538A

  • Road sign detection method and device and computer equipment

    CN117893990A

  • Printer shell defect detection method and device based on deep convolutional neural network

    CN118351118A