Defect identification method and device and electronic equipment

Abstract

The invention provides a defect identification method and device and electronic equipment. The method comprises the following steps: acquiring an apparent image of a to-be-identified component; inputting the apparent image into a defect recognition model to obtain a defect recognition result of the to-be-recognized part output by the defect recognition model and the confidence of the defect recognition result; the defect identification model comprises a backbone network layer and a full connection layer, the backbone network layer is used for acquiring image features of the apparent image, and the full connection layer is used for determining a defect identification result and confidence based on the image features; training samples needing to be newly added are selected based on the confidence coefficient, the defect recognition model is retrained through the newly added training samples, and retraining of the defect recognition model at least comprises optimization of model parameters of the full connection layer. According to the application form image, the newly added training sample can be fused into optimization of the defect identification model in real time, and rapid iteration of the model can be realized only by updating the full connection layer, so that the defect identification accuracy and efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a defect identification method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of artificial intelligence, it has become an industry mainstream trend to use unmanned aerial vehicles to carry intelligent image recognition technology for automatic inspection. However, in the related art, the parameters of the defect identification model are fixed after deployment, and the model cannot adapt to the changing scene of the inspection site, resulting in a decrease in recognition accuracy. When the model needs to be optimized, all samples must be transmitted back to the server for complete retraining and redeployment, which is a long and inefficient process.

[0003] Therefore, how to improve the accuracy and efficiency of defect identification has become a technical problem to be solved in the industry. SUMMARY

[0004] The present application provides a defect identification method, device and electronic equipment to solve the technical problem of how to improve the accuracy and efficiency of defect identification in the prior art.

[0005] In a first aspect, the present application provides a defect identification method, comprising: obtaining an appearance image of a component to be identified; inputting the appearance image into a defect identification model to obtain a defect identification result of the component to be identified and a confidence of the defect identification result output by the defect identification model; the defect identification model comprises a backbone network layer and a fully connected layer, the backbone network layer is used to obtain image features of the appearance image, and the fully connected layer is used to determine the defect identification result and the confidence based on the image features; based on the confidence, selecting training samples that need to be added to retrain the defect identification model using the added training samples, wherein the retraining of the defect identification model at least includes optimizing the model parameters of the fully connected layer.

[0006] In some embodiments, the appearance image of the component to be identified is obtained by: generating an initial inspection route of the unmanned aerial vehicle based on a digital virtual model of an equipment to be identified, wherein the equipment to be identified includes at least one component to be identified; predicting a resolution of a current appearance image to be collected during flight of the unmanned aerial vehicle based on the initial inspection route; if the resolution is greater than or equal to a preset resolution, controlling the unmanned aerial vehicle to collect the appearance image; If the resolution is less than the preset resolution, adjust the drone parameters of the drone until the predicted resolution of the current appearance image to be acquired is greater than or equal to the preset resolution, and control the drone to acquire the appearance image.

[0007] In some embodiments, predicting the resolution of the current appearance image to be acquired includes: The resolution is predicted based on the component size of the component to be identified, the distance between the drone and the component to be identified, the focal length of the drone's lens, and the pixel size of the image acquisition sensor in the drone.

[0008] In some embodiments, inputting the appearance image into the defect recognition model includes: The appearance image is processed, and the processed appearance image is input into the defect recognition model; The processing includes at least one of frequency domain enhancement, background suppression, and grayscale adaptive processing.

[0009] In some embodiments, the frequency domain enhancement includes decomposing the appearance image to obtain multiple sub-bands, and applying nonlinear gain to the mid-frequency sub-bands that include edge information in the multiple sub-bands; The background suppression includes segmenting the component region of the component to be identified in the appearance image and generating a component mask map; The grayscale adaptation includes aligning the image grayscale distribution under different lighting conditions in the apparent image.

[0010] In some embodiments, the defect identification model includes a localization sub-model and an identification sub-model. The localization sub-model is used to locate the component region of the component to be identified in the appearance image, and the identification sub-model is used to extract the image features of the component region and determine the defect identification result and the confidence level. The localization sub-model and the identification sub-model are constructed based on the following steps: The localization sub-model is constructed by embedding a collaborative attention module in the neck region of the YOLOX network. The recognition sub-model is constructed by adding a feature pyramid structure to the residual blocks of ResNet-50.

[0011] In some embodiments, the defect identification result includes the presence of a defect and the absence of a defect, and the determination of new training samples based on the confidence level includes: If the confidence level of the existing defect is less than the preset confidence level, adjust the drone parameters of the drone and re-acquire defect images at the defect locations where the confidence level is less than the preset confidence level; The defect image is input into the defect recognition model to obtain a first defect recognition result and a first confidence level of the first defect recognition result output by the defect recognition model. The first defect recognition result is the defect recognition result of the defect image. If the first defect identification result indicates the presence of a defect and the first confidence level is greater than or equal to the preset confidence level, the training sample is determined based on the defect image, the first defect identification result, and the first confidence level.

[0012] In some embodiments, optimizing the model parameters of the fully connected layer based on the training samples includes: Perform data augmentation processing on the training samples; The model parameters are updated based on the processed training samples and the exponentially decaying learning rate.

[0013] Secondly, this application provides a defect identification device, comprising: The acquisition module is used to acquire the appearance image of the part to be identified; The identification module is used to input the appearance image into the defect identification model to obtain the defect identification result of the part to be identified and the confidence level of the defect identification result output by the defect identification model; the defect identification model includes a backbone network layer and a fully connected layer, the backbone network layer is used to acquire the image features of the appearance image, and the fully connected layer is used to determine the defect identification result and the confidence level based on the image features; An optimization module is used to select new training samples based on the confidence level, so as to retrain the defect identification model using the new training samples. The retraining of the defect identification model includes at least optimizing the model parameters of the fully connected layer.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the above-described method when executing the program through the computer program.

[0015] The defect identification method, apparatus, and electronic device provided in this application use a drone to collect an appearance image of the component to be identified with a resolution greater than or equal to a preset resolution. The appearance image is then input into a defect identification model. Based on the defect identification result and confidence level of the component to be identified output by the defect identification model, new training samples are determined. The model parameters of the fully connected layer of the defect identification model are updated based on the new training samples. The new training samples can be integrated into the model optimization in real time, enabling rapid iteration of the model and greatly improving the accuracy and efficiency of defect identification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts illustrating the defect identification method provided in the embodiments of this application.

[0018] Figure 2 This is a second schematic flowchart of the defect identification method provided in the embodiments of this application.

[0019] Figure 3 This is the third flowchart illustrating the defect identification method provided in the embodiments of this application.

[0020] Figure 4 This is the fourth flowchart illustrating the defect identification method provided in the embodiments of this application.

[0021] Figure 5 This is a schematic diagram of the defect identification device provided in an embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0025] The defect identification method provided in this application is applicable to terminals, which can be various electronic devices with displays and support web browsing, including servers, smartphones, tablets, laptops, and desktop computers.

[0026] Figure 1 This is one of the flowcharts illustrating the defect identification method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes steps 110, 120, and 130. These method steps are merely one possible implementation of this application.

[0027] Step 110: Obtain the appearance image of the component to be identified.

[0028] Specifically, the defect identification method provided in this application is executed by a defect identification device, which can be a hardware device independently set in the terminal or a software program running in the terminal.

[0029] An appearance image is an image that contains the surface appearance of a component to be identified.

[0030] Appearance images can be captured using drones. To ensure image quality, a preset resolution can be set, allowing only appearance images with a resolution greater than or equal to the preset resolution to be captured. Step 120: Input the appearance image into the defect recognition model to obtain the defect recognition result and confidence level of the component to be identified output by the defect recognition model. The defect recognition model includes a backbone network layer and a fully connected layer. The backbone network layer is used to obtain the image features of the appearance image, and the fully connected layer is used to determine the defect recognition result and confidence level based on the image features.

[0031] Specifically, the defect identification result refers to the judgment output by the defect identification model after analyzing the appearance image. For example, the defect identification result may include whether a defect exists or not. The defect identification result can be displayed in the form of an image. For example, the defect identification model outputs an image that includes the part to be identified, with the defect of the part to be identified circled in red, and the defect location information and confidence level written next to it.

[0032] Confidence level refers to the probability value output by the defect identification model, which quantifies the degree of confidence the defect identification model has in its identification results.

[0033] Image features refer to the numerical information extracted from the appearance of an image by the backbone network layers that can represent the image content, such as texture, contour, and color.

[0034] The defect identification model consists of a backbone network layer and a fully connected layer. The backbone network layer extracts image features from the appearance image and inputs these features into the fully connected layer. The fully connected layer analyzes the image features to determine whether a defect exists in the part to be identified and outputs the defect identification result (whether a defect exists or not) along with the confidence level of the defect identification result.

[0035] Step 130: Select new training samples based on confidence level, and retrain the defect identification model using the new training samples. Retraining the defect identification model includes at least optimizing the model parameters of the fully connected layers.

[0036] Specifically, model parameters can include weights and biases in the network.

[0037] To ensure the reliability of defect identification results and avoid false judgments due to low confidence caused by insufficient image quality in a single shot, a confidence threshold can be preset. If the confidence of the defect identification result is greater than or equal to the preset confidence, the defect identification result is considered reliable. If the confidence of the defect identification result is less than the preset confidence, the UAV parameters are adjusted, such as adjusting the UAV position, UAV lens focal length and / or flight attitude angle, and then the appearance image of the part to be identified is re-acquired, or a defect image is acquired for the identified defect location. The appearance image or defect image is then input into the defect identification model.

[0038] If the defect identification model outputs the same defect identification result as the previous defect identification result, but the confidence level is greater than or equal to the preset confidence level, then this set of images, defect identification results, and confidence levels are added to the training set as new training samples. The weights of the fully connected layers of the defect identification model are updated online to achieve incremental learning of the defect identification model in a single inspection.

[0039] A lightweight incremental learning framework can be deployed on the terminal to freeze the backbone network of the defect recognition model and update the model parameters of the fully connected layer according to the newly added training samples to achieve real-time optimization of the defect recognition model, so that the defect recognition model can accurately and efficiently identify the defects of the parts to be identified.

[0040] After being pre-trained on large-scale data, the backbone network layer of the defect classification model already possesses stable feature extraction capabilities. Freezing its weights can prevent the feature extraction capability from degrading due to the small number of newly added training samples in a single inspection. Updating only the fully connected layers can significantly reduce the computational load, adapting to the limited computing power and storage resources of the drone terminal. This design can both retain the existing knowledge of the defect classification model and quickly absorb information from newly added training samples, enabling on-the-fly optimization and online model updates.

[0041] The defect identification method provided in this application collects appearance images of the component to be identified by a drone, inputs the appearance images into a defect identification model, determines new training samples based on the defect identification results and confidence levels of the component to be identified output by the defect identification model, and updates the model parameters of the fully connected layer of the defect identification model based on the new training samples. The new training samples can be integrated into the model optimization in real time, realizing rapid iteration of the model and greatly improving the accuracy and efficiency of defect identification.

[0042] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0043] In some embodiments, step 110 includes: The initial inspection route of the UAV is generated based on the digital virtual model of the device to be identified, which includes at least one component to be identified. During the flight of the UAV based on the initial inspection route, predict the resolution of the current appearance image to be collected; When the resolution is greater than or equal to the preset resolution, control the drone to acquire apparent images; If the resolution is less than the preset resolution, adjust the drone's parameters until the predicted resolution of the current appearance image to be acquired is greater than or equal to the preset resolution, and control the drone to acquire the appearance image.

[0044] The prediction of the resolution of the current appearance image to be acquired includes: The resolution is predicted based on the component size to be identified, the distance between the drone and the component, the focal length of the drone's lens, and the pixel size of the image acquisition sensor in the drone.

[0045] Specifically, a digital virtual model is a three-dimensional model created in a computer that completely corresponds to the physical device to be identified in terms of geometric dimensions, structure, and spatial location.

[0046] The drone will be equipped with a zoom lens and a laser rangefinder module, and will generate an initial inspection route based on the digital virtual model of the device to be identified, to ensure coverage of all components to be identified. The device to be identified can be a tower, and the components can be bolts, angle irons, insulators, etc. Defects on the components to be identified can be rust, cracks, loosening, etc.

[0047] As the drone flies along the initial inspection route, the defect identification device calculates the resolution S, or line pixel count, of the current appearance image to be collected in real time according to the following formula.

[0048] ; in, The actual component size (in meters) of the component to be identified. The distance (in meters) between the drone and the component to be identified. This refers to the focal length (in millimeters) of the drone's lens. The pixel size (in micrometers) of the image acquisition sensor in the drone. It is a multiplication sign.

[0049] The resolution of an apparent image refers to the number of pixels occupied by the part to be identified along the key direction in the current image. The key direction may differ for different parts to be identified. For example, the key direction of angle steel is the length direction because cracks usually extend along the length direction. It reflects the physical basis of image clarity and defect distinguishability.

[0050] like If the resolution is lower than the preset resolution, the drone parameters will be automatically adjusted, such as flight position (approaching or moving away from the object to be identified), gimbal pitch angle, and / or optical zoom, until the actual pixel ratio is greater than or equal to the preset resolution. The preset resolution can be set to 50×50 pixels. Only images with a resolution of ≥50×50 pixels can be clearly identified as microcracks. This data-driven imaging control method avoids image blurring caused by excessive distance or angular deviation, providing high-quality data for subsequent identification.

[0051] For example, exposure can be automatically adjusted based on ambient light to ensure a signal-to-noise ratio >35dB.

[0052] It can be approximated along the normal direction of the component to be identified, and the gimbal pitch angle error can be adjusted to ≤5°.

[0053] The formula for calculating the optical zoom factor Z is as follows: ; in, f 0 The initial focal length, S min Calculation results for the preset resolution Indicates the current distance Next, how many times should the lens be zoomed to make the part to be identified reach the preset resolution in the apparent image?

[0054] During the flight of a drone, if the real-time pixel ratio If the resolution is smaller than the preset resolution, you can first select the part closest to the object to be identified, i.e., reduce the resolution. If position adjustment is limited, zoom is used to magnify the part to be identified, and a theoretically optimal value is calculated using the above formula. Compare again Maximum safe zoom magnification of drone cameras To prevent motion blur and image quality degradation, ≤8 times, the actual implementation is This allows the drone to maintain a safe distance from the part to be identified while also ensuring that the imaging resolution meets the minimum requirements for defect identification. This zoom control, which balances safety and clarity, ensures that defects can still be identified even when the distance cannot be adjusted.

[0055] When it is determined that the resolution of the current appearance image to be acquired can be greater than or equal to the preset resolution, the drone is controlled to acquire the appearance image.

[0056] The defect identification method provided in this application determines the resolution of the appearance image to be acquired by considering the actual size of the component to be identified, the distance of the UAV, the focal length, and the sensor pixel size. It adjusts the UAV parameters in real time, optimizes the flight attitude to keep the pitch angle error of the UAV gimbal within 5°, and reduces the viewing angle deviation by approaching the UAV along the normal direction of the component to be identified. This achieves dynamic adjustment and adaptive imaging, avoiding invalid acquisition due to image blurring and ensuring the effectiveness of the acquired appearance image from the source.

[0057] In some embodiments, step 120 includes: The appearance image is processed, and the processed appearance image is input into the defect recognition model; The processing includes at least one of frequency domain enhancement, background suppression, and grayscale adaptive processing.

[0058] Frequency domain enhancement includes decomposing the appearance image to obtain multiple sub-bands, and applying nonlinear gain to the mid-frequency sub-bands that include edge information in the multiple sub-bands; Background suppression includes segmenting the component region of the component to be identified in the appearance image and generating a component mask map; Gray-level adaptation includes dividing the apparent image into multiple grid regions, using a reference image under preset lighting conditions as a benchmark, and adjusting the gray-level histogram of the apparent image grid by grid so that the gray-level histogram matches the histogram of the corresponding grid in the reference image.

[0059] Specifically, to improve the visibility of minute defects, such as microcracks and rust spots, and to suppress interference from complex backgrounds, multimodal preprocessing is performed on the acquired RGB appearance image, including at least one of frequency domain enhancement, background suppression, and grayscale adaptation.

[0060] Frequency domain enhancement includes decomposing the RGB apparent image into different frequency sub-bands through wavelet transform, enhancing the mid-frequency sub-band to highlight microcracks, and suppressing the high-frequency sub-band.

[0061] First, a two-dimensional discrete wavelet transform (DWT) is performed on the appearance image. A single layer of decomposition using the Daubechies8 wavelet basis is applied, yielding four sub-bands: Low-Low (LL): overall brightness and contour information; Low-High (LH) and High-Low (HL), i.e., mid-frequency: main edge information in the horizontal and vertical directions, referring to the contour edge information of the part to be identified; and High-High (HH): details and noise components. To highlight fine texture features such as microcracks, differential processing is used to enhance the LH and HL sub-bands: an adaptive nonlinear gain is applied to the mid-frequency sub-band containing edge information, as shown in the following formula: ; in, for The new value obtained after adaptive nonlinear gain processing Sub-band coefficient, A value between 1.2 and 1.5 represents the gain coefficient. =0.7 is the inhibition index; The maximum amplitude of the subband coefficients is used for normalization. Threshold denoising based on Bayesian estimation is applied to the HH subband to eliminate high-frequency noise. Finally, the image is reconstructed through inverse DWT to obtain an enhanced image with prominent texture details.

[0062] Background suppression involves using a Gaussian Mixture Model (GMM) to segment the part to be identified from the background, such as trees, sky, and power lines, generating a part mask map, and removing or setting the background area to a uniform gray value, thereby eliminating interference areas and reducing false detections and false negatives in subsequent identification.

[0063] Gray-scale adaptation includes aligning the gray-scale distribution of the same part to be identified under different lighting conditions, eliminating false defect responses caused by shadows and brightness differences, and improving recognition stability.

[0064] The apparent image can be divided into several non-overlapping grid regions. Using a reference image under standard lighting conditions as a benchmark, the grayscale histogram of the current image is adjusted grid by grid to match the shape of the histogram of the corresponding grid in the reference image. The standard lighting conditions are preset lighting conditions.

[0065] Inputting high-quality processed appearance images into the defect recognition model can improve the accuracy of recognition.

[0066] Related technologies often suffer from high false negative rates and frequent misjudgments due to unstable image acquisition quality and limited single-model recognition capabilities. This application addresses this issue by performing a three-layer optimization on the appearance images acquired by the UAV during the data preprocessing stage. First, it enhances the image in the frequency domain using a two-dimensional discrete wavelet transform, decomposing it into low-frequency, mid-frequency, and high-frequency sub-bands using the Daubechies8 wavelet basis. Adaptive nonlinear gain enhancement is applied to the mid-frequency sub-band containing edge information to strengthen fine textures such as microcracks, while Bayesian thresholding is used to denoise the high-frequency sub-band. Next, a Gaussian mixture model is used to segment the tower component from the background, generating a component mask to remove interference from trees and sky. Finally, gridded histogram matching aligns the grayscale distribution under different lighting conditions, eliminating false defects caused by shadows and brightness differences. This processing improves image texture clarity and significantly reduces background interference, providing a high-quality data foundation for subsequent recognition.

[0067] The defect identification method provided in this application improves the texture clarity of the appearance image, reduces background interference, and unifies the lighting conditions by using frequency domain enhancement, background suppression, and grayscale adaptation, making small defects and component outlines more prominent and consistent at the pixel level.

[0068] In some embodiments, the defect identification model includes a localization sub-model and an identification sub-model. The localization sub-model is used to locate the component region of the component to be identified in the appearance image, and the identification sub-model is used to extract image features of the component region and determine the defect identification result and confidence level. The localization sub-model and the identification sub-model are constructed based on the following steps: The YOLOX network is improved by embedding a collaborative attention module in the network neck. A localization sub-model is constructed. A feature pyramid structure is added to the residual blocks of ResNet-50 to construct a recognition sub-model.

[0069] Specifically, the defect identification model includes a localization sub-model and an identification sub-model. The identification sub-model includes a backbone network layer and a fully connected layer. This application's embodiment employs a dual-model collaborative architecture combining an improved YOLOX localization approach and a ResNet-50+ Feature Pyramid Network (FPN) identification approach to construct the defect identification model.

[0070] A lightweight CoordinateAttention (CA) module can be embedded in the neck of the YOLOX network to construct a localization sub-model. The CA module strengthens the model's focus on key features and spatial location weights of small-sized parts through a parallel structure of channel attention and spatial attention, thereby improving the localization accuracy of small-sized parts to be identified.

[0071] Channel attention branch: Aggregate global spatial information through Global Average Pooling (GAP) and Global Max Pooling (GMP), and generate channel attention weights via a shared Multi-Layer Perceptron (MLP). , and the relevant formulas are as follows: ; where is the Sigmoid activation function, is the input feature map, and are three-dimensional tensors, C is the number of channels (pool), H is the height of the feature map, and W is the width of the feature map.

[0072] Spatial attention branch: Concatenate the features after channel pooling, such as average pooling and max pooling, in the channel dimension, and generate spatial attention weights through a 7x7 convolutional layer : .

[0073] Final output is , where is element-wise multiplication.

[0074] Figure 2 This is the second schematic diagram of the process of the defect recognition method provided by the embodiment of the present application. As Figure 2 shown, the specific process of YOLOX component localization training is as follows: Input: Training dataset D, network model YOLOX + CA module, maximum number of training epochs E; Output: Trained model parameters .

[0075] The overall process includes: Initializing model parameters ← Random or pre-trained weights; Initializing the training state: Training epoch (epoch) ← 0; Looping (while epoch < E do) when the training epoch is less than E; Traversing each batch B (batchB) in the training dataset D; Forward propagation: Input batchB → Output predicted bounding boxes + defect probabilities; The CA module calculates channel attention weights, calculates spatial attention weights, and outputs the feature map; Calculating the loss function L: Object detection loss = Classification loss + Bounding box regression loss + Confidence loss, and the loss uses the recognition loss (FocalLoss) (unbalanced small widget samples), , p t is the predicted probability; Backward propagation: Calculating the gradient Update model parameters Set the learning rate; record the training state; end the loop (endfor); epoch ← epoch + 1; endwhile; return the trained model parameters. .

[0076] The recognition sub-model uses ResNet-50 as the backbone network and incorporates an FPN structure.

[0077] After the localization sub-model locates the component region to be identified, the component region in the appearance image can be cropped. The cropped component region is then input into the ResNet-50+FPN sub-model, i.e., the recognition sub-model. The recognition sub-model fuses shallow high-resolution features with deep semantic features to accurately identify defects. The two models work together to cover the entire process from component identification to defect finding.

[0078] The localized component region can be cropped at the subpixel level and input into the recognition sub-model with ResNet-50 as the backbone. The recognition sub-model adds FPN after the third residual block, which integrates shallow high-resolution features and deep semantic features to output defect recognition results and confidence scores.

[0079] For example, FPN is inserted between the conv3_x and conv4_x layers of ResNet-50: the outputs of convolution (conv)2_x (high resolution, low semantics), conv3_x (medium resolution, medium semantics), and conv4_x (low resolution, high semantics) are each passed through a 1×1 convolution to unify the number of channels, then upsampled by a factor of 2 from top to bottom and summed level by level to generate a multi-scale fused feature map; the final defect recognition loss function FocalLoss is used. ;in α t =0.8, =2, p t To predict probabilities and address the imbalance problem of small-sized defect samples.

[0080] Figure 3 This is the third flowchart illustrating the defect identification method provided in the embodiments of this application, as shown below. Figure 3 As shown, the training process for ResNet-50-FPN defect recognition is as follows: Input: Training set Dpart of the located component region, model ResNet-50+FPN, maximum number of training epochs E; Output: Parameters of the trained defect recognition model. .

[0081] The overall process includes: initializing model parameters ← Random or pre-trained weights; Initialize the training state: epoch ← 0; while epoch < E do; Iterate through each batch B in the training dataset Dpart; Forward propagation: Input batch B → Output feature maps of conv2_x, conv3_x, and conv4_x; FPN: Pass the outputs of conv2_x, conv3_x, and conv4_x through 1×1 convolutions to unify the number of channels, perform 2x upsampling自上而下进行2倍上采样并逐级相加生成多尺度融合特征图 from top to bottom and add them逐级相加生成多尺度融合特征图 level by level to generate multi-scale fused feature maps ; Recognition prediction: Input → Output the probability of defect recognition result ; Calculate Focal Loss, ; Backward propagation: Calculate the gradient ; Update the model parameters ← − ; Record the training state; endfor; epoch ← epoch + 1; endwhile; Return the parameters of the trained recognition sub-model .

[0082] The defect recognition method provided by the embodiments of this application, after performing multi-modal processing on the appearance images, will input these high-quality appearance images into the localization sub-model. The improved YOLOX sub-localization model can more accurately identify and frame the positions of small-sized parts to be recognized, and at the same time provide a component area with sufficient texture details and balanced illumination for the subsequent ResNet-50+FPN recognition sub-model, thereby overall improving the detection accuracy and the recognition rate of small defects in the cooperation of the two models.

[0083] The embodiments of this application adopt a dual-model cooperation mechanism. By improving the YOLOX network and embedding a lightweight cooperative attention module in the neck, the spatial weights of small-sized parts to be recognized are strengthened through channel attention and spatial attention, and the localization accuracy is improved by 25% compared with the traditional YOLOX; After the located component area is sub-pixel cropped, it is input into the classification model that combines ResNet-50 and the feature pyramid structure. This model unifies the number of channels of the shallow features with high resolution and low semantics, the middle features with medium resolution and medium semantics, and the deep features with low resolution and high semantics, and then performs upsampling and fusion level by level, and the recognition sensitivity to small defects such as micro-cracks and rust is improved.

[0084] In some embodiments, the defect recognition result includes the presence of a defect and the absence of a defect, and step 130 includes: In the case where the confidence level of the presence of a defect is less than the preset confidence level, adjust the drone parameters of the drone to re-acquire the defect image of the defect position where the confidence level is less than the preset confidence level; The defect image is input into the defect recognition model to obtain the first defect recognition result and the first confidence level of the first defect recognition result output by the defect recognition model. The first defect recognition result is the defect recognition result of the defect image. If the first defect identification result indicates the presence of a defect and the first confidence level is greater than or equal to the preset confidence level, training samples are determined based on the defect image, the first defect identification result, and the first confidence level.

[0085] Step 130 also includes: Perform data augmentation on the training samples; The model parameters are updated based on the processed training samples and the exponentially decaying learning rate.

[0086] Specifically, Figure 4 This is the fourth flowchart illustrating the defect identification method provided in the embodiments of this application. Figure 4 As shown, to ensure the reliability of defect identification results and avoid low-confidence misjudgments due to insufficient single-image quality, this application introduces a confidence-driven dynamic re-image mechanism, including the following: Confidence threshold determination. Setting a preset confidence level. =0.7, which can be adjusted according to the actual scenario. (Confidence level) If the defect identification result is valid, no retake is required; otherwise... If this happens, a reshoot will be triggered.

[0087] Re-enhancing parameters. For defect areas with low confidence, the image acquisition parameters of the first appearance image were reviewed, including the distance between the drone and the component. ,focal length By combining flight attitude angles and other factors with the location information of defects in the appearance images, we analyze the possible causes of low confidence and optimize the reshoot parameters based on the following strategies.

[0088] If the issue is determined to be angular deviation, such as a deviation of more than 5° from the component's normal direction, adjust the drone's pitch and yaw angles to realign the shooting direction with the component's normal, keeping the error within ≤3°; if the issue is determined to be due to excessive distance, adjust the line pixel count. calculate, If the resolution is less than 80×80 pixels, then the priority is to reduce the distance between the drone and the component to be identified. If distance adjustment is limited, zoom according to the zoom formula; if it is determined to be light interference, such as local reflection causing the signal-to-noise ratio to be ≤35dB, adjust the exposure parameters and turn on the fill light on the drone to improve the signal-to-noise ratio of the reshot area to >40dB.

[0089] Targeted re-image capture is performed. Based on the optimized UAV parameters, the UAV only re-captures images of the defect locations where low-confidence defects are found, eliminating the need to re-inspect the entire component to be identified, thus reducing redundant flight and imaging costs. The defect images directly enter the multimodal preprocessing flow, generating optimized defect images which are then input again into the defect recognition model.

[0090] Verify the results of the re-enhancing scan. If the re-enhancing scan still confirms the presence of a defect in the first defect identification result, and the first confidence level is [not specified]... If the defective image is marked as a high-confidence sample, a new sample pool is added, and the corresponding first defect identification result and first confidence level are used to construct new training samples; if the confidence level remains high after three consecutive retakes... If a suspected defect is found in the area, it is marked as a "manual review item" and its location information is recorded in conjunction with the coordinates of the digital virtual model of the device to be identified, for subsequent manual inspection and confirmation.

[0091] Real-time model optimization. New training samples are added to the training set, and the weights of the fully connected layers in the recognition sub-model are updated online, enabling incremental learning of the model in a single inspection. A lightweight incremental learning framework can be deployed on the inspection terminal, freezing the backbone network weights of the recognition sub-model and updating only the fully connected layers. X new After data augmentation by the data augmentation module, such as random rotation of ±5° and brightness adjustment of ±10%, the parameter update module updates the data using an exponentially decaying learning rate. The relevant formula is as follows: ; Among them, the initial =0.001, attenuation coefficient =0.1, To update the steps, y is X new The corresponding true label; calculate the validation set accuracy after each update, and stop updating if it decreases for 3 consecutive times to prevent overfitting.

[0092] The defect identification method provided in this application utilizes a drone to collect surface images according to specific parameters, performs multimodal preprocessing on the surface images, identifies defects through dual-model collaboration, and outputs confidence scores. If the confidence score is low, a re-image is triggered. Parameter optimization, directional re-image capture, and result verification ensure reliable identification. For valid identification results, a lightweight incremental learning framework is deployed on the inspection terminal, the backbone network is frozen, and after augmentation of newly added training samples, the parameters of the fully connected layers are updated with an exponentially decaying learning rate to achieve real-time model optimization. Finally, through the collaboration of each link, defects in the parts to be identified are accurately and efficiently identified, and the identification capability is dynamically optimized. This reduces resource consumption while ensuring accuracy and controls ineffective costs at the source of data collection.

[0093] The defect identification device provided in the embodiments of this application is described below. The defect identification device described below and the defect identification method described above can be referred to each other.

[0094] Figure 5 This is a schematic diagram of the defect identification device provided in the embodiments of this application, as shown below. Figure 5 As shown, the device includes a data acquisition module 510, an identification module 520, and an optimization module 530.

[0095] The acquisition module is used to acquire the appearance image of the part to be identified; The recognition module is used to input the appearance image into the defect recognition model and obtain the defect recognition result and confidence level of the defect recognition result of the part to be identified. The defect recognition model includes a backbone network layer and a fully connected layer. The backbone network layer is used to acquire the image features of the appearance image, and the fully connected layer is used to determine the defect recognition result and confidence level based on the image features. The optimization module is used to select new training samples based on confidence level, so as to retrain the defect recognition model using the new training samples. The retraining of the defect recognition model includes at least optimizing the model parameters of the fully connected layers.

[0096] Specifically, according to the embodiments of this application, any multiple modules among the acquisition module, identification module, and optimization module can be combined into one module, or any one of the modules can be split into multiple modules.

[0097] Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in a single module.

[0098] According to embodiments of this application, at least one of the acquisition module, identification module, and optimization module can be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System-on-a-Chip, System-on-a-Substrate, System-on-Package, Application Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuit, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods.

[0099] Alternatively, at least one of the acquisition module, identification module, and optimization module can be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0100] In some embodiments, the acquisition module is specifically used for: The initial inspection route of the UAV is generated based on the digital virtual model of the device to be identified, which includes at least one component to be identified. During the flight of the UAV based on the initial inspection route, predict the resolution of the current appearance image to be collected; When the resolution is greater than or equal to the preset resolution, control the drone to acquire apparent images; If the resolution is less than the preset resolution, adjust the drone's parameters until the predicted resolution of the current appearance image to be acquired is greater than or equal to the preset resolution, and control the drone to acquire the appearance image.

[0101] In some embodiments, predicting the resolution of the current appearance image to be acquired includes: The resolution is predicted based on the component size to be identified, the distance between the drone and the component, the focal length of the drone's lens, and the pixel size of the image acquisition sensor in the drone.

[0102] In some embodiments, the identification module is specifically used for: The appearance image is processed, and the processed appearance image is input into the defect recognition model; The processing includes at least one of frequency domain enhancement, background suppression, and grayscale adaptive processing.

[0103] In some embodiments, frequency domain enhancement includes decomposing the appearance image to obtain multiple sub-bands, and applying nonlinear gain to the mid-frequency sub-bands that include edge information in the multiple sub-bands. Background suppression includes segmenting the component region of the component to be identified in the appearance image and generating a component mask map; Gray-scale adaptation includes aligning the gray-scale distribution of an image under different lighting conditions in the apparent image.

[0104] In some embodiments, the defect identification model includes a localization sub-model and an identification sub-model. The localization sub-model is used to locate the component region of the component to be identified in the appearance image, and the identification sub-model is used to extract image features of the component region and determine the defect identification result and confidence level. The device also includes a construction module, which is specifically used for: A collaborative attention module is embedded in the neck of the YOLOX network to construct a localization sub-model; A feature pyramid structure is added to the residual blocks of ResNet-50 to construct a recognition sub-model.

[0105] In some embodiments, the defect identification result includes the presence of a defect and the absence of a defect, and the optimization module is specifically used for: If the confidence level of a defect is lower than the preset confidence level, adjust the drone's drone parameters and re-acquire defect images at the defect locations where the confidence level is lower than the preset confidence level. The defect image is input into the defect recognition model to obtain the first defect recognition result and the first confidence level of the first defect recognition result output by the defect recognition model. The first defect recognition result is the defect recognition result of the defect image. If the first defect identification result indicates the presence of a defect and the first confidence level is greater than or equal to the preset confidence level, training samples are determined based on the defect image, the first defect identification result, and the first confidence level.

[0106] In some embodiments, the optimization module is specifically used for: Perform data augmentation on the training samples; The model parameters are updated based on the processed training samples and the exponentially decaying learning rate.

[0107] It should be noted that the defect identification device provided in this application embodiment can implement all the method steps implemented in the above defect identification method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0108] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the computer program in the memory 630 to execute the above-described method.

[0109] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional modules and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the methods provided in the above embodiments.

[0111] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing the processor to execute the methods provided in the above embodiments.

[0112] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0113] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A defect identification method, characterized in that, include: Obtain an appearance image of the component to be identified; The appearance image is input into the defect recognition model to obtain the defect recognition result of the component to be identified and the confidence level of the defect recognition result output by the defect recognition model. The defect identification model includes a backbone network layer and a fully connected layer. The backbone network layer is used to acquire image features of the appearance image, and the fully connected layer is used to determine the defect identification result and the confidence level based on the image features. Based on the confidence level, new training samples are selected to retrain the defect identification model using the new training samples. The retraining of the defect identification model includes at least optimizing the model parameters of the fully connected layer.

2. The defect identification method according to claim 1, characterized in that, The acquisition of the appearance image of the component to be identified includes: An initial inspection route for the UAV is generated based on a digital virtual model of the device to be identified, wherein the device to be identified includes at least one of the components to be identified. During the flight of the UAV based on the initial inspection route, the resolution of the current appearance image to be collected is predicted; When the resolution is greater than or equal to the preset resolution, the drone is controlled to acquire the apparent image; If the resolution is less than the preset resolution, adjust the drone parameters of the drone until the predicted resolution of the current appearance image to be acquired is greater than or equal to the preset resolution, and control the drone to acquire the appearance image.

3. The defect identification method according to claim 2, characterized in that, The prediction of the resolution of the current appearance image to be acquired includes: The resolution is predicted based on the component size of the component to be identified, the distance between the drone and the component to be identified, the focal length of the drone's lens, and the pixel size of the image acquisition sensor in the drone.

4. The defect identification method according to claim 1, characterized in that, The step of inputting the appearance image into the defect recognition model includes: The appearance image is processed, and the processed appearance image is input into the defect recognition model; The processing includes at least one of frequency domain enhancement, background suppression, and grayscale adaptive processing.

5. The defect identification method according to claim 4, characterized in that, The frequency domain enhancement includes decomposing the appearance image to obtain multiple sub-bands, and applying nonlinear gain to the mid-frequency sub-bands that include edge information in the multiple sub-bands. The background suppression includes segmenting the component region of the component to be identified in the appearance image and generating a component mask map; The grayscale adaptation includes dividing the apparent image into multiple grid regions, and using a reference image under preset lighting conditions as a reference, adjusting the grayscale histogram of the apparent image grid by grid so that the grayscale histogram matches the histogram of the corresponding grid in the reference image.

6. The defect identification method according to claim 1, characterized in that, The defect identification model includes a localization sub-model and an identification sub-model. The localization sub-model is used to locate the component region of the component to be identified in the appearance image. The identification sub-model is used to extract the image features of the component region and determine the defect identification result and the confidence level. The localization sub-model and the identification sub-model are constructed based on the following steps: The localization sub-model is constructed by embedding a collaborative attention module in the neck region of the YOLOX network. The recognition sub-model is constructed by adding a feature pyramid structure to the residual blocks of ResNet-50.

7. The defect identification method according to claim 1, characterized in that, The defect identification result includes whether a defect exists or not, and the selection of new training samples based on the confidence level includes: If the confidence level of the defect is lower than the preset confidence level, adjust the drone parameters and re-acquire defect images at the defect locations where the confidence level is lower than the preset confidence level. The defect image is input into the defect recognition model to obtain a first defect recognition result and a first confidence level of the first defect recognition result output by the defect recognition model. The first defect recognition result is the defect recognition result of the defect image. If the first defect identification result indicates the presence of a defect and the first confidence level is greater than or equal to the preset confidence level, the training sample is determined based on the defect image, the first defect identification result, and the first confidence level.

8. The defect identification method according to claim 7, characterized in that, The optimization of the model parameters of the fully connected layer based on the training samples includes: The training samples are subjected to data augmentation processing; The model parameters are updated based on the processed training samples and the exponentially decaying learning rate.

9. A defect identification device, characterized in that, include: The acquisition module is used to acquire the appearance image of the part to be identified; The recognition module is used to input the appearance image into the defect recognition model to obtain the defect recognition result of the component to be identified and the confidence level of the defect recognition result output by the defect recognition model; The defect identification model includes a backbone network layer and a fully connected layer. The backbone network layer is used to acquire image features of the appearance image, and the fully connected layer is used to determine the defect identification result and the confidence level based on the image features. An optimization module is used to select new training samples based on the confidence level, so as to retrain the defect identification model using the new training samples. The retraining of the defect identification model includes at least optimizing the model parameters of the fully connected layer.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the defect identification method according to any one of claims 1 to 8 through the computer program.

Images