Defect detection model construction method, defect detection method, defect detection device and electronic equipment

By constructing a defect detection model that integrates image classification, object detection, and object segmentation modules, the problem of insufficient detection capability of the YOLO model for small objects is solved, achieving high-precision and flexible defect detection results.

CN118918092APending Publication Date: 2024-11-08BINZHOU WEIQIAO NATIONAL SCIENCE & TECHNOLOGY ADVANCED TECHNOLOGY RESEARCH INSTITUTE +1

Patent Information

Application Number
CN202411068899.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The YOLO model has limited ability to detect small targets in defect detection and is highly dependent on the dataset, resulting in poor detection performance.

Method used

A defect detection model is constructed, integrating an image classification module, an object detection module, and an object segmentation module. It is trained by fusing the YOLO feature extraction network with the training feature map, and the model is optimized by using a comprehensive loss function and an error backpropagation algorithm to improve the detection accuracy of small objects.

Benefits of technology

It achieves high-precision defect detection for small targets, improves the overall performance of the detection model, adapts to the detection capability of targets of different sizes, and enhances the flexibility and robustness of the detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918092A_ABST
    Figure CN118918092A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of defect detection, and discloses a defect detection model construction method, a defect detection method, a defect detection device and electronic equipment. The construction method comprises the following steps: inputting sample data into a YOLO feature extraction network to obtain a training feature map; training a pre-constructed defect recognition model by adopting the training feature map; the defect identification model comprises an image classification module, a target detection module and a target segmentation module; fusing the YOLO feature extraction network with the trained defect identification model to obtain a defect detection model; wherein the image classification module is used for classifying an input feature map and outputting a normal image without defects or an abnormal image with defects, the target detection module is used for outputting a bounding box position of a target defect, and the target segmentation module is used for outputting a pixel-level segmentation map of the target defect. The technical effect of the method is explained. According to the invention, the defect detection precision of the defect detection model on the small target can be improved, and the detection performance is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of defect detection technology, for example, to a method for constructing a defect detection model, a defect detection method, a device and an electronic device. Background Art

[0002] With the continuous development of technologies such as machine vision and artificial intelligence, new technologies such as deep learning have been introduced into the field of defect detection to improve the accuracy and efficiency of detection. Among them, the YOLO (You Only Look Once) model is widely used in surface defect detection of workpieces or products due to its advantages such as fast detection speed, high accuracy, strong robustness and easy deployment, so as to detect whether there are scratches, cracks, deformation, color difference, defects and other defects on the surface of workpieces or products.

[0003] In the object detection task, the YOLO model detects and segments known types of object defects by dividing the image into multiple grids and predicting the possible object bounding boxes and categories in each grid. However, when dealing with small objects, the YOLO model may have positioning errors or missed detections, resulting in poor detection results and limited detection capabilities for small objects. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] The embodiments of the present disclosure provide a method for constructing a defect detection model, a defect detection method, a device and an electronic device, which can improve the defect detection accuracy of the defect detection model for small targets and improve the detection performance of the defect detection model.

[0006] In some embodiments, a method for constructing a defect detection model is provided, including: inputting sample data into a YOLO feature extraction network to obtain a training feature map; using the training feature map to train a pre-constructed defect recognition model; the defect recognition model includes an image classification module, a target detection module and a target segmentation module; fusing the YOLO feature extraction network with the trained defect recognition model to obtain a defect detection model; wherein the image classification module is used to classify the input feature map and output a normal image without defects or an abnormal image with defects, the target detection module is used to output the bounding box position of the target defect, and the target segmentation module is used to output a pixel-level segmentation map of the target defect.

[0007] Optionally, the image classification module includes a global average pooling layer and two fully connected layers, the global average pooling layer is used to output the average value of all elements in the training feature map, and the average value is input into two fully connected layers for classification to output a normal image or an abnormal image; the target detection module includes a Detect module, and the Detect module is used to output the bounding box position of the target defect; the target segmentation module includes an encoder and a decoder, and the encoder and decoder at the same level are jump-connected, the encoder is used to extract features from the training feature map to obtain feature data, and the decoder is used to reconstruct the obtained feature data to output a pixel-level segmentation map of the target defect.

[0008] Optionally, a pre-constructed defect recognition model is trained using a training feature map, including: obtaining a comprehensive loss function of the defect recognition model; iteratively training the pre-constructed defect recognition model using an error back propagation algorithm and a training feature map, and calculating a loss value based on the comprehensive loss function; when the rate of change of the loss value is less than or equal to a rate of change threshold, a trained defect recognition model is obtained.

[0009] Optionally, the comprehensive loss function of the defect recognition model is obtained in the following manner: obtain the first loss function, the second loss function and the third loss function of the image classification module, the target detection module and the target segmentation module respectively; and construct a comprehensive loss function based on the first loss function, the second loss function and the third loss function.

[0010] Optionally, a pre-built defect recognition model is trained using a training feature map, including: inputting the training feature map into an image classification module, a target detection module and a target segmentation module for training respectively; or, using the training feature map to train the image classification module; using a feature map corresponding to an abnormal image output by the image classification module to train the target detection module; using a feature map corresponding to an abnormal image output by the image classification module and the bounding box position of the target defect output by the target detection module to train the target segmentation module.

[0011] Optionally, the sample data includes abnormal samples and normal samples; the training feature map includes a first feature map and a second feature map; the training feature map is obtained in the following manner: the abnormal sample is input into the YOLO feature extraction network to obtain the first feature map of the abnormal sample; the normal sample is input into the YOLO feature extraction network to obtain the second feature map of the normal sample.

[0012] Optionally, the training feature map also includes a first average feature map and / or a second average feature map; the training feature map is obtained in the following manner: the average value of multiple first feature maps is calculated to obtain a first average feature map; the average value of multiple second feature maps is calculated to obtain a second average feature map.

[0013] Optionally, the training feature map also includes one or more of a first residual feature map, a second residual feature map, a third residual feature map and a fourth residual feature map; the training feature map is obtained in the following manner: performing residual analysis on the first feature map and the second feature map to obtain a first residual feature map; performing residual analysis on the second average feature map of the first feature map and the second feature map to obtain a second residual feature map; performing residual analysis on the first average feature map of the first feature map and the second feature map to obtain a third residual feature map; performing residual analysis on the first average feature map of the first feature map and the second average feature map of the second feature map to obtain a fourth residual feature map.

[0014] In some embodiments, a defect detection method is provided, including: acquiring a target image of a target to be detected; performing defect detection on the target image using a defect detection model as described in the above embodiments to obtain defect detection data; and determining the defect state of the target to be detected based on the defect detection data.

[0015] In some embodiments, a device for constructing a defect detection model is provided, including: an acquisition module, configured to input sample data into a YOLO feature extraction network to obtain a training feature map; a training module, configured to use the training feature map to train a pre-constructed defect recognition model; and a fusion module, configured to fuse the YOLO feature extraction network with the trained defect recognition model to obtain a defect detection model.

[0016] In some embodiments, a defect detection device is provided, including: an acquisition module, configured to acquire a target image of a target to be detected; a detection module, configured to perform defect detection on the target image using a defect detection model to obtain defect detection data; and an analysis module, configured to determine the defect state of the target to be detected based on the defect detection data.

[0017] In some embodiments, an electronic device is provided, comprising: a processor; and a memory storing program instructions, wherein the processor is configured to execute the defect detection model construction method as described in the above embodiments, or the defect detection method as described in the above embodiments.

[0018] The defect detection model construction method, defect detection method, device and electronic device provided by the embodiments of the present disclosure can achieve the following technical effects: In the disclosed embodiment, a defect recognition model is pre-built, and the defect recognition model integrates an image classification module, a target detection module, and a target segmentation module. Among them, the image classification module is used to output a normal image without defects or an abnormal image with defects, so as to achieve global image classification and determine whether the image contains defects. The target detection module is used to output the bounding box position of the target defect to determine the approximate position of the target defect and quickly locate the defect area. The target segmentation module is used to output a pixel-level segmentation map of the target defect to accurately segment the target defect, facilitate the discovery of subtle defects, and make defect detection more accurate.

[0019] The disclosed embodiment trains the defect recognition model by inputting sample data into the YOLO feature extraction network for feature extraction to obtain a training feature map. Since the defect recognition model integrates three modules: image classification, target detection, and target segmentation, the trained defect recognition model can achieve comprehensive defect detection from coarse to fine and from overall to local, thereby improving the comprehensive detection performance of the defect detection model and making the defect detection model more accurate in the process of detecting small targets.

[0020] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein: Figure 1 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure; Figure 2 is a schematic diagram of a method for constructing a defect detection model provided by an embodiment of the present disclosure; Figure 3 is a schematic diagram of an image classification module provided by an embodiment of the present disclosure; Figure 4 is a schematic diagram of a target segmentation module provided by an embodiment of the present disclosure; Figure 5 is a schematic diagram of a method for constructing a defect detection model provided by another embodiment of the present disclosure; Figure 6 is a schematic diagram of a defect detection method provided by an embodiment of the present disclosure; Figure 7 is a schematic diagram of a device for constructing a defect detection model provided by an embodiment of the present disclosure; Figure 8 is a schematic diagram of a defect detection device provided by an embodiment of the present disclosure; Fig. 9 is a schematic diagram of a controller provided by an embodiment of the present disclosure.

[0022] Reference numerals: 1. Electronic equipment; 10. Equipment body; 70. Defect detection model construction device; 71. Acquisition module; 72. Training module; 73. Fusion module; 80. Defect detection device; 81. Acquisition module; 82. Detection module; 83. Analysis module; 90. Controller; 900. Processor; 901. Memory; 902. Communication interface; 903. Bus. DETAILED DESCRIPTION

[0023] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0024] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0025] Unless otherwise stated, the term "plurality" means two or more.

[0026] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0027] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A, B, A and B.

[0028] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0030] As an efficient deep learning algorithm, the YOLO (You Only Look Once) model has been widely used in the field of defect detection due to its advantages such as fast detection speed, high accuracy, strong robustness and easy deployment. The YOLO model is an algorithm that transforms the target detection problem into a regression problem. It can simultaneously predict multiple bounding boxes (Bounding Box) and their categories in a single network. The YOLO series of algorithms include YOLOv1, YOLOv2 (YOLO9000), YOLOv3, YOLOv4 and YOLOv5. Each version is improved and optimized based on the previous version to improve the detection speed and accuracy.

[0031] Although the application of the YOLO model in defect detection has many advantages, it also has some disadvantages. These include: (1) Limited detection capability for small targets: When dealing with small targets, the YOLO model may have positioning errors or missed detections, and the feature extraction of small targets is not fine enough, resulting in poor detection results. (2) Strong dataset dependence: The performance of the YOLO model depends on the quality and diversity of the training dataset. The YOLO model can only detect targets of known defect types, and the targets are relatively similar. If the coverage of defect types in the training dataset is not wide enough or the annotations are not accurate enough, the detection effect of the trained model in actual applications will be affected, and the detection effect will be poor for defects with large differences or changes or unknown defects.

[0032] Although each version of the YOLO series of algorithms has been improved and optimized based on the previous version, increasing the detection speed and accuracy, the above problems still exist.

[0033] The embodiments of the present disclosure provide a method for constructing a defect detection model, a defect detection method, a device, and an electronic device, which can improve the defect detection accuracy of the defect detection model for small targets and improve the detection performance of the defect detection model. The method for constructing a defect detection model and the defect detection method are run on electronic devices such as mobile phones, tablets, computers, etc.

[0034] Combination Figure 1 As shown, the embodiment of the present disclosure provides an electronic device 1, including: a device body 10 and a controller 90. The controller 90 is installed in the device body 10.

[0035] Optionally, combined Fig. 9As shown, the controller 90 includes a processor 900. The processor 900 can input sample data into the YOLO feature extraction network to obtain a training feature map; the training feature map can be used to train a pre-built defect recognition model; the YOLO feature extraction network can be integrated with the trained defect recognition model to obtain a defect detection model.

[0036] Combination Figure 1 and Fig. 9 The electronic device 1 shown in the figure, the embodiment of the present disclosure provides a method for constructing a defect detection model, such as Figure 2 As shown, the construction method includes: S201, the processor inputs the sample data into the YOLO feature extraction network to obtain a training feature map.

[0037] In the disclosed embodiment, the YOLO feature extraction network adopts the feature extraction network structure of YOLOv8 or YOLOv10.

[0038] S202, the processor uses the training feature graph to train the pre-built defect recognition model.

[0039] Among them, the defect recognition model includes an image classification module, a target detection module and a target segmentation module. The image classification module is used to classify the input feature map and output a normal image without defects or an abnormal image with defects. The target detection module is used to output the bounding box position of the target defect. The target segmentation module is used to output the pixel-level segmentation map of the target defect.

[0040] S203, the processor fuses the YOLO feature extraction network with the trained defect recognition model to obtain a defect detection model.

[0041] In the method for constructing a defect detection model provided by the embodiment of the present disclosure, a defect recognition model is pre-constructed, and the defect recognition model integrates an image classification module, a target detection module, and a target segmentation module. Among them, the image classification module is used to output a normal image without defects or an abnormal image with defects, so as to achieve global image classification and determine whether the image contains defects. The target detection module is used to output the bounding box position of the target defect to determine the approximate position of the target defect and quickly locate the defect area. The target segmentation module is used to output a pixel-level segmentation map of the target defect to accurately segment the target defect, facilitate the discovery of subtle defects, and make defect detection more accurate.

[0042] The disclosed embodiment trains the defect recognition model by inputting sample data into the YOLO feature extraction network for feature extraction to obtain a training feature map. Since the defect recognition model integrates three modules: image classification, target detection, and target segmentation, the trained defect recognition model can achieve comprehensive defect detection from coarse to fine and from overall to local, thereby improving the comprehensive detection performance of the defect detection model and making the defect detection model more accurate in the process of detecting small targets.

[0043] Optionally, the image classification module includes a global average pooling layer and two fully connected layers, the global average pooling layer is used to output the average value of all elements in the training feature map, and the average value is input into the two fully connected layers for classification to output a normal image or an abnormal image.

[0044] In this embodiment, the image classification module is designed as a global average pooling layer and two fully connected layers connected in sequence, so as to further extract high-level features from the training feature map and perform accurate image classification. This enables the defect detection model to achieve global image classification and meet the needs of focusing on whether there are defects globally first, and then focusing on the specific location of the defects in the field of defect detection.

[0045] In this embodiment, the global average pooling layer is used to average all elements in each training feature map and output the average value corresponding to each training feature map. The average value represents the average activation degree of all features in each training feature map. Since the global average pooling layer is insensitive to the size of the feature map (the global average pooling layer always outputs a vector of a fixed size), the image classification module can process input images of different sizes without additional adjustment or cropping, thereby improving the flexibility and practicality of the defect detection model. The average value corresponding to each training feature map output by the global average pooling layer is then used as the input of the subsequent fully connected layer. The fully connected layer can learn the complex mapping relationship between the input data (the output of the global average pooling layer) and the output category (i.e., normal image and abnormal image), and map each training feature map to its corresponding category, i.e., normal image or abnormal image. Among them, the first fully connected layer is used to further extract high-level features and perform feature dimensionality reduction, and the second fully connected layer directly outputs the classification result, i.e., whether the image is a normal image or an abnormal image.

[0046] For example, Figure 3 As shown in the figure, the training feature map of (H, W, C) (representing a training feature map with C channels and a spatial resolution of H×W) is used as the input of the image classification module. After passing through the global average pooling layer and the first fully connected layer, the number of channels is reduced to C / 2. After passing through the second fully connected layer, it is mapped to a normal image without defects or an abnormal image with defects to achieve global image classification.

[0047] Optionally, the target detection module includes a Detect module, and the Detect module is used to output the bounding box position of the target defect.

[0048] The Detect module is a key component in the YOLO network and is mainly responsible for the final output processing of target detection. The Detect module is responsible for receiving the feature map from the previous network layer and outputting the target detection results, including the category, confidence and bounding box coordinates of the target. In this embodiment, the Detect module is used to receive the training feature map output by the YOLO feature extraction network or the feature map corresponding to the abnormal image output by the image classification module, and output the bounding box position of the target (defect) (including the center point coordinates, width and height of the bounding box).

[0049] In some embodiments, the target detection module includes a Detect module of YOLOv8 or YOLOv10. Compared with the Detect module of previous versions, the Detect module of YOLOv8 introduces CSPDarknet53 as the backbone network, which improves the perception of spatial information and thus improves the efficiency of target detection. YOLOv8 uses PANet (Path Aggregation Network) as the neck network. Through fast feature fusion, the Detect module can obtain features from multiple levels of the underlying network, further improving the accuracy of target recognition. YOLOv8 achieves effective fusion of multi-scale features through enhanced feature fusion mechanisms such as PANet, improving the model's detection ability for targets of different sizes. Compared with the Detect module of previous versions, the Detect module of YOLOv10 introduces a dual allocation strategy, eliminating the need for NMS (non-maximum suppression) in the traditional YOLO model, thereby achieving faster and more efficient target detection.

[0050] Optionally, the target segmentation module includes an encoder and a decoder, and the encoder and decoder at the same level are jump-connected. The encoder is used to extract features from the training feature map to obtain feature data, and the decoder is used to reconstruct the obtained feature data to output a pixel-level segmentation map of the target defect.

[0051] In this embodiment, combined Figure 4As shown in the figure, the target segmentation module is constructed based on the U-Net network model. The target segmentation module includes an encoder and a decoder, and the encoder and decoder at the same level are skipped. The encoder gradually reduces the resolution of the feature map through convolutional layers and pooling layers, and extracts high-level features. The decoder gradually restores the resolution of the feature map through upsampling operations, and uses skip connections to fuse the feature map of the encoder with the feature map of the decoder to restore the detailed information of the image. The skip connection directly concatenates the feature map of the encoder with the feature map of the corresponding layer of the decoder, thereby realizing the fusion of features of different scales, so as to retain the detailed information of the image and improve the accuracy of segmentation.

[0052] Optionally, the number of encoders is two, the number of decoders is two, and corresponding encoders and decoders are jump-connected.

[0053] In this embodiment, combined Figure 4 As shown, the encoder (i.e., downsampling) gradually reduces the resolution of the image through convolution (conv) and maximum pooling (maxpool) operations, and extracts high-level features of the image to capture global information and structural features in the image. The decoder (i.e., upsampling) gradually restores the resolution of the image through convolution (conv) and upconvolution (upconv) operations, and uses jump connections to concatenate (Concat) the feature map of the encoder with the feature map of the corresponding layer of the decoder to retain the detailed information of the image and improve the accuracy of segmentation. In the last two layers of the decoder, convolutional layers are used to generate the final segmentation map. The number of channels (k) of the final segmentation map is the same as the number of categories, and each channel represents the score or probability of a category. In some embodiments, these scores or probabilities are converted into final category labels through a softmax function or an argmax operation.

[0054] Optionally, a pre-built defect recognition model is trained using a training feature map, including: inputting the training feature map into an image classification module, a target detection module, and a target segmentation module for training.

[0055] In this embodiment, the training feature map is simultaneously input into the image classification module, the target detection module and the target segmentation module for joint training. In each iteration of the defect recognition model, the three modules update their internal parameters according to the input training feature map to minimize their respective loss functions. Since the three modules are trained synchronously, they can share the same data and context information, so as to work more harmoniously, so as to improve the overall performance of the defect recognition model, including the accuracy of classification, the precision of detection and the meticulousness of segmentation. Since the three modules update parameters at the same time in each iteration without waiting for the training of other modules to be completed, the training time is reduced. In addition, synchronous training enables the defect recognition model to handle multiple tasks at the same time to improve the robustness of the defect recognition model, so that the defect recognition model can learn richer feature representations.

[0056] Optionally, a pre-built defect recognition model is trained using a training feature map, including: using the training feature map to train an image classification module; using a feature map corresponding to the abnormal image output by the image classification module to train a target detection module; using a feature map corresponding to the abnormal image output by the image classification module and the bounding box position of the target defect output by the target detection module to train a target segmentation module.

[0057] In this embodiment, sequential training is performed based on the dependency between the modules, that is, the image classification module, the target detection module and the target segmentation module are trained in sequence using the training feature graph. The sequential training makes full use of the correlation between the modules to gradually improve the recognition ability of the entire defect recognition model.

[0058] Specifically, first, the image classification module is trained using the training feature map. The training feature map contains various types of defect samples and normal samples, so that the image classification module can learn to distinguish whether there are defects in the image and make preliminary classifications of the defects. After the image classification module is trained, the feature map corresponding to the abnormal image (i.e., the image classified as containing defects) output by it is used as input to train the target detection module, so that the target detection module can learn to accurately locate the position of the defect in the abnormal image, that is, generate the bounding box position of the defect. Finally, the target segmentation module is trained using the feature map corresponding to the abnormal image output by the image classification module and the bounding box position of the target defect output by the target detection module. By combining the feature map and the bounding box position of the target defect in the feature map, the target segmentation module can more accurately identify the specific shape and contour of the defect and achieve pixel-level defect segmentation.

[0059] In this embodiment, through sequential training, each module can be further optimized based on its previous module to avoid data duplication and waste, ensuring that each module can fully utilize the most relevant information. Since there is a sequence and dependency relationship between the training of modules, the modules can work better together.

[0060] Combination Figure 5 As shown, the embodiment of the present disclosure provides another method for constructing a defect detection model, including: S501, the processor inputs the sample data into the YOLO feature extraction network to obtain a training feature map.

[0061] S502: The processor obtains a comprehensive loss function of a defect recognition model.

[0062] Among them, the defect recognition model includes an image classification module, a target detection module and a target segmentation module. The image classification module is used to classify the input feature map and output a normal image without defects or an abnormal image with defects. The target detection module is used to output the bounding box position of the target defect. The target segmentation module is used to output the pixel-level segmentation map of the target defect.

[0063] Optionally, the comprehensive loss function of the defect recognition model is obtained in the following manner: obtain the first loss function, the second loss function and the third loss function of the image classification module, the target detection module and the target segmentation module respectively; and construct a comprehensive loss function based on the first loss function, the second loss function and the third loss function.

[0064] In this embodiment, the loss functions of the image classification module, the target detection module, and the target segmentation module are calculated respectively, so as to more accurately evaluate the performance of each module and optimize it in a targeted manner. At the same time, the first loss function, the second loss function, and the third loss function are integrated into a comprehensive loss function to guide the training of the entire model, and global optimization is performed by adjusting the loss weights of each module.

[0065] Optionally, a comprehensive loss function is constructed in the following manner: L0=α×L1+β×L2+γ×L3; wherein L0, L1, L2, L3 are the comprehensive loss function, the first loss function, the second loss function and the third loss function, respectively, α, β, γ are the loss weights of the first loss function, the second loss function and the third loss function, respectively, α+β+γ=1, and α≥0, β≥0, γ≥0.

[0066] In this embodiment, global optimization is performed by adjusting the loss weights corresponding to the first loss function, the second loss function, and the third loss function, namely, α, β, and γ. The adjustment of α, β, and γ is gradually adjusted during the iterative training of the pre-built defect recognition model by the error back propagation algorithm, and when the change rate of the loss value is less than or equal to the change rate threshold, the corresponding α, β, and γ values ​​are used as the final loss weight values.

[0067] It should be noted that the first loss function is used to measure the performance of the image classification module in the image classification task, reflecting the difference between the image category predicted by the image classification module and the actual category. The first loss function is cross-entropy loss (Cross-Entropy Loss) or other loss functions suitable for classification tasks. The second loss function is used to evaluate the accuracy of the target detection module in locating the defect position to optimize the position of the bounding box and the detected defect category. The second loss function is a bounding box regression loss (such as Smooth L1 Loss) or a classification loss (such as cross-entropy loss) function. The third loss function is used to evaluate the pixel-level segmentation accuracy of the defect area of ​​the target segmentation module, characterizing the difference between the segmentation map predicted by the target segmentation module and the actual segmentation map. The third loss function is Dice Loss, cross-entropy loss function or a combination thereof. The specific forms of the first loss function, the second loss function and the third loss function need to be set by technicians according to the actual training process or application scenario.

[0068] S503, the processor uses the error back propagation algorithm and the training feature map to iteratively train the pre-built defect recognition model, and calculates the loss value according to the comprehensive loss function.

[0069] The error back propagation algorithm calculates the gradient of the loss function to the network parameters and updates the network parameters in the opposite direction of the gradient to minimize the loss function. In the embodiment of the present disclosure, the loss function is a comprehensive loss function.

[0070] S504: When the change rate of the loss value is less than or equal to the change rate threshold, the processor obtains a trained defect recognition model.

[0071] S505, the processor fuses the YOLO feature extraction network with the trained defect recognition model to obtain a defect detection model.

[0072] In the method for constructing a defect detection model provided by the embodiment of the present disclosure, a comprehensive loss function is used to evaluate the performance of the defect recognition model and guide the optimization direction of the defect recognition model. In the process of iteratively training the pre-constructed defect recognition model using the training feature map, an error back propagation algorithm (such as a back propagation algorithm) is used to continuously update the weights and parameters of the defect recognition model. The error back propagation algorithm adjusts the model parameters by gradient descent and other methods according to the loss value calculated by the comprehensive loss function to reduce the loss value and improve the recognition accuracy of the model. At the same time, the rate of change of the loss value is monitored, and the iterative training is stopped when the rate of change of the loss value is less than or equal to the preset rate of change threshold to avoid overfitting or unnecessary long-term training. When the rate of change of the loss value meets the stopping condition (less than or equal to the rate of change threshold), it is considered that the defect detection model has been trained, and the defect detection model can more accurately identify defects in the image, including classification, positioning, and segmentation.

[0073] In this embodiment, the comprehensive loss function is used to evaluate and optimize the performance of the defect detection model in multiple dimensions, so as to improve the accuracy of defect recognition more comprehensively. The combination of the error back propagation algorithm and iterative training enables the defect detection model to adjust parameters according to the loss value feedback in each iteration, so as to gradually approach the optimal solution, thereby improving the training speed and effect of the model.

[0074] Optionally, the sample data includes abnormal samples and normal samples; the training feature map includes a first feature map and a second feature map; the training feature map is obtained in the following manner: the abnormal sample is input into the YOLO feature extraction network to obtain the first feature map of the abnormal sample; the normal sample is input into the YOLO feature extraction network to obtain the second feature map of the normal sample.

[0075] In this embodiment, an abnormal sample refers to an image sample including a defect, and a normal sample refers to an image sample not including a defect. By inputting the abnormal sample and the normal sample into the YOLO feature extraction network respectively, feature extraction is performed to obtain a first feature map of the abnormal sample and a second feature map of the normal sample. The first feature map and the second feature map are used as training feature maps so that the subsequent defect recognition model can more easily distinguish them and learn the difference between them, thereby improving the accuracy of defect detection.

[0076] Optionally, the training feature map also includes a first average feature map and / or a second average feature map; the training feature map is obtained in the following manner: the average value of multiple first feature maps is calculated to obtain a first average feature map; the average value of multiple second feature maps is calculated to obtain a second average feature map.

[0077] In this embodiment, when there are multiple abnormal samples, each abnormal sample will be input into the YOLO feature extraction network to generate its own first feature map. By calculating the average value of these first feature maps at the same spatial position, a first average feature map is obtained to represent the common features of all abnormal samples in the feature space. Similarly, for multiple normal samples, each normal sample will be input into the YOLO feature extraction network to generate its own second feature map. By calculating the average value of these second feature maps at the same spatial position, a second average feature map is obtained to represent the common features of all normal samples in the feature space.

[0078] In this embodiment, by using the first average feature map and / or the second average feature map as a training feature map, the influence of noise and outliers in a single sample feature map can be reduced, thereby improving the robustness of the defect detection model. In addition, since the average feature map represents the common features of multiple samples in the feature space, the average feature map can capture more generalized feature information, thereby enabling the defect detection model to perform better on unseen samples, thereby enhancing the generalization ability of the defect detection model.

[0079] Specifically, the average value of the first feature maps is calculated in the following manner to obtain the first average feature map: first average feature map = (first feature Figure 1 +First Feature Figure 2 +…+first feature map n) / n. The first feature map n represents the nth first feature map, and the value of n is 1, 2, 3, 4…

[0080] Specifically, the average value of the plurality of second feature maps is calculated in the following manner to obtain the second average feature map: second average feature map = (second feature Figure 1 + Second Feature Figure 2 +…+second feature map n) / n. The second feature map n represents the nth second feature map, and the value of n is 1, 2, 3, 4…

[0081] Optionally, the training feature map also includes one or more of a first residual feature map, a second residual feature map, a third residual feature map and a fourth residual feature map; the training feature map is obtained in the following manner: performing residual analysis on the first feature map and the second feature map to obtain a first residual feature map; performing residual analysis on the second average feature map of the first feature map and the second feature map to obtain a second residual feature map; performing residual analysis on the first average feature map of the first feature map and the second feature map to obtain a third residual feature map; performing residual analysis on the first average feature map of the first feature map and the second average feature map of the second feature map to obtain a fourth residual feature map.

[0082] In this embodiment, the training feature map is further enriched by residual analysis, and the training feature map also includes one or more of the first residual feature map, the second residual feature map, the third residual feature map, and the fourth residual feature map. These residual feature maps are obtained by comparing the differences between feature maps at different levels, aiming to capture more detailed and in-depth feature information and reveal more subtle and in-depth differences between abnormal samples and normal samples. These difference information can improve the detection sensitivity of the defect detection model to abnormal samples. By comparing the differences between feature maps at different levels and fusing them, the robustness and generalization ability of the defect detection model can be improved, so that the defect detection model can maintain good detection performance on unseen samples.

[0083] Specifically, the first feature map and the second feature map are subjected to residual analysis in the following manner to obtain a first residual feature map: first residual feature map=relu(conv(first feature map-second feature map)), wherein conv is a convolution operation, and relu is a relu function activation operation.

[0084] Specifically, residual analysis is performed on the first feature map and the second average feature map of the second feature map in the following manner to obtain the second residual feature map: second residual feature map=relu(conv(first feature map-second average feature map)), wherein conv is a convolution operation, and relu is a relu function activation operation.

[0085] Specifically, the first average feature map and the second feature map of the first feature map are subjected to residual analysis in the following manner to obtain a third residual feature map: the third residual feature map = relu (conv (first average feature map - second feature map)), wherein conv is a convolution operation, and relu is a relu function activation operation.

[0086] Specifically, residual analysis is performed on the first average feature map of the first feature map and the second average feature map of the second feature map in the following manner to obtain a fourth residual feature map: fourth residual feature map=relu(conv(first average feature map-second average feature map)), where conv is a convolution operation and relu is a relu function activation operation.

[0087] Combination Figure 6 As shown, the present disclosure provides a defect detection method, including: S601: The processor obtains a target image of a target to be detected.

[0088] S602: The processor performs defect detection on the target image using the defect detection model to obtain defect detection data.

[0089] Among them, the defect detection model described in the embodiment of the present disclosure is a defect detection model constructed according to the defect detection model construction method described in the above embodiment.

[0090] S603: The processor determines the defect status of the target to be detected according to the defect detection data.

[0091] By adopting the defect detection method provided by the embodiment of the present disclosure, it is possible to obtain a target image of a target to be detected. The target to be detected is any object with a physical entity. The target image can be the overall structure of the target to be detected, or it can be a partial structural component. After obtaining the target image of the detection target, the target image is input into a defect detection model, and defect detection is performed on the target image to obtain defect detection data. The defect detection data includes the image category, that is, whether it is an abnormal image or a normal image. In the case of an abnormal image, the defect detection data also includes the bounding box position of the target defect and the pixel-level segmentation map of the target defect, so as to make the defect area in the target image more intuitive and improve the accuracy of defect detection.

[0092] In addition, since the defect detection model described in the embodiment of the present disclosure is a defect detection model constructed according to the defect detection model construction method described in the above embodiment, the technical effects of the defect detection model construction method described in the above embodiment are also possessed by this embodiment and will not be repeated here.

[0093] Combination Figure 7 As shown, the embodiment of the present disclosure provides a defect detection model construction device 70, including an acquisition module 71, a training module 72 and a fusion module 73. The acquisition module 71 is configured to input sample data into a YOLO feature extraction network to obtain a training feature map; the training module 72 is configured to use the training feature map to train a pre-constructed defect recognition model; the fusion module 73 is configured to fuse the YOLO feature extraction network with the trained defect recognition model to obtain a defect detection model.

[0094] The defect detection model construction device 70 provided by the embodiment of the present disclosure can execute the defect detection model construction method described in the above embodiment. Therefore, the technical effects of the defect detection model construction method described in the above embodiment are also possessed by this embodiment and will not be repeated here.

[0095] Optionally, the acquisition module 71 is further configured to input the abnormal sample into the YOLO feature extraction network to obtain a first feature map of the abnormal sample; and input the normal sample into the YOLO feature extraction network to obtain a second feature map of the normal sample.

[0096] Optionally, the acquisition module 71 is further configured to calculate an average value of multiple first feature maps to obtain a first average feature map.

[0097] Optionally, the acquisition module 71 is further configured to calculate an average value of multiple second feature maps to obtain a second average feature map.

[0098] Optionally, the acquisition module 71 is further configured to perform residual analysis on the first feature map and the second feature map to obtain a first residual feature map.

[0099] Optionally, the acquisition module 71 is further configured to perform residual analysis on the first feature map and the second average feature map of the second feature map to obtain a second residual feature map.

[0100] Optionally, the acquisition module 71 is further configured to perform residual analysis on the first average feature map and the second feature map of the first feature map to obtain a third residual feature map.

[0101] Optionally, the acquisition module 71 is further configured to perform residual analysis on the first average feature map of the first feature map and the second average feature map of the second feature map to obtain a fourth residual feature map.

[0102] Optionally, the training module 72 is also configured to obtain a comprehensive loss function of the defect recognition model; use the error back propagation algorithm and the training feature map to iteratively train the pre-constructed defect recognition model, and calculate the loss value according to the comprehensive loss function; when the rate of change of the loss value is less than or equal to the rate of change threshold, obtain the trained defect recognition model.

[0103] Optionally, the training module 72 is also configured to obtain the first loss function, the second loss function and the third loss function of the image classification module, the target detection module and the target segmentation module respectively; and construct a comprehensive loss function based on the first loss function, the second loss function and the third loss function.

[0104] Optionally, the training module 72 is further configured to input the training feature map into the image classification module, the target detection module and the target segmentation module for training respectively.

[0105] Optionally, the training module 72 is also configured to use the training feature map to train the image classification module; use the abnormal image output by the image classification module to train the target detection module; use the abnormal image output by the image classification module and the bounding box position of the target defect output by the target detection module to train the target segmentation module.

[0106] Combination Figure 8As shown, the embodiment of the present disclosure provides a defect detection device 80, including an acquisition module 81, a detection module 82 and an analysis module 83. The acquisition module 81 is configured to acquire a target image of a target to be detected; the detection module 82 is configured to perform defect detection on the target image using a defect detection model to obtain defect detection data; the analysis module 83 is configured to determine the defect state of the target to be detected according to the defect detection data.

[0107] The defect detection device 80 provided in the embodiment of the present disclosure can execute the defect detection method described in the above embodiment. Therefore, the technical effects of the defect detection method described in the above embodiment are also possessed by this embodiment and will not be repeated here.

[0108] Combination Figure 1 As shown, an embodiment of the present disclosure provides an electronic device 1, including: a device body 10 and a controller 90. The controller 90 is installed in the device body 10. The installation relationship described here is not limited to being placed inside the device body 10, but also includes installation connections with other components of the electronic device 1, including but not limited to physical connections, electrical connections, or signal transmission connections. It can be understood by those skilled in the art that the controller 90 can be adapted to a feasible device body 10, thereby realizing other feasible embodiments.

[0109] Optionally, combined Fig. 9 As shown, the controller 90 includes a processor 900 and a memory 901. Optionally, the controller 90 may also include a communication interface 902 and a bus 903. The processor 900, the communication interface 902, and the memory 901 may communicate with each other through the bus 903. The communication interface 902 may be used for information transmission. The processor 900 may call the logic instructions in the memory 901 to execute the defect detection model construction method described in the above embodiment, or the defect detection method described in the above embodiment.

[0110] In addition, the logic instructions in the memory 901 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0111] The memory 901 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 900 executes the function application and data processing by running the program instructions / modules stored in the memory 901, that is, the method for constructing the defect detection model in the above embodiment, or the defect detection method in the above embodiment.

[0112] The memory 901 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 901 may include a high-speed random access memory and may also include a non-volatile memory.

[0113] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the defect detection model construction method described in the above embodiment, or the defect detection method described in the above embodiment.

[0114] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0115] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0116] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0117] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0118] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for constructing a defect detection model, characterized in that: include: Input the sample data into the YOLO feature extraction network to obtain the training feature map; The pre-built defect recognition model is trained using the training feature graph; The defect recognition model includes an image classification module, an object detection module, and an object segmentation module; The YOLO feature extraction network is integrated with the trained defect recognition model to obtain a defect detection model; Among them, the image classification module is used to classify the input feature map and output a normal image without defects or an abnormal image with defects. The target detection module is used to output the bounding box position of the target defect. The target segmentation module is used to output the pixel-level segmentation map of the target defect.

2. The construction method according to claim 1, characterized in that: The image classification module includes a global average pooling layer and two fully connected layers. The global average pooling layer is used to output the average value of all elements in the training feature map, and the average value is input into the two fully connected layers for classification to output a normal image or an abnormal image. The target detection module includes a Detect module, which is used to output the bounding box position of the target defect; The target segmentation module includes an encoder and a decoder. The encoder and decoder at the same level are jump-connected. The encoder is used to extract features from the training feature map to obtain feature data, and the decoder is used to reconstruct the obtained feature data to output a pixel-level segmentation map of the target defect.

3. The construction method according to claim 1 or 2, characterized in that: The pre-built defect recognition model is trained using the training feature graph, including: Obtain the comprehensive loss function of the defect recognition model; The error back propagation algorithm and training feature graph are used to iteratively train the pre-built defect recognition model, and the loss value is calculated according to the comprehensive loss function; When the rate of change of the loss value is less than or equal to the rate of change threshold, a trained defect recognition model is obtained.

4. The construction method according to claim 3, characterized in that: The comprehensive loss function of the defect recognition model is obtained as follows: Obtaining a first loss function, a second loss function, and a third loss function of an image classification module, an object detection module, and an object segmentation module, respectively; A comprehensive loss function is constructed based on the first loss function, the second loss function and the third loss function.

5. The construction method according to claim 1 or 2, characterized in that: The pre-built defect recognition model is trained using the training feature graph, including: Input the training feature map into the image classification module, the object detection module and the object segmentation module for training respectively; or, The image classification module is trained using the training feature map; the target detection module is trained using the feature map corresponding to the abnormal image output by the image classification module; the target segmentation module is trained using the feature map corresponding to the abnormal image output by the image classification module and the bounding box position of the target defect output by the target detection module.

6. The construction method according to claim 1 or 2, characterized in that: The sample data includes abnormal samples and normal samples; the training feature map includes a first feature map and a second feature map; The training feature map is obtained as follows: Input the abnormal sample into the YOLO feature extraction network to obtain the first feature map of the abnormal sample; The normal sample is input into the YOLO feature extraction network to obtain the second feature map of the normal sample.

7. The construction method according to claim 6, characterized in that: The training feature map also includes a first average feature map and / or a second average feature map; the training feature map is obtained in the following manner: Calculate an average of the plurality of first characteristic maps to obtain a first average characteristic map; An average value of the plurality of second characteristic maps is calculated to obtain a second average characteristic map.

8. The construction method according to claim 6, characterized in that: The training feature map also includes one or more of the first residual feature map, the second residual feature map, the third residual feature map, and the fourth residual feature map; The training feature map is obtained as follows: Performing residual analysis on the first feature map and the second feature map to obtain a first residual feature map; Performing residual analysis on the first feature map and the second average feature map of the second feature map to obtain a second residual feature map; Performing residual analysis on the first average feature map and the second feature map of the first feature map to obtain a third residual feature map; Perform residual analysis on the first average characteristic map of the first characteristic map and the second average characteristic map of the second characteristic map to obtain a fourth residual characteristic map.

9. A defect detection method, characterized in that: include: Acquire a target image of a target to be detected; Using the defect detection model according to any one of claims 1 to 8, performing defect detection on the target image to obtain defect detection data; Determine the defect status of the target to be inspected based on the defect inspection data.

10. A device for constructing a defect detection model, characterized in that: include: The acquisition module is configured to input the sample data into the YOLO feature extraction network to obtain a training feature map; A training module, configured to train a pre-built defect recognition model using a training feature graph; The fusion module is configured to fuse the YOLO feature extraction network with the trained defect recognition model to obtain a defect detection model.

11. A defect detection device, characterized in that: include: An acquisition module is configured to acquire a target image of a target to be detected; The detection module is configured to perform defect detection on the target image using the defect detection model to obtain defect detection data; The analysis module is configured to determine the defect status of the target to be detected based on the defect detection data.

12. An electronic device, characterized in that: include: processor; and A memory storing program instructions, wherein the processor is configured to execute the method for constructing a defect detection model as described in any one of claims 1 to 8, or the defect detection method as described in claim 9.

Citation Information

Patent Citations

  • PCB defect image detection method based on improved deep learning algorithm

    CN115409797A

  • Spinning cake defect detection method based on improved YOLOv5 model

    CN115705637A

  • Defect identification method and device fusing target detection model and image segmentation model

    CN116363064A

Cited By

  • Defect target detection method and device for robot fish operation in transformer

    CN119169009A

  • End-to-end anomaly detection method and system based on positive and negative sample fusion, and medium

    CN119557817A

  • Electric power operation image detection distillation method based on yov10

    CN119600407A

  • Image recognition system and method based on big data technology

    CN119693801A

  • Hull rust defect laser cleaning method and wall-climbing robot

    CN119832067A