A method for training a defect detection model, as well as related methods and devices.

By training a defect detection model that combines defect segmentation and classification models, the accuracy problem of defect detection in the production environment is solved, and efficient defect detection is achieved.

CN116030000BActive Publication Date: 2026-11-10SHENZHENSHI YUZHAN PRECISION TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202211694035.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-28
Publication Date
2026-11-10
Estimated Expiration
2042-12-28

AI Technical Summary

Technical Problem

Existing defect detection models cannot accurately detect defective products on the production line in a production environment, leading to misjudgments and wasted resources.

Method used

By training a defect segmentation model based on defect segmentation training data and a defect classification model based on defect classification training data, a defect detection model is formed, which can be adapted to perform segmentation and classification operations in complex production environments.

Benefits of technology

It improves the accuracy and precision of defect detection, adapts to complex and ever-changing detection environments, avoids the problem of decreased detection accuracy, and enhances detection results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116030000B_ABST
    Figure CN116030000B_ABST
Patent Text Reader

Abstract

This application discloses a training method for a defect detection model, comprising: training an initial segmentation model based on acquired defect segmentation training data to obtain a defect segmentation model; training an initial classification model based on acquired defect classification training data to obtain a defect classification model; and combining the defect segmentation model and the defect classification model into a defect detection model. The defect detection operation, combining the segmentation operation of the defect segmentation model and the classification operation of the defect classification model, adapts to complex and variable detection environments, rather than only performing defect detection on clear and stable images. This avoids the problem of decreased detection accuracy due to the complexity of the production environment, improves the accuracy and precision of the defect detection model, and enhances the detection effect. This application also discloses a defect detection method and a defect detection system, which have the above-mentioned beneficial effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image data processing technology, and in particular to a method for training a defect detection model, a defect detection method, and a defect detection system. Background Technology

[0002] With the continuous development of information technology, artificial intelligence models are being applied in more and more fields to improve efficiency.

[0003] In related technologies, artificial intelligence models are widely used in everyday life scenarios. However, when applied to production environments, the high complexity of these environments presents numerous challenges. Existing defect detection models are unable to accurately detect defective products on production lines, easily leading to misjudgments and wasted resources.

[0004] Therefore, improving the accuracy of defect detection is a key issue of concern to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a training method for a defect detection model, a defect detection method, and a defect detection system to improve the accuracy of defect detection.

[0006] To address the aforementioned technical problems, this application provides a method for training a defect detection model, comprising:

[0007] The initial segmentation model is trained based on the acquired defect segmentation training data to obtain the defect segmentation model.

[0008] The initial classification model is trained based on the acquired defect classification training data to obtain the defect classification model;

[0009] The defect segmentation model and the defect classification model are combined into a defect detection model.

[0010] Optionally, the initial segmentation model is trained based on the acquired defect segmentation training data to obtain a defect segmentation model, including:

[0011] Acquire sample images;

[0012] Image segmentation of the sample images is performed based on visual matching templates to obtain the defect segmentation training data.

[0013] The initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model.

[0014] Optionally, the visual matching template is a template obtained based on defect feature labeling. Image segmentation is performed on the sample image based on the visual matching template to obtain the defect segmentation training data, including:

[0015] The visual matching template is matched with the sample image to determine the region in the sample image that corresponds to the visual matching template, and the region is cropped to obtain the segmented defect segmentation training data.

[0016] Optionally, the initial segmentation model is trained based on the defect segmentation training data to obtain a defect segmentation model, including:

[0017] The initial segmentation model is built based on the received defect segmentation model construction information and the first training framework;

[0018] The initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model.

[0019] Optionally, the initial segmentation model is trained based on the defect segmentation training data to obtain a defect segmentation model, including:

[0020] The initial UNet semantic segmentation model is built based on the received defect segmentation UNet model construction information and the first training framework;

[0021] The defect segmentation training data is formatted according to the model input data format to obtain the original image data and mask image data as segmentation training input data.

[0022] The initial UNet semantic segmentation model is configured based on the received defect segmentation training configuration information to obtain the configured initial UNet semantic segmentation model.

[0023] The configured initial UNet semantic segmentation model is trained based on the segmentation training input data to obtain the UNet semantic segmentation model as the defect segmentation model.

[0024] Optionally, the initial classification model is trained based on the acquired defect classification training data to obtain a defect classification model, including:

[0025] Obtain the segmented sample data;

[0026] The segmented sample data is labeled based on the acquired annotation information to obtain the defect classification training data;

[0027] The initial classification model is trained based on the defect classification training data to obtain the defect classification model.

[0028] Optionally, the initial classification model is trained based on the defect classification training data to obtain the defect classification model, including:

[0029] The initial classification model is constructed based on the received defect classification model construction information and the second training framework;

[0030] The initial classification model is trained based on the defect classification training data to obtain the defect classification model.

[0031] Optionally, the initial classification model is trained based on the defect classification training data to obtain the defect classification model, including:

[0032] The initial DenseNet classification model is constructed based on the received defective image DenseNet classification model construction information and the second training framework;

[0033] Based on the defect morphology label of each image data in the defect classification training data, the defect classification training data is classified to obtain multiple categories of defect classification training data;

[0034] According to the input path of each category, the defect classification training data of the corresponding category is input into the initial DenseNet classification model, and the model is trained to obtain the DenseNet classification model as the defect classification model.

[0035] This application also provides a defect detection method, including:

[0036] Acquire the target image;

[0037] The target image is segmented using a visual matching template or a defect segmentation model to obtain an image to be detected; wherein, the defect segmentation model is a model trained based on defect segmentation training data, and the visual matching template is a template obtained based on defect feature labeling.

[0038] The image to be detected is classified using a defect classification model to obtain defect detection results; wherein, the defect classification model is trained based on defect classification training data.

[0039] Optionally, the target image is segmented using a visual matching template or a defect segmentation model to obtain an image to be detected, including:

[0040] Obtain the machine parameters and determine the segmentation method to use from the visual matching template and the defect segmentation model based on the machine parameters;

[0041] The target image is segmented based on a determined segmentation method to obtain the image to be detected.

[0042] Optionally, the machine parameters include one or more of the following: the clarity of the target image, the machine model, and the manually set machine processing status;

[0043] Accordingly, the segmentation method used is determined from the visual matching template and the defect segmentation model based on the machine parameters, including:

[0044] When the clarity of the target image obtained from the machine parameters is greater than the preset clarity, the segmentation method is determined to be a visual matching template; when the clarity of the target image obtained from the machine parameters is less than the preset clarity, the segmentation method is determined to be a defect segmentation model.

[0045] And / or when the machine model in the machine parameters belongs to the model type with obvious defects, the segmentation method is determined to be a visual matching template; when the machine model in the machine parameters does not belong to the model type with obvious defects, the segmentation method is determined to be a defect segmentation model.

[0046] And / or when the machine processing status in the machine parameters is template matching, the segmentation method is determined to be visual matching template; when the machine processing status in the machine parameters is model segmentation, the segmentation method is determined to be defect segmentation model.

[0047] Optionally, when the segmentation method is a defect segmentation model, the target image is segmented based on the determined segmentation method to obtain the image to be detected, including:

[0048] Obtain the UNet semantic segmentation model; wherein the UNet semantic segmentation model is trained based on defect segmentation training data;

[0049] The target image is input into the defect segmentation feature processing layer of the UNet semantic segmentation model and defect segmentation processing is performed to obtain the coordinate position of the defect segmentation.

[0050] The target image is cropped based on the coordinate position to obtain the image to be detected.

[0051] Optionally, a defect classification model is used to classify the image to be detected to obtain defect detection results, including:

[0052] Obtain the DenseNet classification model; wherein the DenseNet classification model is trained based on defect classification training data;

[0053] The image to be detected is input into the DenseNet classification model, and defect classification is performed to obtain the defect category corresponding to the image to be detected.

[0054] The defect category is used as the defect detection result.

[0055] Optionally, the target image is segmented using a visual matching template or a defect segmentation model to obtain the image to be detected, including:

[0056] Acquire multiple target images from the machine;

[0057] The multiple target images are classified based on the information of each target image to obtain target images corresponding to multiple threads;

[0058] The target image corresponding to each thread is input to the corresponding segmentation processing thread and segmented to obtain the image to be detected for each thread; wherein, the segmentation processing thread includes a visual matching template or a defect segmentation model;

[0059] Accordingly, a defect classification model is used to classify the image to be detected, and defect detection results are obtained, including:

[0060] The image to be detected for each thread is input into the thread of the corresponding defect classification model and classified, so as to obtain the defect detection result for each thread.

[0061] Optionally, the target image is segmented using a visual matching template or a defect segmentation model to obtain the image to be detected, including:

[0062] Acquire target image from machine;

[0063] The target image is segmented using the first segmentation processing thread to obtain the corresponding first image to be detected;

[0064] Based on the information of the first image to be detected, determine whether the first image to be detected is a re-segmented image;

[0065] If so, the second segmentation processing thread is used to segment the first image to be detected to obtain the second image to be detected;

[0066] Accordingly, a defect classification model is used to classify the image to be detected, and defect detection results are obtained, including:

[0067] The first defect classification model is used to classify the first image to be detected, and the first defect detection result is obtained;

[0068] The second defect classification model is used to classify the second image to be detected, and the second defect detection result is obtained.

[0069] This application also provides a defect detection system, including:

[0070] The first server is used to receive and parse the defect detection request sent by the client, obtain the target image and its corresponding machine parameters, and determine the segmentation method of the target image based on the machine parameters; if the determined segmentation method is a visual matching template, the target image is segmented using the visual matching template to obtain the first image to be detected, and a defect classification request is sent to the second server; if the determined segmentation method is model segmentation, an image segmentation request is sent to the second server.

[0071] The second server is used to receive and parse the request sent by the first server. If the request is an image segmentation request, it receives the target image sent by the first server, segments the target image using a defect segmentation model to obtain a second image to be detected, wherein the defect segmentation model is trained based on defect segmentation training data; and classifies the second image to be detected using a defect classification model to obtain a defect detection result. If the request is a defect classification request, it receives the first image to be detected sent by the first server, classifies the first image to be detected using a defect classification model to obtain a defect detection result; wherein the defect classification model is trained based on defect classification training data.

[0072] Optionally, the first server is further configured to acquire defect segmentation training data and defect classification training data respectively, and send them to the second server;

[0073] The second server is also used to train the initial segmentation model based on the received defect segmentation training data to obtain a defect segmentation model; and to train the initial classification model based on the received defect classification training data to obtain a defect classification model.

[0074] The present application provides a training method for a defect detection model, comprising: training an initial segmentation model based on acquired defect segmentation training data to obtain a defect segmentation model; training an initial classification model based on acquired defect classification training data to obtain a defect classification model; and combining the defect segmentation model and the defect classification model into a defect detection model.

[0075] A defect segmentation model is trained using defect segmentation training data, and a defect classification model is trained using defect classification training data. These models are then combined to form a defect detection model. This enables defect detection operations based on the combination of segmentation operations from the defect segmentation model and classification operations from the defect classification model. This approach adapts to complex and variable detection environments, rather than only performing defect detection on clear and stable images. It avoids the problem of decreased detection accuracy caused by the complexity of the production environment, thereby improving the accuracy and precision of the defect detection model and enhancing the detection performance.

[0076] This application also provides a defect detection method and a defect detection system, which have the above-mentioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0078] Figure 1 A flowchart illustrating a training method for a defect detection model provided in an embodiment of this application;

[0079] Figure 2 A flowchart illustrating a defect detection method provided in an embodiment of this application;

[0080] Figure 3 This is a schematic diagram illustrating the processing of a first scenario provided in an embodiment of this application;

[0081] Figure 4 This is a schematic diagram illustrating the processing of the second scenario provided in the embodiments of this application;

[0082] Figure 5 This is a schematic diagram illustrating the processing of a third scenario provided in an embodiment of this application;

[0083] Figure 6 This is a schematic diagram of the structure of a defect detection system provided in an embodiment of this application. Detailed Implementation

[0084] The core of this application is to provide a training method for a defect detection model, a defect detection method, and a defect detection system to improve the accuracy of defect detection.

[0085] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] In related technologies, artificial intelligence models are widely used in everyday life scenarios. However, when applied to production environments, the high complexity of these environments presents numerous challenges. Existing defect detection models are unable to accurately detect defective products on production lines, easily leading to misjudgments and wasted resources.

[0087] Therefore, this application provides a training method for a defect detection model. A defect segmentation model is trained using defect segmentation training data, and a defect classification model is trained using defect classification training data. These two models are then combined to form a defect detection model. This method enables defect detection based on a combination of segmentation operations from the defect segmentation model and classification operations from the defect classification model. This adapts to complex and variable detection environments, rather than only performing defect detection on clear and stable images. It avoids the problem of decreased detection accuracy due to the complexity of the production environment, improves the accuracy and precision of the defect detection model, and enhances the detection effect.

[0088] The following example illustrates a training method for a defect detection model provided in this application.

[0089] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a training method for a defect detection model provided in an embodiment of this application.

[0090] In this embodiment, the method may include:

[0091] S11, the initial segmentation model is trained based on the acquired defect segmentation training data to obtain the defect segmentation model.

[0092] As can be seen, this step aims to train the initial segmentation model based on the acquired defect segmentation training data, thereby obtaining a defect segmentation model. In other words, it trains a model for segmenting images based on the defect segmentation training data.

[0093] This is mainly because images captured in production environments are highly complex and may contain unclear defect areas. Using template matching for segmentation would reduce the accuracy of defect detection. Therefore, in this embodiment, an image segmentation model is first trained for image segmentation. Then, during the detection process, the defect segmentation model can be used to segment the image to improve the accuracy of defect detection.

[0094] Furthermore, the network structure of this defect segmentation model can be a segmentation network, a semantic segmentation network, or a fully convolutional network. Regardless of the network structure used, the corresponding defect segmentation model can be trained based on this step. The specific training method can be selected based on the network structure, and no specific limitations are made here.

[0095] The defect segmentation training data can be obtained by segmenting the sample data manually, or by segmenting the sample data using template matching. Alternatively, it can be obtained by segmenting relatively blurry sample data manually and segmenting relatively clear sample data using template matching. Using a hybrid approach of manual segmentation and template matching can improve the efficiency of sample data processing while maintaining the accuracy of the defect segmentation training data.

[0096] Furthermore, to improve the accuracy of training the initial segmentation model, reduce the cost of the training process, and increase the efficiency of training, this step may include:

[0097] S111, Obtain the sample image;

[0098] S112, Perform image segmentation on the sample images based on visual matching templates to obtain defect segmentation training data;

[0099] S113, The initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model.

[0100] As can be seen, this optional solution mainly describes how to train the defect segmentation model. In this optional solution, sample images are first acquired. These can be acquired in real-time from the production line, historical sample image data from a database, or relevant sample image data from the internet. Then, image segmentation is performed on the sample images based on a visual matching template to obtain defect segmentation training data. The visual matching template can be a matching template obtained by technicians based on defect feature labeling, or a standard template obtained from a template library based on defect image features. Using visual matching templates for automated image segmentation of sample images effectively improves the efficiency of sample data processing. Finally, the initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model.

[0101] In this embodiment of the application, image segmentation of sample images is performed based on visual matching templates to obtain defect segmentation training data, including:

[0102] The visual matching template is matched with the sample image to determine the region in the sample image that corresponds to the visual matching template. This region is then cropped to obtain the segmented defect segmentation training data.

[0103] Furthermore, for solder joint defect detection, the sample image can be a production image obtained from the solder joint operation station on the production floor, such as a solder joint image acquired from the machine. The sample image can also be a stored historical image, such as solder joint images stored on a server. Furthermore, the sample image can also be image data obtained from the network, such as solder joint images obtained from the network.

[0104] Furthermore, to improve the accuracy of acquiring sample images, S111 in the previous optional scheme may include:

[0105] S1111, determine the corresponding machine from the production site based on the production line information to be tested;

[0106] S1112, acquire sample images from the machine.

[0107] As can be seen, this optional solution mainly describes how to acquire sample images. In this optional solution, the corresponding machine is determined from the production site based on the information of the production line to be inspected. This production line information can be information input by technicians based on requirements, or information determined by the production line system based on production conditions. Regardless of how the production line information is determined, a corresponding sample image can be identified based on the specific characteristics of the production line, improving the accuracy of the inspection for that production line. Finally, the sample image is acquired from the machine.

[0108] Furthermore, to improve the effectiveness and efficiency of training the initial segmentation model and to increase the efficiency of model construction, S112 in the previous optional scheme may include:

[0109] S1121, Build an initial segmentation model based on the received defect segmentation model construction information and the first training framework;

[0110] S1122, The initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model.

[0111] As can be seen, this optional solution mainly describes how to train a defect segmentation model based on the first training framework. In this optional solution, an initial segmentation model is first built based on the received defect segmentation model construction information and the first training framework. The defect segmentation model construction information can be construction information input by technicians, historical construction information obtained from a database, or construction information filtered from a network structure library based on defect segmentation characteristics. The first training framework can be the Keras training framework. Using the Keras training framework can improve the efficiency of model construction, further improving the efficiency of model training. Finally, the initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model. The training process is also executed within the Keras training framework.

[0112] Keras is an open-source artificial neural network library written in Python. It can be used as a high-level application interface for TensorFlow (a symbolic mathematical system based on dataflow programming that is widely used in the programming implementation of various machine learning algorithms) to design, debug, evaluate, apply, and visualize deep learning models.

[0113] Furthermore, to improve the effectiveness and efficiency of training the initial segmentation model and to increase the efficiency of model construction, taking the UNet semantic segmentation model as the segmentation model and the Keras training framework as the first training framework as an example, S112 in the previous optional scheme can include:

[0114] S1121, Based on the received defect segmentation UNet model construction information and Keras training framework, build the initial UNet semantic segmentation model;

[0115] S1122, The defect segmentation training data is formatted according to the model input data format to obtain the original image data and mask image data as the segmentation training input data;

[0116] S1123, Configure the initial UNet semantic segmentation model based on the received defect segmentation training configuration information to obtain the configured initial UNet semantic segmentation model;

[0117] S1124, Train the configured initial UNet semantic segmentation model based on the segmentation training input data to obtain the UNet semantic segmentation model as the defect segmentation model.

[0118] As can be seen, this optional solution mainly explains how to train the defect segmentation model. In this optional solution, an initial UNet semantic segmentation model is built based on the received defect segmentation UNet model construction information and the Keras training framework; that is, the initial UNet semantic segmentation model is built based on the Keras training framework.

[0119] Among them, the U-Net (Semantic Segmentation Network) semantic segmentation model is a semantic segmentation network that can improve the image segmentation effect in practical applications.

[0120] Furthermore, the defect segmentation training data is formatted according to the model input data format to obtain the original image data and mask image data as the segmentation training input data. In other words, the format of the defect segmentation training data is converted into a data format suitable for training the U-Net network.

[0121] Then, based on the received defective segmentation training configuration information, the initial UNet semantic segmentation model is configured to obtain the configured initial UNet semantic segmentation model. The items that need to be configured include, but are not limited to, the learning rate, the number of training steps, and the batch size.

[0122] Finally, the initial UNet semantic segmentation model is trained based on the segmented training input data to obtain the UNet semantic segmentation model as the defect segmentation model, thus realizing the training of the UNet semantic segmentation model.

[0123] In this embodiment, a visual matching template is used to perform image segmentation preprocessing on the sample image to obtain defect segmentation training data; the initial segmentation model is trained based on the defect segmentation training data to obtain a defect segmentation model, which can accurately segment the region to be detected on the image.

[0124] S12, the initial classification model is trained based on the acquired defect classification training data to obtain the defect classification model.

[0125] This step aims to train the initial classification model based on the acquired defect classification training data, thereby obtaining a defect classification model. In other words, it trains a model to classify defects in the segmented image. This defect classification model can then determine whether defects exist in the image, thus achieving defect detection.

[0126] This is mainly due to the various manufacturing defects that exist during the production process. However, manual defect detection suffers from low accuracy, leading to missed defects. Therefore, this step trains a defect classification model to detect defects in the images to be inspected, thereby improving the accuracy of defect detection and reducing labor costs.

[0127] The defect classification model can use a convolutional neural network, a deep convolutional network, or a deep residual network as its network structure, and it must be trained before use. Regardless of the network structure used, the corresponding defect classification model can be trained based on this step. The specific training method can be based on the training method corresponding to the network structure, and no specific limitation is made here.

[0128] Furthermore, to improve the effectiveness of training the defect classification model and increase training efficiency, this step may include:

[0129] S121, Obtain the segmented sample data;

[0130] S122, Based on the obtained annotation information, the segmented sample data is annotated to obtain defect classification training data;

[0131] S123, The initial classification model is trained based on the defect classification training data to obtain the defect classification model.

[0132] As can be seen, this optional solution mainly describes how to train the defect classification model. In this optional solution, segmented sample data is obtained. The segmented sample data can be image sample data obtained by real-time segmentation of existing sample data, image sample data that has already been segmented from historical data, or image sample data that has already been labeled and cropped in a database.

[0133] Then, based on the acquired annotation information, the segmented sample data is labeled to obtain defect classification training data. That is, the presence or absence of defects in the segmented sample data is labeled, with the labeling content including the presence or absence of defects, in order to form corresponding defect classification training data. The acquired annotation information refers to the annotation information received during the image annotation process. This annotation information indicates whether defects exist in the corresponding image. Furthermore, the annotation information can be annotation information input by the annotator, annotation information obtained through classification by other models, or annotation information acquired simultaneously with image acquisition from the network.

[0134] Finally, the initial classification model is trained based on the defect classification training data to obtain the defect classification model.

[0135] Furthermore, to improve the training effect and increase the efficiency of model construction, S123 in the previous optional scheme may include:

[0136] S1231, Construct an initial classification model based on the received defect classification model construction information and the second training framework;

[0137] S1232, The initial classification model is trained based on the defect classification training data to obtain the defect classification model.

[0138] As can be seen, this optional solution mainly describes how to specifically train the defect classification model. In this optional solution, an initial classification model is first constructed based on the received defect classification model construction information and a second training framework. The defect classification model construction information can be construction information input by technicians, historical construction information obtained from a database, or construction information selected from a network structure library based on defect classification characteristics. The second training framework can use the TensorFlow training framework. Since the TensorFlow training framework can quickly construct the corresponding model based on the input construction information, it avoids complex operations during the construction process and improves the efficiency of constructing the initial classification model. Then, the initial classification model is trained based on defect classification training data to obtain the defect classification model. Correspondingly, the training process of the initial classification model is also executed within the TensorFlow training framework, which can effectively improve the training effect and efficiency.

[0139] TensorFlow, a symbolic mathematics system based on dataflow programming, is widely used in the programming implementation of various machine learning algorithms. Models trained using TensorFlow can perform inference under the TensorRT engine, thereby improving the performance and efficiency of image segmentation models.

[0140] Furthermore, to improve training effectiveness and model building efficiency, taking the DenseNet classification model as the defect classification model and the TensorFlow training framework as the second training framework as an example, S123 in the previous optional scheme can also include:

[0141] S1231, Construct an initial DenseNet classification model based on the received flawed image DenseNet classification model construction information and TensorFlow training framework;

[0142] S1232, Based on the defect morphology label of each image data in the defect classification training data, the defect classification training data is classified to obtain defect classification training data of multiple categories;

[0143] S1233, according to the input path of each category, input the defect classification training data of the corresponding category into the initial DenseNet classification model, and train it to obtain the DenseNet classification model as the defect classification model.

[0144] As can be seen, this optional solution mainly illustrates how to specifically train the defect classification model. In this optional solution, an initial DenseNet classification model is constructed based on the received defect image DenseNet classification model construction information and the TensorFlow training framework. The defect image DenseNet classification model construction information can be construction information input by technicians, historical construction information obtained from a database, or construction information selected from a network structure library based on defect classification characteristics; no specific limitations are imposed here. The TensorFlow training framework allows for rapid construction of the corresponding classification model, improving the efficiency of model construction. Furthermore, the DenseNet classification model is a densely connected convolutional network, effectively improving the accuracy of defect image classification.

[0145] Then, based on the defect morphology label of each image in the defect classification training data, the defect classification training data is classified into multiple categories. In other words, it is classified according to different defect morphology labels to allow for more targeted model training and improve the recognition accuracy for each type of defect.

[0146] Finally, the defect classification training data for each category is input into the initial DenseNet classification model according to the input path of each category, and then trained to obtain the DenseNet classification model as the defect classification model. That is, the data is input into the model according to the corresponding path and the corresponding training operation is performed.

[0147] In this embodiment, segmented sample data is labeled to obtain defect classification training data; the initial classification model is trained using the defect classification training data to obtain a defect classification model, which can classify images into OK and NG categories, thereby improving the accuracy of image defect detection.

[0148] S13, combine the defect segmentation model and the defect classification model into a defect detection model.

[0149] Building upon S11 and S12, this step aims to combine the defect segmentation model and the defect classification model into a defect detection model. In other words, the operational models for defect segmentation and classification are combined into a model for defect detection. This defect detection model is then directly used in the defect detection method to improve the accuracy of defect detection and the efficiency of the inference process.

[0150] Furthermore, in this embodiment, to improve the efficiency of model inference, the defect detection model can be format-converted so that it can be applied to the acceleration engine. Therefore, S13 may include:

[0151] Convert the model format of the defect segmentation model and the defect classification model;

[0152] The defect segmentation model and defect classification model after format conversion are identified as the defect detection model.

[0153] In one specific embodiment, the process of converting the defect segmentation model may include:

[0154] First, the defect segmentation model in .h5 format trained by the Keras framework is converted into a defect segmentation model file in ONNX (Open Neural Network Exchange, an open format for deep models). Then, the defect segmentation model in ONNX format is converted into a defect segmentation model file in TensorRT plan format.

[0155] Then, the TensorRT engine is used to convert the above ONNX format defect segmentation model file and TensorRT plan format defect segmentation model file into the first TensorRT engine file respectively.

[0156] Finally, the test set can be used to perform inference tests on the first TensorRT engine file in ONNX format and the first TensorRT engine file in TensorRT plan format, respectively, to obtain the accuracy and running speed, and select the optimal format conversion method without affecting the accuracy.

[0157] In one specific embodiment, the process of converting the defect classification model may include:

[0158] First, the defect classification model in .pb format trained by the TensorFlow framework is converted to ONNX format. Then, the defect classification model file in ONNX format is converted to TensorRT plan format.

[0159] Then, the TensorRT engine is used to convert the above ONNX format defect classification model file and TensorRT plan format defect classification model file into a second TensorRT engine file.

[0160] Finally, the test set can be used to perform inference tests on the second TensorRT engine file in ONNX format and the second TensorRT engine file in TensorRT plan format respectively, to obtain accuracy and running speed, and select the optimal format conversion method without affecting accuracy. In this embodiment, the defect segmentation model and defect classification model after format conversion are determined as the defect detection model.

[0161] In summary, this embodiment trains a defect segmentation model using defect segmentation training data and a defect classification model using defect classification training data, and combines them into a defect detection model. This enables defect detection based on a combination of segmentation operations from the defect segmentation model and classification operations from the defect classification model, adapting to complex and variable detection environments rather than only performing defect detection on clear and stable images. This avoids the problem of decreased detection accuracy due to the complexity of the production environment, improves the accuracy and precision of the defect detection model, and enhances the detection effect.

[0162] The following specific embodiment demonstrates how, based on the defect detection model obtained through the training method of the defect detection model, a defect detection method is implemented by applying the defect detection model to improve the efficiency and accuracy of defect detection and increase the utilization of hardware performance.

[0163] Please refer to Figure 2 , Figure 2 This is a flowchart of a defect detection method provided in an embodiment of this application.

[0164] In this embodiment, the method may include:

[0165] S21, acquire the target image.

[0166] This step aims to acquire the target image. The target image is the image to be inspected for defects. This target image can be a production image captured from the production site, such as a solder joint image captured in real-time from the machine. Alternatively, the target image can be image data acquired from the network, such as a solder joint image obtained from the network. Therefore, the method of acquiring the target image in this embodiment is not limited, and an appropriate acquisition method can be selected based on the specific circumstances.

[0167] S22, the target image is segmented using a visual matching template or a defect segmentation model to obtain the image to be detected; wherein, the defect segmentation model is a model trained based on defect segmentation training data, and the visual matching template is a template obtained based on defect feature labeling.

[0168] Building upon step S21, this step aims to segment the target image using either a visual matching template or a defect segmentation model to obtain the image to be detected. The defect segmentation model is trained using defect segmentation training data, while the visual matching template is a template derived from defect feature labeling.

[0169] Furthermore, this step can determine the appropriate method based on the content of the image to be detected or the state of the production process. For example, if the clarity of the target image is greater than a threshold, a visual matching template is used for image segmentation; if the clarity of the target image is less than or equal to the threshold, a defect segmentation model is used. That is, for relatively clear images, a visual matching template can be used directly for segmentation, without needing to use a defect segmentation model, thus improving efficiency. Therefore, the method for segmenting the target image includes, but is not limited to, determining based on image clarity, the machine model on the production line, manually set processing status, or automatic judgment. Based on these methods, it can be determined whether to use a visual matching template or a defect segmentation model for target image segmentation.

[0170] The process of training the defect segmentation model based on the defect segmentation training data can be referred to in the previous embodiment, and will not be repeated here.

[0171] Furthermore, to improve the accuracy of segmenting the target image, this step may include selecting a suitable segmentation method, which may include:

[0172] S221, Obtain machine parameters, and determine the segmentation method to be used from the visual matching template and defect segmentation model based on the machine parameters;

[0173] S222, The target image is segmented based on the determined segmentation method to obtain the image to be detected.

[0174] As can be seen, this optional solution mainly explains how to select the corresponding segmentation method. In this optional solution, the machine parameters are obtained, and based on these parameters, the segmentation method to be used is determined from the visual matching template and the defect segmentation model. The target image is then segmented using the determined segmentation method to obtain the image to be detected. In other words, different segmentation methods can be selected under different machine parameters to perform appropriate segmentation operations for different machines.

[0175] The machine parameters specify the segmentation method required in the production process. For example, the machine parameters include, but are not limited to, the resolution of the target image, the machine model, and the manually set machine processing status.

[0176] Specifically, when the resolution of the target image obtained from the machine parameters is greater than the preset resolution, a visual matching template is used to segment the target image to obtain the image to be detected. When the resolution of the target image obtained from the machine parameters is less than the preset resolution, a defect segmentation model is used to segment the target image to obtain the image to be detected.

[0177] Specifically, when the machine model in the machine parameters belongs to a model type with obvious defects, a visual matching template is used to segment the target image to obtain the image to be detected. When the machine model in the machine parameters does not belong to a model type with obvious defects, a defect segmentation model is used to segment the target image to obtain the image to be detected.

[0178] Specifically, when the machine processing status in the machine parameters is template matching, the visual matching template is used to segment the target image to obtain the image to be detected. When the machine processing status in the machine parameters is model segmentation, the defect segmentation model is used to segment the target image to obtain the image to be detected.

[0179] In addition, depending on whether the solder joint features are complete, it is possible to determine whether to use a visual matching template or a defect segmentation model to segment the target image. Combining the two methods can accurately extract the image to be detected.

[0180] Furthermore, to improve the accuracy of segmenting the target image using the defect segmentation model, taking the UNet semantic segmentation model as an example, S222 in the previous optional scheme may include:

[0181] S2221, Obtain the UNet semantic segmentation model; wherein, the UNet semantic segmentation model is trained based on defect segmentation training data;

[0182] S2222, the target image is input into the defect segmentation feature processing layer of the UNet semantic segmentation model and defect segmentation processing is performed to obtain the coordinate position of the defect segmentation;

[0183] S2223, crop the target image based on the coordinate position to obtain the image to be detected.

[0184] As can be seen, this optional solution mainly describes how to segment the target image based on the model. In this optional solution, a UNet semantic segmentation model is obtained; wherein, the UNet semantic segmentation model is trained based on defect segmentation training data. The training process of the UNet semantic segmentation model can refer to the training process in the previous embodiment, and will not be repeated here. Then, the target image is input into the defect segmentation feature processing layer of the UNet semantic segmentation model and defect segmentation processing is performed to obtain the coordinate positions of the defect segments; based on the coordinate positions, the target image is cropped to obtain the image to be detected.

[0185] S23, use a defect classification model to classify the image to be detected and obtain the defect detection result; wherein, the defect classification model is trained based on defect classification training data.

[0186] Building upon step S22, this step aims to classify the image to be detected using a defect classification model to obtain defect detection results. The defect classification model is trained using defect classification training data.

[0187] The process of training the defect classification model based on the defect classification training data can refer to the training process in the previous embodiment, and will not be repeated here.

[0188] Furthermore, to improve the accuracy of image classification, taking the DenseNet classification model as an example for defect classification, this step may include:

[0189] S231, Obtain the DenseNet classification model; wherein, the DenseNet classification model is trained based on defect classification training data;

[0190] S232, Input the image to be detected into the DenseNet classification model and perform defect classification to obtain the defect category corresponding to the image to be detected;

[0191] S233, use the defect category as the defect detection result.

[0192] As can be seen, this optional solution mainly describes how to classify defects based on a classification model. In this optional solution, a DenseNet classification model is obtained; the DenseNet classification model is trained based on defect classification training data; the training process of the DenseNet classification model can refer to the training process in the previous embodiment, and will not be repeated here. The image to be detected is input into the DenseNet classification model, and defect classification is performed to obtain the defect category corresponding to the image to be detected; the defect category is used as the defect detection result. That is, the image is divided into two categories: pass and fail. The DenseNet network reduces the risk of gradient vanishing, enhances feature transfer and utilization, and reduces the number of parameters to a certain extent, making it suitable for image classification of complex solder joint shapes.

[0193] The process of classifying defects using the DenseNet classification model may include:

[0194] The image to be detected is input into the input layer of the DenseNet classification model for feature extraction to obtain image features; the image features are then input into the convolutional layer of the DenseNet classification model for convolution processing to obtain the convolution result; the convolution result is then input into the output layer of the DenseNet classification model for processing to obtain the defect category.

[0195] Furthermore, this embodiment can provide implementation methods for single-model parallelism, multi-model parallelism, and multi-model concatenation in practical application scenarios. It can significantly improve the detection speed and load capacity of segmentation and classification models while ensuring accuracy, and simplify the application of multiple models in concatenation or parallel. It also improves server resource utilization and reduces the manpower and material resources required for business upgrades and maintenance. For example, different processing flows exist during implementation. These can include: processing defect images from multiple machines simultaneously, using multiple different models to detect defects, or using multiple models to process images in a pipeline manner.

[0196] Therefore, the defect detection method in this embodiment may include:

[0197] S31, acquire multiple target images from the machine;

[0198] S32, classify multiple target images based on the information of each target image to obtain target images corresponding to multiple threads;

[0199] S33, the target image corresponding to each thread is input to the corresponding segmentation processing thread and segmented to obtain the image to be detected for each thread; wherein, the segmentation processing thread includes a visual matching template or a defect segmentation model;

[0200] S34, input the image to be detected corresponding to each thread into the thread of the corresponding defect classification model and perform classification processing to obtain the defect detection result corresponding to each thread.

[0201] Among them, the information based on the target image can be the state information of the target image, the production equipment information of the target image, or the time information of the target image.

[0202] For example, based on the information of the target images mentioned above, the images can be classified into categories A, B, and C. Correspondingly, threads can be set up based on image classification: thread A for category A target images, thread B for category B target images, and thread C for category C target images. Therefore, by classifying multiple target images based on the information of each target image, the target images corresponding to each thread can be obtained. That is, category A target images for thread A, category B target images for thread B, and category C target images for thread C. The specific category of the target image can be determined based on the information of the target image and is not specifically limited here. Furthermore, each thread includes its own segmentation processing thread and defect classification model thread. Therefore, the target images corresponding to each thread can be input into the segmentation processing thread and the defect classification model thread. That is, category A target images are input into the segmentation processing thread and defect classification model thread of thread A, category B target images are input into the segmentation processing thread and defect classification model thread of thread B, and category C target images are input into the segmentation processing thread and defect classification model thread of thread C.

[0203] As can be seen, this embodiment inputs the target image into the corresponding segmentation processing thread for processing based on the target image information. This allows for the execution of corresponding segmentation operations on different target images based on different information, improving the targeting of image processing. Simultaneously, it also enables parallel processing of the target image, improving image processing efficiency.

[0204] Furthermore, this embodiment can be applied to scenarios involving multiple machines and multiple workstations.

[0205] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the processing of a first scenario provided in an embodiment of this application.

[0206] Figure 3 Multiple target images can be acquired through multiple machines, and then defects are detected in the target images through multiple segmentation processing operations and multiple defect classification operations.

[0207] The defect detection process in this scenario may include:

[0208] Step 1: Acquire multiple target images from multiple machines;

[0209] Step 2: Classify multiple target images based on the production equipment information of each target image to obtain target images corresponding to multiple threads;

[0210] Since multiple machines correspond to multiple different threads, the target images generated in each thread or the defect characteristics of each machine are different. Therefore, in this embodiment, after obtaining the target images of multiple machines, the multiple target images are classified according to the production equipment information corresponding to the target images, and the target images corresponding to each thread are divided, which facilitates the subsequent targeted segmentation and preprocessing of the target images.

[0211] Step 3: Input the target image corresponding to each thread into the corresponding segmentation processing thread and perform segmentation processing to obtain the image to be detected for each thread; wherein, the segmentation processing thread includes a visual matching template or a defect segmentation model;

[0212] In this embodiment, a dedicated segmentation processing thread is configured for each thread, enabling parallel processing of target images from multiple different machines and improving the defect detection efficiency in multi-machine application scenarios.

[0213] Step 4: Input the image to be detected for each thread into the thread of the corresponding defect classification model and perform classification processing to obtain the defect detection result for each thread.

[0214] As can be seen, in order to perform defect detection on a multi-machine production line, this embodiment sets up multiple threads and implements parallel defect detection of target images acquired by multiple machines using multiple threads. This improves the defect detection efficiency in multi-machine application scenarios, while also reducing the demand for hardware resources and lowering the cost in multi-machine detection scenarios.

[0215] Furthermore, this embodiment can also accommodate scenarios where a single workstation and multiple models run in parallel.

[0216] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the processing of the second scenario provided in the embodiments of this application.

[0217] Figure 4 In this process, multiple target images are acquired through a single workstation, and these images are processed based on multiple models.

[0218] The defect detection process in this scenario may include:

[0219] Step 1: Acquire multiple target images from the machine;

[0220] Step 2: Classify multiple target images based on the type information of each target image to obtain target images of multiple types;

[0221] Step 3: Input the target image of each type into the segmentation processing thread corresponding to that type and perform segmentation processing to obtain the image to be detected for each type; wherein, the segmentation processing thread includes a visual matching template or a defect segmentation model;

[0222] Step 4: Input the image to be detected for each type into the thread of the corresponding defect classification model and perform classification processing to obtain the defect detection result for each type.

[0223] Based on the target image type information mentioned above, images can be classified into categories A, B, and C. Correspondingly, threads can be configured based on the image type classification: thread A for category A target images, thread B for category B target images, and thread C for category C target images.

[0224] Therefore, by classifying multiple target images based on the information of each target image, we can obtain the target image corresponding to each thread. That is, thread A corresponds to target image of class A, thread B corresponds to target image of class B, and thread C corresponds to target image of class C.

[0225] Furthermore, each thread includes its own segmentation processing thread and defect classification model thread. Therefore, the target image corresponding to each thread can be input into the segmentation processing thread and the defect classification model thread. That is, target images of class A are input into the segmentation processing thread and defect classification model thread of thread A, target images of class B are input into the segmentation processing thread and defect classification model thread of thread B, and target images of class C are input into the segmentation processing thread and defect classification model thread of thread C.

[0226] As can be seen, this embodiment inputs the target image into the corresponding segmentation processing thread for processing based on the target image information. This allows for the execution of corresponding segmentation operations on target images of different types based on different information, improving the targeting of image processing. Simultaneously, it also enables parallel processing of the target image, improving image processing efficiency.

[0227] Furthermore, this embodiment can be applied to scenarios where a single workstation and multiple models run serially.

[0228] Please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the processing of a third scenario provided in an embodiment of this application.

[0229] Figure 5 In this process, the target image is acquired through a single workstation, and then processed through multiple serial models.

[0230] The defect detection method in this embodiment may include:

[0231] S41, acquire the target image from the machine;

[0232] S42, the first segmentation processing thread is used to segment the target image to obtain the corresponding first image to be detected;

[0233] S43, determine whether the first image to be detected is a re-segmented image based on the information of the first image to be detected;

[0234] S44, If not, use the first defect classification model to classify the first image to be detected and obtain the first defect detection result;

[0235] S45, if so, the second segmentation processing thread is used to segment the first image to be detected to obtain the second image to be detected;

[0236] S46, the second defect classification model is used to classify the second image to be detected, and the second defect detection result is obtained.

[0237] As can be seen, in this step, the target image acquired by the machine can be processed serially, and different defect classification models can be applied to obtain different segmentation results, resulting in defect detection results corresponding to different segmentation processing threads. This realizes defect detection under different segmentation methods, improves the diversity of the defect detection process, and eliminates the need to set up separate defect detection processes for other defect detection results. This improves the accuracy of defect detection and reduces the cost of defect detection.

[0238] In summary, this embodiment implements defect detection operations in specific machine application scenarios, including single-machine parallel processing, multi-machine parallel processing, and single-machine serial processing, improving the efficiency and accuracy of defect detection. In the specific detection process, images are segmented using visual matching templates or image segmentation models to obtain the image to be detected. Then, a defect classification model is used to classify defects in the image to facilitate defect detection. Based on image segmentation and defect classification models, complex images can be effectively processed, rather than simply relying on image matching for defect detection, thus improving the accuracy of defect detection.

[0239] The following describes a defect detection system provided in this application through another specific embodiment.

[0240] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a defect detection system provided in an embodiment of this application.

[0241] In this embodiment, the system may include:

[0242] The first server 601 is used to receive and parse the defect detection request sent by the client, obtain the target image and its corresponding machine parameters, and determine the segmentation method of the target image based on the machine parameters; if the determined segmentation method is visual matching, the target image is segmented using a visual matching template to obtain the first image to be detected, and a defect classification request is sent to the second server; if the segmentation method is determined to be model segmentation, an image segmentation request is sent to the second server.

[0243] The second server 602 is used to receive and parse requests sent by the first server 601. If the request is an image segmentation request, it receives the target image sent by the first server, segments the target image using a defect segmentation model to obtain a second image to be detected, wherein the defect segmentation model is trained based on defect segmentation training data; and classifies the second image to be detected using a defect classification model to obtain a defect detection result. If the request is a defect classification request, it receives the first image to be detected sent by the first server, classifies the first image to be detected using a defect classification model to obtain a defect detection result; wherein the defect classification model is trained based on defect classification training data. In this embodiment, the first server implements the defect detection request and image segmentation operation, while the second server implements model training, image segmentation, and defect classification operations, thereby improving hardware utilization, rationally allocating hardware resources, and improving the efficiency of various operations.

[0244] In some embodiments, the first server 601 is further configured to acquire defect segmentation training data and defect classification training data;

[0245] The second server 602 is also used to receive defect segmentation training data and defect classification training data obtained by the first server 601, train the initial segmentation model based on the defect segmentation training data to obtain a defect segmentation model, and train the initial classification model based on the defect classification training data to obtain a defect classification model.

[0246] In some embodiments, the first server 601 acquires defect segmentation training data including:

[0247] Sample images are acquired, and image segmentation is performed on the sample images based on visual matching templates to obtain defect segmentation training data. The visual matching templates are templates obtained based on defect feature labeling.

[0248] In this embodiment of the application, the second server 602 trains the initial segmentation model based on the defect segmentation training data to obtain the defect segmentation model, which includes:

[0249] An initial segmentation model is built based on the received defect segmentation model construction information and the first training framework;

[0250] The initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model.

[0251] In some embodiments, the first training framework may be the Keras training framework.

[0252] In this embodiment of the application, the first server 601 acquires defect classification training data including:

[0253] Obtain the segmented sample data;

[0254] The segmented sample data is labeled based on the acquired annotation information to obtain defect classification training data.

[0255] In this embodiment of the application, the second server 602 trains the initial classification model based on the defect classification training data to obtain the defect classification model, which includes:

[0256] The initial classification model is constructed based on the received defect classification model construction information and the second training framework;

[0257] The initial classification model is trained based on the defect classification training data to obtain the defect classification model.

[0258] In some embodiments, the second training framework may employ the TensorFlow training framework.

[0259] In some embodiments, the first server 601 performs segmentation processing on the target image using a visual matching template, including:

[0260] The visual matching template is matched with the target image to determine the region in the target image that corresponds to the visual matching template, and this region is cropped to obtain the first image to be detected.

[0261] The first server 601 receives the defect detection request from the client, and simultaneously receives the image data and the corresponding machine and production number. It performs simple preprocessing on the image, uses a visual matching template to obtain the area to be detected, cuts out the area to be detected and converts it into the size required by the model, and sends an image segmentation and defect classification operation request to the second server 602.

[0262] In some embodiments, the first server 601 is further configured to receive and store the defect detection results sent by the second server 602.

[0263] In some embodiments, the second server 602 performs segmentation processing on the target image using a defect segmentation model, including:

[0264] Obtain the trained defect segmentation model;

[0265] The target image is input into the defect segmentation feature processing layer of the defect segmentation model and defect segmentation is performed to obtain the coordinate position of the defect segmentation.

[0266] The target image is cropped based on its coordinate position to obtain the second image to be detected.

[0267] In this embodiment of the application, the defect segmentation model can be the Unet semantic segmentation model.

[0268] In some embodiments, the second server 602 classifies the first / second images to be detected using a defect classification model, including:

[0269] Obtain a trained defect classification model;

[0270] The first / second images to be detected are input into the defect classification model, and defect classification is performed to obtain the defect category corresponding to the first / second images to be detected.

[0271] Use the defect category as the defect detection result.

[0272] In this embodiment of the application, the defect classification model can be the DenseNet classification model.

[0273] In this embodiment, image segmentation and defect classification operations are performed through a second server. In defect detection, the specific location of the region to be detected in the image is detected, and the content in the image is segmented based on the specific location to obtain the region to be detected. A defect classification model is used to classify the image to be detected into different types, such as: pass / fail / uncertain.

[0274] In this embodiment, image preprocessing, image segmentation, and defect classification are deployed on the first server and the second server respectively, forming the effect of parallel and hierarchical operation, which improves the processing efficiency of high batch image samples and allows for better configuration of hardware resources.

[0275] The aforementioned defect detection system performs preprocessing operations and image segmentation and defect classification operations in a hierarchical manner through a first server and a second server, thereby realizing parallel processing of image preprocessing and defect classification operations in the defect detection method and improving the overall execution speed of defect detection.

[0276] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0277] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0278] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0279] The above provides a detailed description of the training method, defect detection method, training device, and defect detection system for a defect detection model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A training method for a defect detection model, characterized in that, include: The initial segmentation model is trained based on the acquired defect segmentation training data to obtain the defect segmentation model. The defect segmentation model is used to segment the region to be detected on the image; Specifically, the corresponding machine is determined from the production site based on the production line information to be inspected, sample images are obtained from the machine, and image segmentation is performed on the sample images based on a visual matching template to obtain the defect segmentation training data; the visual matching template is obtained based on defect feature calibration or based on defect image features. The initial classification model is trained based on the acquired defect classification training data to obtain a defect classification model; the defect classification model is used to determine whether defects exist in an image. The defect segmentation model and the defect classification model are combined into a defect detection model; The defect segmentation model is obtained by training the initial segmentation model based on the defect segmentation training data, including: The initial segmentation model is built based on the received defect segmentation model construction information and the first training framework; The initial segmentation model is trained based on the defect segmentation training data to obtain the defect segmentation model; The initial classification model is trained based on the acquired defect classification training data to obtain a defect classification model, including: Obtain the segmented sample data; The segmented sample data is labeled based on the acquired annotation information to obtain the defect classification training data; The initial DenseNet classification model is constructed based on the received defective image DenseNet classification model construction information and the second training framework; Based on the defect morphology label of each image data in the defect classification training data, the defect classification training data is classified to obtain multiple categories of defect classification training data; According to the input path of each category, the defect classification training data of the corresponding category is input into the initial DenseNet classification model, and the model is trained to obtain the DenseNet classification model as the defect classification model.

2. The training method according to claim 1, characterized in that, The visual matching template is a template obtained based on defect feature labeling. Image segmentation is performed on the sample image based on the visual matching template to obtain the defect segmentation training data, including: The visual matching template is matched with the sample image to determine the region in the sample image that corresponds to the visual matching template, and the region is cropped to obtain the segmented defect segmentation training data.

3. The training method according to claim 1, characterized in that, The initial segmentation model is trained based on the defect segmentation training data to obtain a defect segmentation model, including: The initial UNet semantic segmentation model is built based on the received defect segmentation UNet model construction information and the first training framework; The defect segmentation training data is formatted according to the model input data format to obtain the original image data and mask image data as segmentation training input data. The initial UNet semantic segmentation model is configured based on the received defect segmentation training configuration information to obtain the configured initial UNet semantic segmentation model. The configured initial UNet semantic segmentation model is trained based on the segmentation training input data to obtain the UNet semantic segmentation model as the defect segmentation model.

4. The training method according to claim 1, characterized in that, The initial classification model is trained based on the defect classification training data to obtain the defect classification model, including: The initial classification model is constructed based on the received defect classification model construction information and the second training framework; The initial classification model is trained based on the defect classification training data to obtain the defect classification model.

5. A defect detection method, characterized in that, include: Acquire the target image; The target image is segmented using a visual matching template or a defect segmentation model to obtain an image to be detected; wherein, the defect segmentation model is a model trained based on defect segmentation training data, and the visual matching template is a template obtained based on defect feature labeling or a template obtained based on defect image features; the defect segmentation model is used to segment the region to be detected on the image. The image to be detected is classified using a defect classification model to obtain defect detection results; wherein, the defect classification model is trained based on defect classification training data; the defect classification model is used to determine whether defects exist in the image; The process of segmenting the target image using a visual matching template or a defect segmentation model to obtain the image to be detected includes: acquiring machine parameters, and determining the segmentation method to be used from the visual matching template and the defect segmentation model based on the machine parameters; and segmenting the target image based on the determined segmentation method to obtain the image to be detected. The machine parameters include one or more of the following: the clarity of the target image, the machine model, and the manually set machine processing status. Accordingly, the segmentation method used is determined from the visual matching template and the defect segmentation model based on the machine parameters, including: When the clarity of the target image obtained from the machine parameters is greater than the preset clarity, the segmentation method is determined to be a visual matching template; when the clarity of the target image obtained from the machine parameters is less than the preset clarity, the segmentation method is determined to be a defect segmentation model. And / or when the machine model in the machine parameters belongs to the model type with obvious defects, the segmentation method is determined to be a visual matching template; when the machine model in the machine parameters does not belong to the model type with obvious defects, the segmentation method is determined to be a defect segmentation model. And / or when the machine processing status in the machine parameters is template matching, the segmentation method is determined to be visual matching template; when the machine processing status in the machine parameters is model segmentation, the segmentation method is determined to be defect segmentation model.

6. The defect detection method according to claim 5, characterized in that, When the segmentation method is a defect segmentation model, the target image is segmented based on the determined segmentation method to obtain the image to be detected, including: Obtain the UNet semantic segmentation model; wherein the UNet semantic segmentation model is trained based on defect segmentation training data; The target image is input into the defect segmentation feature processing layer of the UNet semantic segmentation model and defect segmentation processing is performed to obtain the coordinate position of the defect segmentation. The target image is cropped based on the coordinate position to obtain the image to be detected.

7. The defect detection method according to claim 5, characterized in that, The image to be detected is classified using a defect classification model to obtain defect detection results, including: Obtain the DenseNet classification model; wherein the DenseNet classification model is trained based on defect classification training data; The image to be detected is input into the DenseNet classification model, and defect classification is performed to obtain the defect category corresponding to the image to be detected. The defect category is used as the defect detection result.

8. The defect detection method according to claim 5, characterized in that, The target image is segmented using a visual matching template or a defect segmentation model to obtain the image to be detected, including: Acquire multiple target images from the machine; The multiple target images are classified based on the information of each target image to obtain target images corresponding to multiple threads; The target image corresponding to each thread is input to the corresponding segmentation processing thread and segmented to obtain the image to be detected for each thread; wherein, the segmentation processing thread includes a visual matching template or a defect segmentation model; Accordingly, a defect classification model is used to classify the image to be detected, and defect detection results are obtained, including: The image to be detected for each thread is input into the thread of the corresponding defect classification model and classified, so as to obtain the defect detection result for each thread.

9. The defect detection method according to claim 5, characterized in that, The target image is segmented using a visual matching template or a defect segmentation model to obtain the image to be detected, including: Acquire target image from machine; The target image is segmented using the first segmentation processing thread to obtain the corresponding first image to be detected; Based on the information of the first image to be detected, determine whether the first image to be detected is a re-segmented image; If so, the second segmentation processing thread is used to segment the first image to be detected to obtain the second image to be detected; Accordingly, a defect classification model is used to classify the image to be detected, and defect detection results are obtained, including: The first defect classification model is used to classify the first image to be detected, and the first defect detection result is obtained; The second defect classification model is used to classify the second image to be detected, and the second defect detection result is obtained.

10. A defect detection system, characterized in that, include: The first server receives and parses the defect detection request sent by the client, obtains the target image and its corresponding machine parameters, and determines the segmentation method for the target image based on the machine parameters. If the determined segmentation method is a visual matching template, the target image is segmented using the visual matching template to obtain a first image to be detected, and a defect classification request is sent to the second server. If the determined segmentation method is model segmentation, an image segmentation request is sent to the second server. The visual matching template is a template obtained based on defect feature calibration or a template obtained based on defect image features. The defect segmentation model is used to segment the region to be detected on the image. Among them, the machine parameters are obtained, and the segmentation method to be used is determined from the visual matching template and the defect segmentation model based on the machine parameters; The machine parameters include one or more of the following: the clarity of the target image, the machine model, and the manually set machine processing status. When the clarity of the target image obtained from the machine parameters is greater than the preset clarity, the segmentation method is determined to be a visual matching template; when the clarity of the target image obtained from the machine parameters is less than the preset clarity, the segmentation method is determined to be a defect segmentation model. And / or when the machine model in the machine parameters belongs to the model type with obvious defects, the segmentation method is determined to be a visual matching template; when the machine model in the machine parameters does not belong to the model type with obvious defects, the segmentation method is determined to be a defect segmentation model. And / or when the machine processing status in the machine parameters is template matching, the segmentation method is determined to be visual matching template; when the machine processing status in the machine parameters is model segmentation, the segmentation method is determined to be defect segmentation model. The second server receives and parses requests sent by the first server. If the request is an image segmentation request, it receives the target image sent by the first server, segments the target image using a defect segmentation model to obtain a second image to be detected, wherein the defect segmentation model is trained based on defect segmentation training data; and classifies the second image to be detected using a defect classification model to obtain a defect detection result. If the request is a defect classification request, it receives the first image to be detected sent by the first server, classifies the first image to be detected using a defect classification model to obtain a defect detection result; wherein the defect classification model is trained based on defect classification training data; and the defect classification model is used to determine whether defects exist in the image. The first server is further configured to acquire defect segmentation training data and defect classification training data respectively, and send them to the second server. The second server is also used to train the initial segmentation model based on the received defect segmentation training data to obtain a defect segmentation model; and to train the initial classification model based on the received defect classification training data to obtain a defect classification model.

Citation Information

Patent Citations

  • Image segmentation model training method and classification-based enhanced image segmentation method

    CN110866931A

  • Model training method, welding spot defect detection method and device, and electronic equipment

    CN111696108A

  • Semantic segmentation method and system for space science experiment data

    CN112101362A

  • Model training method and device, storage medium and electronic equipment

    CN114677567A

  • Model training method, detection method and device for small target detection

    CN115511776A