Defect detection method, defect detection network training method and related equipment
Through pre-trained defect detection network and RTCP algorithm, combined with adaptive learning of multiple detection heads and fields, the problems of low manual detection efficiency and low accuracy are solved, and efficient and accurate product appearance defect detection is achieved, suitable for multiple applications. Type of product.
Patent Information
- Application Number
- CN202410481156.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-04-22
AI Technical Summary
Existing product appearance inspection mainly relies on artificial naked eyes, resulting in low detection efficiency, low accuracy and easy missed inspection, and the inability to achieve full-time rapid detection.
The pre-trained defect detection network is adopted, combined with multiple defect detection heads and RTCP algorithms, and efficient and accurate detection of product appearance defects through image preprocessing, motion trajectory control and field adaptive learning.
It improves the accuracy and efficiency of product appearance defect detection, reduces the risk of defect missed detection, is suitable for different models and types of products, and enhances the robustness and identification accuracy of the detection network.
Smart Images

Figure CN118314110B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence (AI) detection, and in particular to a defect detection method, a defect detection network training method, and related equipment. Background Art
[0002] During the production and processing of products, their surfaces are more likely to have defects such as scratches, bruises, bumps, and raised particles. If the quality of the product appearance cannot be effectively tested, defective products will enter the market.
[0003] Existing product appearance inspections are generally conducted manually with the naked eye. Due to human conditions and limited energy, full-time inspections and rapid inspections are not possible. Inspection efficiency and accuracy are greatly affected by personal energy, physical strength and other physiological factors. Summary of the invention
[0004] In view of the above, it is necessary to provide a defect detection method, a defect detection network training method and related equipment, which can solve the problem that existing manual inspection is prone to missed detection and has low detection efficiency.
[0005] In a first aspect, the present application provides a defect detection method, which is applied to a computer device, wherein the computer device is deployed with a pre-trained defect detection network, wherein the defect detection network includes multiple defect detection heads, wherein the multiple defect detection heads include a defect detection head for detecting a defect whose aspect ratio is within a preset ratio range, a defect detection head for detecting a defect whose aspect ratio is outside a preset ratio range, and a defect detection head for determining whether a defect exists. The defect detection method includes: obtaining an appearance image of a product to be inspected; inputting the appearance image into a pre-trained defect detection network; obtaining a detection result of each defect detection head of the multiple defect detection heads for the appearance image; and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected.
[0006] By adopting the above technical scheme, the pre-trained defect detection network includes multiple detection heads, which can ensure the detection accuracy of regular size defects while being able to detect defects with abnormal aspect ratios, such as aspect ratios that are too large or too small. At the same time, by configuring the detection head to judge whether there are defects, it can be used to detect new types of defects that appear in the product production process, reduce the risk of missed defects, and generate the final detection result of the product to be inspected based on the detection result of each defect detection head, which can improve the accuracy of product appearance defect detection. For example, the union of the detection results of each defect detection head is used as the detection result of the product to be inspected. For example, the defect detection network can realize the union of the detection results of each defect detection head, or the computer device can realize the union of the detection results of each defect detection head.
[0007] In one possible implementation, inputting an appearance image into a pre-trained defect detection network includes: preprocessing the appearance image, the preprocessing including grayscale conversion processing and image cutting and splicing processing, the image cutting and splicing processing including cutting the appearance image into multiple sub-images in the length direction, and splicing at least two of the multiple sub-images in the width direction; inputting the preprocessed appearance image into the pre-trained defect detection network.
[0008] By adopting the above technical solution, before the image is input into the network, the image is preprocessed, including but not limited to grayscale conversion, image cutting and splicing, so as to reduce the data processing amount of the network and ensure that the size of the input image meets the input size requirement of the network, which can not only ensure the accuracy of appearance defect recognition, but also greatly improve the network recognition rate. For example, for an image taken by a line scan camera, the appearance image can be cut into multiple sub-images in the length direction, and at least two of the multiple sub-images can be spliced in the width direction.
[0009] In one possible implementation, obtaining the detection result of each defect detection head among multiple defect detection heads for the appearance image, and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected includes: obtaining the detection result of each defect detection head among multiple defect detection heads for the appearance image and the confidence of the detection result; if there is at least one defect detection head whose detection result confidence is greater than or equal to a preset confidence, determining that the detection result of the product to be inspected is defective, and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected; if the confidence of the detection result of each defect detection head is less than the preset confidence, determining that the detection result of the product to be inspected is non-defective.
[0010] By adopting the above technical scheme, when there is a situation where the confidence level of the detection result of at least one defect detection head is greater than or equal to the preset confidence level, it indicates that the defect detection network has detected the existence of a credible defect in the product to be detected, and the detection result of the product to be detected can be determined as the existence of a defect. The union of the detection results of each defect detection head can be used as the final detection result of the product to be detected to avoid missed defect detection. When the confidence level of the detection result of each defect detection head is less than the preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product to be detected can be determined as the absence of a defect.
[0011] In a possible implementation, the defect detection network may also be configured with a first preset confidence and a second preset confidence, the first preset confidence being greater than the second preset confidence, and in the case where the confidence of the detection result of at least one defect detection head is greater than or equal to the first preset confidence, it indicates that the defect detection network has detected a credible defect in the product to be detected, and it can be determined that the detection result of the product to be detected is a defect, and the union of the detection results of each defect detection head can be used as the final detection result of the product to be detected; in the case where the confidence of the detection result of each defect detection head is less than the second preset confidence, it indicates that the defect detection network has not detected a credible defect, and it can be determined that the detection result of the product to be detected is no defect; in the case where the confidence of the detection result of at least one defect detection head is less than the second preset confidence, it indicates that the defect detection network has not detected a credible defect, and it can be determined that the detection result of the product to be detected is no defect; When the first preset confidence level is greater than or equal to the second preset confidence level, and there is no detection result of any defect detection head whose confidence level is greater than or equal to the first preset confidence level, it indicates that the defect detection network cannot determine whether there is a defect in the product appearance. The defect detection network can output a vector identification information of the detection abnormality, and the computer device can output a preset detection abnormality prompt information based on the vector identification information. For example, the preset detection abnormality prompt information is "It is impossible to determine whether there is a defect in the product appearance, please perform secondary image acquisition and detection". The staff can adjust the acquisition parameters of the image acquisition device (for example, the shooting angle of the line scan camera, the brightness of the ambient light source, etc.) to perform image acquisition and detection on the product to be inspected again. The value of the preset confidence level can be set according to actual needs.
[0012] In one possible implementation, a computer device is communicatively connected to an image acquisition device, the image acquisition device includes a line scan camera, and before acquiring an appearance image of a product to be inspected, the defect detection method further includes: acquiring motion trajectory coordinates of the product to be inspected; based on a rotated tool center point (RTCP) algorithm and the motion trajectory coordinates, controlling the line scan camera to perform interpolation motion so that the distance between the line scan camera and the product to be inspected is maintained within a preset distance, and the preset distance is set based on the focal length of the line scan camera.
[0013] By adopting the above technical solution, since the line scan camera has the characteristics of high shooting clarity and small focal length, by combining RTCP technology with the characteristics of the line scan camera to perform linked image acquisition, the line scan camera can perform trajectory following movement for products with irregular corners in appearance, so that the line scan camera can always maintain a distance from the product to be inspected within the focal length of the camera during complex movements, thereby improving the clarity of the camera's image acquisition and the accuracy of data collection.
[0014] In a possible implementation, based on the RTCP algorithm and the motion trajectory coordinates, the line scan camera is controlled to perform interpolation motion, including: calibrating the image acquisition endpoint of the line scan camera as the tool tip following point in the RTCP algorithm, calibrating the product to be inspected as the processed product in the RTCP algorithm, and calibrating the point to be photographed on the product to be inspected as the actual contact point between the tool tip and the processed product in the RTCP algorithm; based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, the line scan camera is controlled to perform interpolation motion so that the distance between the image acquisition endpoint and the point to be photographed is within a preset distance.
[0015] By adopting the above technical scheme, the image acquisition endpoint of the line scan camera is calibrated as the tool tip following point in the RTCP algorithm, the product to be inspected is calibrated as the processed product in the RTCP algorithm, and the point to be photographed on the product to be inspected is calibrated as the actual contact point between the tool tip and the processed product in the RTCP algorithm. Based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, the axis where the line scan camera is located is controlled to perform interpolation motion, thereby realizing the interpolation motion of the line scan camera, and then combining the RTCP technology with the characteristics of the line scan camera, so that the line scan camera can perform linkage image acquisition. During complex motion, the distance between the line scan camera and the point to be photographed can always be kept within the focal length of the camera, thereby improving the clarity of the line scan camera image acquisition and the accuracy of data acquisition.
[0016] In a possible implementation, the line scan camera is controlled to perform interpolation motion based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, including: constructing a camera coordinate system corresponding to the line scan camera; converting the motion trajectory coordinates of the point to be photographed into the camera coordinate system to obtain the target trajectory coordinates of the point to be photographed in the camera coordinate system; and controlling the line scan camera to perform interpolation motion based on the RTCP algorithm and the target trajectory coordinates.
[0017] By adopting the above technical solution, the target trajectory coordinates are obtained by converting the motion trajectory coordinates of the point to be photographed into the camera coordinate system, and the line scan camera is controlled to perform interpolation motion based on the target trajectory coordinates and the RTCP algorithm, so that the distance between the line scan camera and the point to be photographed can always be kept within the focal length of the camera during complex motion, thereby improving the clarity of the line scan camera's image acquisition and the accuracy of data acquisition.
[0018] In a possible implementation, a defect detection network is obtained based on a training sample set and a preset domain adaptive network training, the training sample set includes a first sample set with labels and a second sample set without labels, and the second sample set and the first sample set belong to different data domains, the training sample set also includes a third sample set, the third sample set is constructed based on the first sample set and the second sample set, and the third sample set is processed by domain classification, so that the image feature differences of training samples belonging to different data domains in the third sample are minimized.
[0019] By adopting the above technical scheme, the first sample set can be a labeled training sample from the source data domain, and the second sample set can be an unlabeled training sample from the target data domain. The domain adaptive network is trained based on training samples from different data domains, which can improve the robustness of the defect detection network obtained by training for different types of product appearance defect detection, and the number of samples in the second sample set can be much smaller than the number of samples in the first sample set, which is convenient for cross-domain detection, and the third sample set is constructed by domain classification processing, and the image feature differences of product appearance images belonging to different data domains in the third sample set are minimized. For example, minimizing the image feature differences can mean that product appearance images belonging to different data domains in the third sample set cannot be domain classified (cannot distinguish whether they are from the source data domain or the target data domain), and the third sample set is used to optimize the training of the defect detection network, which can further improve the robustness and recognition accuracy of the defect detection network for different types of product appearance defect detection.
[0020] In a possible implementation, the training sample set also includes a fourth sample set, which is constructed based on the first sample set and the second sample set, and the fourth sample set is processed by maximum mean discrepancy (MMD) so that the maximum mean discrepancy of training samples belonging to different data domains in the fourth sample set is minimized.
[0021] By adopting the above technical solution, the fourth sample set is constructed through the MMD method, so that the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set is minimized, and the defect detection network is optimized and trained using the fourth sample set, which can further improve the robustness and recognition accuracy of the defect detection network for different types of product appearance defect detection.
[0022] In a second aspect, the present application provides a network training method, the method comprising: using a first sample set to train a preset domain adaptive network, the training samples in the first sample set including labeled product appearance image samples; using a second sample set to train a preset domain adaptive network to obtain a defect detection network for product appearance detection, the training samples in the second sample set including unlabeled product appearance image samples, and the second sample set and the first sample set belong to different data domains; constructing a third sample set based on the first sample set and the second sample set, and processing the third sample set so that the image feature differences of product appearance images belonging to different data domains in the third sample set are minimized; using the third sample set to optimize the training of the defect detection network.
[0023] By adopting the above technical solution, the first sample set can be a labeled training sample from the source data domain, and the second sample set can be an unlabeled training sample from the target data domain. The domain adaptive network is trained based on training samples from different data domains, which can improve the robustness of the detection network for detecting appearance defects of products of different types / models, and the number of samples in the second sample set can be much smaller than the number of samples in the first sample set, which can facilitate cross-domain detection applications. The third sample set is constructed by domain classification processing, and the image feature differences of product appearance images belonging to different data domains in the third sample set are minimized. For example, minimizing the image feature differences can mean that product appearance images belonging to different data domains in the third sample set cannot be domain classified (cannot distinguish whether they are from the source data domain or the target data domain), and the defect detection network is optimized and trained using the third sample set, which can further improve the robustness and recognition accuracy of the defect detection network for detecting appearance defects of different types of products.
[0024] In one possible implementation, the network training method also includes: constructing a fourth sample set based on the first sample set and the second sample set, and processing the fourth sample set so that the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set is minimized; and using the fourth sample set to optimize the training of the defect detection network.
[0025] By adopting the above technical solution, the fourth sample set is constructed through the MMD method, so that the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set is minimized, and the defect detection network is optimized and trained using the fourth sample set, which can further improve the robustness and recognition accuracy of the defect detection network for different types of product appearance defect detection.
[0026] In one possible implementation, the training of the defect detection network includes a first loss function, a second loss function, a third loss function and a fourth loss function. The first loss function is a loss function for training the preset domain adaptive network using the first sample set, the second loss function is a loss function for training the preset domain adaptive network using the second sample set, the third loss function is a loss function for minimizing the image feature differences of product appearance images belonging to different data domains in the third sample set, and the fourth loss function is a loss function for minimizing the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set.
[0027] By adopting the above technical scheme, the total loss function of the defect detection network is configured to be composed of the first to fourth loss functions. The first loss function may be a loss function for training the domain adaptive network using the first sample set, the second loss function may be a loss function for training the preset domain adaptive network using the second sample set, or a loss function for training the preset domain adaptive network using the first sample set and the second sample set jointly. The third loss function may be a loss function that minimizes the image feature differences of product appearance images belonging to different data domains in the third sample set, and the fourth loss function is a loss function that minimizes the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set, thereby realizing the construction of domain adaptive product appearance defect detection capability, which can improve the robustness and recognition accuracy of the defect detection network for different types of product appearance defect detection.
[0028] In a possible implementation, a preset domain adaptive network includes a source domain network and a target domain network, the source domain network and the target domain network have the same network structure and share network parameters, and the preset domain adaptive network is trained using a first sample set, including: preprocessing the first sample set, and training the source domain network using the preprocessed first sample set, the preprocessing including at least one of grayscale conversion, image cutting and splicing, image data augmentation, and normalization processing; the preset domain adaptive network is trained using a second sample set, including: preprocessing the second sample set and the first sample set, and training the target domain network using the preprocessed second sample set and the first sample set.
[0029] By adopting the above technical scheme, before using the sample set to train the network, the sample set is preprocessed. The preprocessing includes but is not limited to grayscale conversion, image cutting and splicing, image data augmentation, and normalization processing. This can reduce the data processing amount of the network and improve the training efficiency and generalization ability of the network. In addition, the source domain network is trained using labeled training samples, the target domain network is trained using unlabeled training samples, or the target domain network is trained using unlabeled training samples and labeled training samples together. This can improve the robustness of the defect detection network for different types of product appearance defect detection.
[0030] In a possible implementation, both the source domain network and the target domain network include multiple defect detection heads, including an anchor-based detection head, an anchor-free detection head, and an anomaly detection head. The Anchor-based detection head is used to detect defect types whose length-to-width ratio is within a preset ratio range, the Anchor-free detection head is used to detect defect types whose length-to-width ratio is outside the preset ratio range, and the anomaly detection head is used to perform binary classification detection of the presence of defects and the absence of defects.
[0031] By adopting the above technical solution, by configuring multiple detection heads with different defect detection functions, it is possible to detect defects with different aspect ratios and new types of defects. For example, the network's defect detection head is designed to be based on Anchor-free and Anchor-based detection methods. While ensuring the detection accuracy of regular-sized defects, it is able to detect defects with abnormal aspect ratios, such as defects with an aspect ratio that is too large or too small. In addition, by setting a binary classification detection head for determining whether a defect exists or not, it is possible to detect new types of defects and reduce the risk of missed defect detection.
[0032] In a possible implementation, the preset domain adaptation network is trained using the second sample set, including: using the preset domain adaptation network trained with the first sample set to add a pseudo label to each training sample in the second sample set; obtaining the confidence of the pseudo label of each training sample in the second sample set, and setting the training weight of the corresponding training sample based on the confidence of the pseudo label; and training the preset domain adaptation network based on each training sample and the training weight of each training sample.
[0033] By adopting the above technical solution, by adding pseudo labels to unlabeled training samples, the network can be trained using a semi-supervised learning method, and based on the confidence of the pseudo labels, corresponding training weights are set for each training sample with added pseudo labels, so that training samples with high confidence have higher training weights, and training samples with low confidence have lower training weights, which can improve the training performance and recognition accuracy of the network.
[0034] In a possible implementation, the training weight of the corresponding training sample is set based on the confidence of the pseudo label, including: when the confidence of the pseudo label is greater than a preset threshold, the training weight of the corresponding training sample is set based on the confidence of the pseudo label.
[0035] By adopting the above technical solution, when the confidence of the pseudo-label is lower than the preset threshold, the training sample with the pseudo-label added is not used for network training. The preset threshold can be set according to the actual training needs. For pseudo-labels with confidence greater than or equal to the preset threshold, different training weights are configured for the corresponding unlabeled samples and used for network training, so that training samples with high confidence have higher training weights and training samples with low confidence have lower training weights, which can improve the training performance and recognition accuracy of the network. As the number of network training increases, the confidence of the pseudo-label added to the unlabeled training samples will change. The network can also be iterated multiple times by using unlabeled training samples to fully utilize the unlabeled training samples for network training.
[0036] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store computer-readable instructions; the processor is used to read the computer-readable instructions and implement the method provided by the first aspect and its optional implementation, or implement the method provided by the second aspect and its optional implementation.
[0037] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by a processor, the method provided by the first aspect and its optional implementation manner is implemented, or the method provided by the second aspect and its optional implementation manner is implemented.
[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer-readable instructions. When the computer-readable instructions are executed by a processor, the method provided by the first aspect and its optional implementation method is implemented, or the method provided by the second aspect and its optional implementation method is implemented.
[0039] The technical effects obtained by the above-mentioned third aspect, fourth aspect and fifth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect or the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 A schematic diagram of an artificial intelligence main framework provided for an embodiment of the present application;
[0042] Figure 2 A schematic diagram of a mid-frame defect detection system provided in an embodiment of the present application;
[0043] Figure 3-Figure 5 A schematic diagram of different types of defects in the middle frame provided in an embodiment of the present application;
[0044] Figure 6 A schematic diagram of an image at a corner of a middle frame captured by controlling a line scan camera to follow the movement of the middle frame based on an RTCP algorithm provided in an embodiment of the present application;
[0045] Figure 7 A schematic diagram of a process of controlling a line scan camera to follow the movement of a middle frame to be detected based on an RTCP algorithm provided in an embodiment of the present application;
[0046] Figure 8 A schematic diagram of cutting and splicing a middle frame image captured by a line drawing camera provided in an embodiment of the present application;
[0047] Fig. 9 A schematic diagram of the network structure of a defect detection network provided in an embodiment of the present application;
[0048] Fig.10 A schematic diagram of different types of defects that can be detected by an Anchor-free detection head and an Anchor-based detection head in a defect detection network provided in an embodiment of the present application;
[0049] Fig.11 A flowchart of a network training method provided in an embodiment of the present application;
[0050] Fig.12 A schematic diagram of a defect detection method according to an embodiment of the present invention;
[0051] Fig.13 A schematic diagram of the structure of a defect detection device provided in an embodiment of the present application;
[0052] Fig.14 A schematic diagram of the structure of a network training device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary", "or", "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "or", "for example" is intended to present related concepts in a concrete way.
[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. It should be understood that, unless otherwise specified in the present application, " / " means or. For example, A / B can represent A or B. "And / or" in the present application is only a kind of association relationship describing associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c seven situations. It should be understood that the order of the steps shown in the flowchart of this article can be changed, and some can also be omitted.
[0056] In order to facilitate understanding of the embodiments of the present application, first Figure 1 Describe the overall workflow of the artificial intelligence system. Figure 1 The figure shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is applicable to the needs of the general artificial intelligence field.
[0057] The following will explain the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).
[0058] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".
[0059] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.
[0060] (1) Infrastructure
[0061] The infrastructure provides computing power support for the AI system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system on the basic platform for calculation.
[0062] (2) Data
[0063] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data includes graphics, images, voice, text, IoT data of traditional devices, business data of existing systems, and perception data such as force, displacement, liquid level, temperature, and humidity.
[0064] (3) Data processing
[0065] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.
[0066] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0067] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0068] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
[0069] (4) General ability
[0070] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0071] (5) Smart products and industry applications
[0072] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.
[0073] During the production and processing of products, their surfaces are prone to scratches, bruises, bumps, particles and other defects. If the quality of the product appearance cannot be effectively tested, defective products will flow into the market. Existing product appearance testing generally uses manual visual inspection. Due to human conditions and limited energy, it is impossible to perform full-time testing or rapid testing. The efficiency and accuracy of testing are greatly affected by personal energy, physical strength and other physiological factors.
[0074] The following uses the middle frame of a terminal device as an example. The middle frame is the "skeleton" of the terminal device and can be used to carry various components of the terminal device. The middle frame is generally made of metal or plastic materials. The middle frame not only serves as a decoration, but is also a key structure that connects different parts of the device, helping to protect the internal components from external impacts. The shape, color, size, and material of the middle frame vary depending on the terminal device model and brand used. The middle frame is the hardest hit area for the appearance quality of the terminal device. During the production and processing process, various types of defects may exist. If a defective middle frame enters the market, it may cause customer complaints.
[0075] The existing middle frame inspection generally adopts the method of manual visual inspection, but the following problems are common in manual visual inspection: 1. Inspectors are prone to human errors. Different inspection standards, inspection experience, and inspection methods may lead to inconsistent qualified standards for middle frame inspection; 2. Inspectors are prone to inspection fatigue, especially when doing the same work for a long time, which is easy to cause visual fatigue, resulting in the inability to completely detect defective products; 3. Manual inspection is prone to missed inspections; 4. Manual inspection is relatively inefficient. Due to human conditions, it is impossible to perform full-time inspection or fast-rate inspection. The inspection efficiency is greatly affected by personal energy, physical strength and other physiological factors. If deep learning technology is used to train the defect detection network, a large number of labeled middle frame samples are required. However, it is difficult to collect a large number of defective middle frame samples from the production line, and the terminal equipment models processed on the production line are not fixed. When middle frames of different models or colors are switched on the production line, the detection accuracy of the defect detection network will drop significantly.
[0076] In view of this, an embodiment of the present application provides a defect detection method, which comprehensively collects product appearance data from different angles, uses a deep learning detection algorithm to detect product appearance defects of various shapes, and uses a domain adaptive learning algorithm to improve the robustness of cross-domain recognition, so that the defect detection network has high defect detection accuracy for product appearance inspections of different models and types.
[0077] The defect detection method implemented in this application can be applied to computer equipment, which can be computers (for example, industrial computers, personal computers, etc.), servers, server clusters, etc. Taking the defect detection method for detecting the appearance of the middle frame as an example, the middle frame can be the middle frame of a terminal device, for example, it can be the middle frame of a smart phone, a tablet computer, a smart watch, a wearable device, etc., and the embodiments of this application are not limited to this.
[0078] It can be understood that when the sample data used to train the defect detection network is not the appearance data of the middle frame, but the appearance data of other products (for example, car doors, battery covers, and auto parts), the defect detection method of the present application can also be applied to the appearance inspection of other products.
[0079] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.
[0080] (1) Neural Network
[0081] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:
[0082] ;
[0083] Where s=1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.
[0084] (2) Deep Neural Networks
[0085] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.
[0086] Although DNN looks complicated, the work of each layer can be represented by the following linear relationship expression: ,in, is the input vector, is the output vector, is the offset vector, is the weight matrix (also called coefficient), is the activation function. Each layer is just an input vector After processing, the output vector . Since DNN has many layers, the coefficient and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as .
[0087] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better depict complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).
[0088] (3) Convolutional Neural Network
[0089] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.
[0090] (4) Loss function
[0091] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.
[0092] (5) Domain Adaptive Learning
[0093] Domain adaptive learning is a method of transfer learning. Domain adaptation can be applied to solve the challenge of inconsistent distribution of training data sets, test data sets and real-world scenario data faced by neural network models. The idea of domain adaptive learning is to map data features from different fields to the same feature space, so that data from other fields can be used to enhance professional field training.
[0094] like Figure 2 FIG. 1 is a schematic diagram of the structure of a middle frame defect detection system 10 according to an embodiment of the present application.
[0095] The middle frame defect detection system 10 may include a middle frame loading module 101 , an image acquisition module 102 , a middle frame appearance detection module 103 , and a middle frame post-processing module 104 . For example, the middle frame loading module 101 can use a manipulator to grab the middle frame to be assembled and place it on the middle frame fixing piece, which is used to fix the middle frame. The image acquisition module 102 can use a motion control mechanism and multiple groups of cameras to realize multi-angle acquisition of the appearance data of the middle frame. For example, the motion control mechanism can control the rotation of the middle frame fixing piece and the camera. The image acquisition module 102 can transmit the collected middle frame appearance data to the middle frame appearance detection module 103 for data processing and recognition to determine whether there are defects in the appearance of the middle frame, such as determining whether there are scratches on the outer frame of the middle frame. The middle frame appearance detection module 103 can transmit the appearance detection result of the middle frame to the middle frame post-processing module 104. The middle frame post-processing module 104 can complete the subsequent operation of the middle frame according to the appearance detection result of the middle frame. For example, the defect-free middle frame is transmitted to the next process for subsequent assembly operation, while the defective middle frame is placed in a waste box to wait for subsequent manual processing.
[0096] In some embodiments, the middle frame defect detection system 10 may adopt a left-in-right-out layout to detect and transmit the middle frame. The middle frame loading module 101 can realize automatic loading and fixing of the middle frame. For example, the middle frame loading module 101 may include a manipulator and a middle frame fixing member, and the manipulator is used to grab the middle frame to be assembled from the upper frame of the conveyor belt and place it on the middle frame fixing member. The middle frame fixing member may be provided with a positioning table for fixing the middle frame. The embodiment of the present application does not limit the shape of the positioning table. For example, the positioning table is a trapezoidal positioning table. The positioning table can cooperate with the positioning holes of the middle frame to achieve high-precision positioning of the middle frame. The positioning holes of the middle frame may refer to screw holes, mounting holes, etc. provided on the middle frame itself. In order to prevent the middle frame fixing member from relative displacement with the middle frame during rotation, a vacuum suction cup may also be provided on the middle frame fixing member, and the vacuum suction cup is used to achieve double fixation of the middle frame.
[0097] In some embodiments, due to the various shapes of the appearance defects of the middle frame, some defects are in the shape of tiny dots, and some defects are in the shape of narrow strips. For small-sized middle frame defects, the defect features are not obvious, difficult to distinguish, and difficult to extract features in the image captured by a low-resolution camera, while a high-resolution line scan camera / camera can capture clear defect images, which is convenient for subsequent feature extraction and network training. The image acquisition module 102 may include one or more two-dimensional line scan cameras / cameras. Some scratches, abrasions and other defects need to be captured more clearly from different perspectives. Therefore, in order to improve the acquisition efficiency of the entire middle frame defect and the comprehensiveness of the defect scanning, the image acquisition module 102 may include at least three line scan cameras, and the three line scan cameras can be fixedly mounted on the camera fixing parts. The number of camera fixing parts can be one or equal to the number of line scan cameras. The installation position of the three line scan cameras can be set according to actual needs, and the embodiments of the present application are not limited to this. For example, one of the three line scan cameras is facing the outside of the frame of the center frame to shoot, and the other two line scan cameras are symmetrically arranged relative to the line scan camera, one is tilted at +30 degrees to shoot the outside of the frame of the center frame, and the other is tilted at -30 degrees to shoot the outside of the frame of the center frame, so that three center frame images of different angles can be shot at the same position of the center frame.
[0098] In some embodiments, in order to be able to capture appearance images of different positions of the middle frame from different angles, the image acquisition module 102 may also include a motion control device and a motion device. The motion control device is used to control the motion of the motion device. The motion dimension of the motion device can be selected according to actual needs. For example, it can be three-axis motion (X, Y, Z axis), four-axis motion (X, Y, Z axis + one rotation axis), and five-axis motion (X, Y, Z axis + two rotation axes). The motion device may include a motor / motor assembly, and the motion control device may include a motor controller / motor controller. The motion control device and the motion device may adopt existing motion control devices and motion devices. The motion device is used to drive the middle frame fixing member and the camera fixing member to move, so that the middle frame and the line scan camera can move accordingly.
[0099] In some embodiments, since the middle frame is generally exposed to the user's field of vision on the outside of the four borders after the product is assembled, the image acquisition module 102 can only capture images of the outside of the four borders of the middle frame for inspection, thereby improving the image acquisition and defect detection speed of the middle frame.
[0100] For example, the defect image of the middle frame captured by the line scan camera is as follows: Figure 3-Figure 5 As shown, Figure 3 It indicates that there are dot-shaped protrusion defects in the middle frame. Figure 4 It indicates that there are scratches on the middle frame. Figure 5 It shows that there are jagged edges on the middle frame.
[0101] In some embodiments, since the focal length of a two-dimensional line scan camera is generally relatively small, defocusing is prone to occur when capturing images at the corners of the center frame. In order to reduce the probability of defocusing, the existing motion control algorithm requires relatively high motion control of the motion device. In order to solve this problem, the embodiment of the present application applies a rotated tool center point (RTCP) algorithm to the linkage motion control of the line scan camera and the center frame, and combines the five-axis linkage technology to improve the following stability of the line scan camera relative to the center frame and the real-time performance of image acquisition, thereby capturing a clear image at the corner of the center frame. Motion control based on the RTCP algorithm allows the line scan camera to always maintain a distance from the center frame within the focal length range of the camera during complex movements, which can improve the image acquisition clarity of the line scan camera and the accuracy of the captured image data. For example, the image at the corner of the center frame captured by the line scan camera based on the RTCP algorithm is as follows: Figure 6 shown.
[0102] The existing RTCP algorithm is mainly used to accurately control the tool tip center point of a five-axis machine tool. In the embodiment of the present application, the image acquisition endpoint of the line scan camera is equated with the tool tip following point in the RTCP algorithm, the product to be inspected (for example, the middle frame) is equated with the processed product, and the point on the product to be inspected that is directly opposite to the image acquisition endpoint is the point to be photographed (equivalent to the actual contact point between the tool tip and the surface of the processed product). The rotation axis of the line scan camera and the rotation axis of the product to be inspected are equivalent to the two rotation axes of the five-axis machine tool. By calibrating the rotation center of the rotation axis of the product to be inspected in the camera coordinate system, the coordinate transformation of the motion trajectory coordinates of the product to be inspected is performed based on the rotation center in the camera coordinate system, and an encoder control signal is generated based on the coordinate transformation result to control the encoder to output a pulse signal. The motion device realizes the interpolation motion of the line scan camera based on the pulse signal. The line scan camera can also perform image acquisition based on the pulse signal, so that the distance between the line scan camera and the middle frame is always kept within the focal length range of the camera (which can be achieved by maintaining the distance between the image acquisition endpoint and the point to be photographed unchanged, or keeping it within a preset distance), which can improve the following stability and acquisition real-time performance of the line scan camera for the product to be inspected.
[0103] like Figure 7 As shown, it is a flow chart of controlling the motion of the line scan camera based on the RTCP algorithm.
[0104] S41, setting the line scan camera and the middle frame to be detected to a linkage motion mode, and setting the motion speed of the middle frame to be detected.
[0105] In some implementations, the linkage motion mode may mean that the line scan camera can interpolate the motion of the middle frame to be detected. The motion speed of the middle frame to be detected can be set according to actual image acquisition requirements, and the embodiment of the present application does not limit this.
[0106] S42, turning on the line scan camera so that the line scan camera can receive the pulse signal and perform image acquisition.
[0107] S43, acquiring the position and vector of the point to be photographed, and obtaining the motion trajectory coordinates of the point to be photographed.
[0108] After the position and vector of the point to be photographed are acquired, the motion trajectory coordinates of the point to be photographed may be acquired based on the position and vector of the point to be photographed.
[0109] In some embodiments, after the middle frame is fixed, a middle frame coordinate system can be constructed based on the rotation axis of the middle frame to obtain the position and vector of the point to be photographed, and the motion trajectory coordinates of the point to be photographed can be the motion trajectory coordinates in the middle frame coordinate system.
[0110] S44, converting the motion trajectory coordinates of the point to be photographed into the camera coordinate system to obtain the target trajectory coordinates.
[0111] In some embodiments, a camera coordinate system corresponding to the line scan camera is constructed, and then the motion trajectory coordinates of the point to be photographed are converted to the camera coordinate system to obtain the target trajectory coordinates.
[0112] S45, performing interpolation operation on the image acquisition endpoints of the line scan camera based on the RTCP algorithm and the target trajectory coordinates.
[0113] S46, outputting an interpolation control instruction for controlling the axis where the line scan camera is located to perform interpolation motion according to the interpolation operation result.
[0114] S47, outputting a pulse signal according to the interpolation control instruction, the pulse signal is used to make the axis where the line scan camera is located perform interpolation motion and perform image acquisition.
[0115] In some embodiments, the pulse signal can be used to drive the motion device, so that the motion device drives the axis where the line scan camera is located to perform interpolation motion, thereby realizing the interpolation motion of the line scan camera. The line scan camera can also perform image acquisition based on the pulse signal, so that the line scan camera can always perform image acquisition of the center frame within the focal length range and ensure the real-time performance of image acquisition.
[0116] In some embodiments, the middle frame appearance inspection module 103 may include a defect recognition device that is communicatively connected to the line scan camera. For example, the defect recognition device may be a computer device, or a module including a processor and a memory. The defect recognition device may be deployed with a defect detection network for performing middle frame defect detection.
[0117] In some embodiments, the middle frame appearance detection module 103 can be divided into three functional units according to the process of the middle frame appearance detection module 103 processing the middle frame image collected by the image acquisition module 102: a data preprocessing unit, a defect detection unit based on a defect detection network, and a detection result post-processing unit. The data preprocessing unit can be used to preprocess the middle frame image, and the preprocessing may include grayscale conversion, image cutting and splicing, image data augmentation, normalization processing, etc.
[0118] For example, the size of the middle frame image captured by the line scan camera is relatively large. For example, the resolution of a single long side image of the middle frame of a mobile phone can generally reach 30000*4096 pixels. Such a large image cannot be directly input into the defect detection network for recognition. If the image is scaled and then input into the defect detection network, it will affect the recognition accuracy of small-scale defects. The image cutting and splicing of the embodiment of the present application may include intercepting the area of interest of the middle frame image, removing the pixel area other than the middle frame, and then cutting the intercepted image into multiple middle frame image blocks along the length direction according to a certain overlapping ratio, and then splicing the cut middle frame image blocks together in the width direction. The pixel size for cutting and splicing the middle frame image can be set according to the image input size supported by the detection network, and the embodiment of the present application is not limited to this.
[0119] like Figure 8 As shown, it is illustrated that the image I1 of the line scan camera is cut into multiple middle frame image blocks I2 along the long side according to a certain overlap ratio. The middle frame image blocks I2 can have the same length and are spliced in the width direction to obtain the processing effect of image I3. By cutting and splicing the middle frame image collected by the line scan camera, on the one hand, useless image pixels can be removed, the data processing amount and noise interference of the network can be reduced, and the processing speed and recognition accuracy of the network can be improved. On the other hand, splicing multiple middle frame image blocks together can further improve the recognition speed of the network compared to inputting a single middle frame image block into the network for recognition.
[0120] In some embodiments, the image data augmentation methods used in the embodiments of the present application may include but are not limited to geometric transformation, color gamut transformation, noise injection, cross-domain image generation and other data augmentation methods. Cross-domain image generation may refer to the use of existing image generation algorithms to generate cross-domain middle frame images of different scenes / different products to increase the diversity of training data and the scale of data training, so as to improve the domain adaptation ability of the network from the data level. Middle frame images of different scenes may refer to middle frame images with different background colors, different ambient light intensities, different contrasts, etc.
[0121] In some embodiments, the middle frame image captured by the line scan camera is generally a color image (for example, an RGB image). In order to reduce the amount of data processing of the network, the middle frame image input to the network for recognition is a grayscale image. Grayscale conversion may refer to converting the color middle frame image captured by the line scan camera into a grayscale middle frame image. Normalization processing may refer to mapping the 0-255 grayscale value to the range of 0-1 to simplify the data processing process and improve the efficiency and accuracy of network training.
[0122] The defect detection unit uses the defect detection network to extract features and detect defects on the preprocessed middle frame image. Fig. 9 As shown, the defect detection network includes two sub-networks (hereinafter referred to as the source domain network and the target domain network). The source domain network and the target domain network are twin neural networks, that is, the source domain network and the target domain network have the same network structure and share network parameters. For example, the source domain network and the target domain network share weights with each other.
[0123] The source domain network is generally a network trained based on labeled data in the source data domain (e.g., labeled bounding box images of product A). After the source domain network is trained, the source domain network can be used to detect defects in the bounding box images of product A. The label can include information on whether a defect exists, and the specific defect type information if a defect exists.
[0124] The target domain network is generally a network trained based on unlabeled data of the target data domain (e.g., unlabeled border images of product B), or a network jointly trained based on unlabeled data of the target data domain and labeled data of the source data domain, so as to perform defect detection on the border images of product B later. The number of samples in the target data domain can be much smaller than the number of samples in the source data domain, so that the defect detection network can have the ability to perform defect detection on the border images of product A and product B. The source data domain and the target data domain can be defined according to actual needs, and the embodiments of the present application do not limit this. Generally speaking, the source data domain and the target data domain involve product images of different models, different brands, or different series.
[0125] Similarly, the target data domain of product C can be constructed, and the target domain network can be trained using the target data domain of product C, so that the defect detection network can have the ability to detect defects in the border images of products A, B, and C.
[0126] Since the source domain network and the target domain network have the same network structure, the following only illustrates the network structure of the source domain network. The source domain network may include a feature extraction layer and a defect detection layer. The feature extraction layer may include multiple network layers Conv1~ConvN for image feature extraction, where N is a positive integer greater than 1. The image features extracted by the feature extraction layer include but are not limited to texture features, brightness features, contrast features, grayscale features, etc. Each network layer Conv1~ConvN can perform operations such as convolution, batch normalization (BN), pooling, and nonlinear activation. For example, nonlinear activation operations are performed based on nonlinear activation functions: Sigmoid function, tanh function, ReLU function, Leaky ReLU function, etc.
[0127] The defect detection layer may include multiple defect detection heads, each of which may include a convolutional layer and a fully connected layer, or only a fully connected layer, and each defect detection head may be constructed based on network features of different levels and / or different scales to achieve corresponding defect detection functions. The number of defect detection heads may be set according to the actual mid-frame detection requirements, and the embodiments of the present application are not limited to this. Fig. 9 Three defect detection heads are illustrated for illustration: Anchor-based detection head, Anchor-free detection head and Anomaly detection head. The Anchor-based detection head and the Anchor-free detection head are constructed based on the CNN network, that is, both the Anchor-based detection head and the Anchor-free detection head can include multiple convolutional layers, pooling layers, fully connected layers, etc., and the Anomaly detection head can include multiple fully connected layers.
[0128] The anchor-based detection head creates a series of anchor boxes with different sizes and aspect ratios in an explicit or implicit way, and then classifies or regresses the content in the anchor box. It is suitable for the detection of mid-frame defects with moderate aspect ratios (such as point-shaped convex defects), for example, the detection of mid-frame defects with aspect ratios between 0.5 and 2. The anchor-free detection head predicts the probability that each pixel in the image belongs to the object to be detected and the bounding box information of the object, and then generates a bounding box based on this information, and classifies or regresses the content in the bounding box. It is suitable for the detection of mid-frame defects with unbalanced aspect ratios (such as narrow and long scratches), for example, the detection of mid-frame defects with aspect ratios less than 0.5 or greater than 2. The anomaly detection head is used for binary classification training to identify whether there are defects in the mid-frame image, thereby reducing the risk of missing new types of mid-frame defects in the detection network.
[0129] It is understandable that different defect detection heads may be set according to other defect types, sizes or detection methods. The embodiments of the present application are not limited to setting Anchor-based detection heads, Anchor-free detection heads and Anomaly detection heads.
[0130] like Fig.10 As shown in (a) and (b) in FIG. 1 , the narrow and long middle frame defect detected by the anchor-free detection head is shown in FIG. Fig.10 As shown in (c) in the figure, it illustrates the mid-frame defect with a moderate aspect ratio detected by the Anchor-based detection head.
[0131] The defect detection network of the embodiment of the present application can identify the middle frame defect type learned in the training stage through the Anchor-based detection head and the Anchor-free detection head, that is, the Anchor-based detection head and the Anchor-free detection head output the specific defect type existing in the middle frame image; the middle frame defect type not learned in the training stage can be identified through the Anomaly detection head, that is, for a new type of middle frame defect, although the Anchor-based detection head and the Anchor-free detection head cannot identify it, the Anomaly detection head can identify that there is a defect in the middle frame image (the defect type is unknown). In this case, it can be manually checked, and when it is determined to be a new type of middle frame defect, a label is added to the middle frame image. The labeled middle frame image can be input into the defect detection network for training, so that this type of middle frame defect can be subsequently recognized by the Anchor-based detection head and / or the Anchor-free detection head.
[0132] In some embodiments, the training of the defect detection network may include: training the source domain network based on the training samples of the source data domain with labels, and training the target domain network based on the training samples of the target data domain without labels (or jointly training the target domain network based on the training samples of the target data domain without labels and the training samples of the source data domain with labels). In the network training process, the maximum mean discrepancy (MMD) algorithm and the domain classification algorithm are used at the feature extraction level to reduce the feature difference between the source data domain and the target data domain, so as to further improve the robustness of the cross-scenario deployment of the defect detection network.
[0133] For example, assuming that a batch of training samples includes m labeled training samples (from the source data domain) and n unlabeled training samples (from the target data domain), m and n are positive integers, and the values of m and n can be equal or unequal. A loss function can be constructed based on the MMD algorithm to process the training samples in the batch to minimize the difference between the feature mean of the m labeled training samples and the feature mean of the n unlabeled training samples, thereby reducing the feature difference between the source data domain and the target data domain. After the batch of training samples is processed by minimizing the mean, it can be used for network optimization training. Minimizing the difference in feature mean based on the MMD algorithm can refer to minimizing the difference in defect feature mean / image feature mean of the same defect type, or minimizing the difference in defect feature mean / image feature mean of training samples randomly selected from the source data domain and the target data domain.
[0134] For example, a domain classification algorithm may refer to making it impossible to distinguish training samples from the source data domain from training samples from the target data domain. Assuming that a batch of training samples includes m labeled training samples (from the source data domain) and n unlabeled training samples (from the target data domain), each training sample in the batch can be domain classified (i.e., it can be determined whether it belongs to the source data domain or the target data domain). By constructing a loss function, the training samples in the batch can be processed to minimize the differences between training samples in different data domains. For example, the training samples in the batch can be processed through a fully connected layer so that the training samples in the batch cannot be domain classified, that is, the training samples in the batch cannot be classified into the source data domain and the target data domain, thereby reducing the feature differences between the source data domain and the target data domain. After being processed by the domain classification algorithm, the training samples in the batch can be used for network optimization training.
[0135] For example, a loss function can be constructed to measure the difference in defect features or image features of training samples from different domains (source data domain or target data domain) that contain the same defect type. By minimizing the difference, these training samples containing the same defect type cannot be domain classified and are then used to optimize the network training. Alternatively, the training samples randomly selected from the source data domain and the target data domain can be processed to prevent domain classification.
[0136] In some embodiments, the target domain network may be trained using training samples of the target data domain without labels by using the target domain network to create pseudo labels for each training sample without labels, and then training the target domain network based on the training samples with pseudo labels added.
[0137] In order to improve the training performance and accuracy of the target domain network, corresponding training weights can also be set for each training sample with a pseudo-label added based on the confidence of the pseudo-label, so that training samples with high confidence have higher training weights, and training samples with low confidence have lower training weights. For example, an unlabeled training sample P1 is input into the target domain network to obtain a detection result and a confidence level. The detection result can be added to the training sample P1 as a pseudo-label, and the confidence level is used as the confidence level of the pseudo-label. The higher the confidence level, the higher the training weight of the training sample P1 for the target domain network, and the lower the confidence level, the lower the training weight of the training sample P1 for the target domain network. The conversion coefficient between confidence level and training weight can be set according to actual needs, and the embodiments of the present application are not limited to this.
[0138] In some embodiments, a preset threshold value may also be set. When the confidence of the pseudo-label is lower than the preset threshold value, the training sample with the pseudo-label added is not used for network training. The preset threshold value may be set according to the actual training requirements, and the embodiments of the present application are not limited to this. For example, the preset threshold value is 0.3, that is, for training samples with pseudo-labels with a confidence lower than 0.3, they are not used for training the target domain network. For training samples with pseudo-labels with a confidence greater than or equal to 0.3, different training weights are configured and used for training the target domain network. As the number of training times of the target domain network increases, the confidence of the pseudo-label added to the unlabeled training samples will change. The training samples of the target data domain may be used to perform multiple iterative training on the target domain network, so as to fully utilize the training samples of the target data domain to train the target domain network.
[0139] In some embodiments, the detection result post-processing unit can be used to combine the detection results of each defect detection head to obtain the final detection result of the middle frame, for example, combining the detection results of the anchor-based detection head, the anchor-free detection head and the anomaly detection head. For detection results with a high confidence level, the final detection result of the middle frame can be obtained by taking the union method. For ambiguous detection results, the acquisition parameters of the image acquisition module 102 can be adjusted (for example, adjusting the illumination intensity of the lighting source, the scanning angle of the line scan camera, etc.) to perform secondary image acquisition and detection.
[0140] For example, each defect detection head can output the detection result and the confidence of the detection result. When the confidence of the detection result of one or more defect detection heads is greater than the preset confidence, it indicates that the defect detection network has detected a credible defect. The final detection result of the middle frame image can be obtained by taking the union method, that is, the final detection result is the union of the detection results of all defect detection heads. For example, when the confidence of the detection results of all defect detection heads is less than the preset confidence, it indicates that the defect detection network has not detected a credible defect. The final detection result of the middle frame image can be directly determined as no defect, that is, there is no need to take the union. The value of the preset confidence can be set according to actual needs, and the embodiment of the present application is not limited to this. For example, the confidence range of the defect detection head output is 0-1, and the preset confidence can be set to 0.6.
[0141] For example, for the Anchor-based detection head, the output detection result is: XX defect, confidence level 0.7, indicating that the Anchor-based detection head has detected a reliable XX defect. For the Anomaly detection head, if the output detection result is: defect exists, confidence level 0.7, it indicates that the Anomaly detection head has determined that the product to be detected has a defect, and the result is reliable. If the output detection result is: defect exists, confidence level 0.4, it indicates that the Anomaly detection head's determination that the product to be detected has a defect is not reliable. Based on this result, it can be considered that the Anomaly detection head indicates that the product to be detected does not have a defect.
[0142] For example, the defect detection network may also be configured with a first preset confidence and a second preset confidence, and the first preset confidence is greater than the second preset confidence. In the case where the confidence of the detection result of at least one defect detection head is greater than or equal to the first preset confidence, it indicates that the defect detection network detects that there are credible defects in the product to be detected, and the detection result of the product to be detected can be determined to be defective, and the union of the detection results of each defect detection head can be used as the final detection result of the product to be detected. In the case where the confidence of the detection result of each defect detection head is less than the second preset confidence, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product to be detected can be determined to be defective. In the case where the confidence of the detection result of at least one defect detection head is less than the first preset confidence and greater than or equal to the second preset confidence, and there is no case where the confidence of the detection result of any defect detection head is greater than or equal to the first preset confidence, it indicates that the defect detection network cannot determine whether there are defects in the product appearance, and the acquisition parameters of the image acquisition module 102 can be adjusted to perform image acquisition and detection on the center frame again. The values of the first preset confidence and the second preset confidence can be set according to actual needs, and the embodiment of the present application does not limit this. For example, the confidence range of the defect detection head output is 0-1, the first preset confidence can be set to 0.8, and the second preset confidence can be set to 0.3.
[0143] In some embodiments, the middle frame post-processing module 104 is used to perform a blanking operation on the middle frame according to the final inspection result. For example, the middle frame post-processing module 104 may include a manipulator, and the middle frame post-processing module 104 controls the manipulator to complete the blanking of the middle frame according to the final inspection result of the middle frame. For a defect-free middle frame, the manipulator puts the middle frame into the guide rail to transfer it to the next assembly station, for example, the assembly station of the middle frame and the touch screen; for a defective middle frame, the manipulator places the middle frame into a waste bin and waits for the operator to process it, for example, to verify and repair defects.
[0144] In some embodiments, for the middle frames in the waste bin that are verified by the operator as not actually defective, labeled training samples can be constructed based on these middle frames and input into the defect detection network again for optimization training.
[0145] The defect detection network involved in the embodiment of the present application includes a training phase and an inference phase. The training phase and the inference phase are described in detail below.
[0146] 1. Training Phase
[0147] In the embodiment of the present application, the training phase may refer to the process in which the network training device uses the training data to train the domain adaptive network constructed based on the neural network to obtain the defect detection network. The domain adaptive network may include a source domain network and a target domain network. The source domain network and the target domain network have the same network structure and share network parameters, for example, the source domain network and the target domain network share weights. The network training device may be a server, a computer, or a server cluster, etc., which is not limited in the embodiment of the present application.
[0148] like Fig.11 As shown, a network training method provided in an embodiment of the present application is provided. The method can be applied to a network training device. The sequence of steps in the method can be adjusted according to actual needs, and the embodiment of the present application does not limit this. The method includes the following steps:
[0149] Step S81: Obtain first training data with labels, and use the first training data to train a domain adaptation network.
[0150] In some embodiments, the first training data may be pre-stored in a designated storage area / storage device. For example, the first training data may be a border image of product P1, and the first training data may include a plurality of defective border images and a plurality of non-defective border images, and labels are added. The first training data may be used to train the source domain network in the domain adaptive network, and the first training data may be divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used for network training, and the validation set is used to validate the network trained with the training set, and then the hyperparameters of the network may be adjusted based on the validation results, and the test set is used to test the trained network.
[0151] In some embodiments, after acquiring the first training data, the first training data may be subjected to a first preprocessing. For example, the first preprocessing includes image data augmentation and normalization. The preprocessed first data is then divided into a training set, a validation set, and a test set, so that the trained network has stronger generalization capabilities.
[0152] Step S82, obtaining second training data without labels, and adding pseudo labels to the second training data using a domain adaptation network trained with the first training data, wherein the second training data and the first training data belong to different data domains.
[0153] For example, the first training data is a training sample of a source data domain, and the second training data is a training sample of a target data domain.
[0154] In some embodiments, the second training data is training data without labels, and the second training data may be a frame image of product P2 without labels. The amount of data of the second training data may be much smaller than the amount of data of the first training data. Adding pseudo labels to the second training data using a domain adaptation network trained with the first training data may refer to: inputting each frame image contained in the second training data into a target domain network in the domain adaptation network, and adding the detection results of each frame image obtained by the target domain network as labels to the corresponding frame image.
[0155] Step S83: training the domain adaptation network based on the second training data with pseudo labels added.
[0156] In some embodiments, after adding pseudo labels to the second training data, the domain adaptation network can be trained based on the second training data with pseudo labels added, for example, the target domain network in the domain adaptation network is trained so that the trained domain adaptation network can simultaneously have the border defect detection function of products P1 and P2. The trained domain adaptation network can be used as a defect detection network in the inference stage.
[0157] For example, corresponding training weights may be set for each training sample with a pseudo label added based on the confidence of the pseudo label, so that training samples with high confidence have higher training weights and training samples with low confidence have lower training weights.
[0158] For example, a preset threshold may be set. When the confidence of the pseudo-label is lower than the preset threshold, the training sample with the pseudo-label added is not used for network training. When the confidence of the pseudo-label is greater than or equal to the preset threshold, corresponding training weights are set for each training sample with the pseudo-label added for network training.
[0159] In some embodiments, after adding pseudo labels to the second training data, the target domain network can be trained in combination with the first training data and the second training data with pseudo labels added, so that the trained domain adaptive network can simultaneously have the border defect detection function of products P1 and P2. The trained domain adaptive network can be used as a defect detection network in the inference stage.
[0160] Before using the second training data with pseudo labels added to train the target domain network, or before using the first training data and the second training data with pseudo labels added to train the target domain network, the second training data with pseudo labels added may be subjected to a first preprocessing, or the first training data and the second training data with pseudo labels added may be subjected to a first preprocessing. For example, the first preprocessing includes image data augmentation and normalization, and then the training set, validation set, and test set are divided, so that the trained network has a stronger generalization ability.
[0161] In some embodiments, the network training method may further include step S84, which can reduce the feature difference between the source data domain and the target data domain and enhance the cross-domain adaptability of the network.
[0162] Step S84, constructing third training data including training samples of different data domains, and optimizing the domain adaptive network using the third training data, wherein the training samples belonging to different data domains in the third training data meet the first preset condition.
[0163] In some embodiments, the third training data may include multiple training samples from the source data domain and multiple training samples from the target data domain. The first preset condition may be to minimize the feature mean of the training samples in the source data domain and the feature mean of the training samples in the target data domain based on the MMD algorithm, and then use the third training data to optimize the training of the domain adaptive network, for example, use the third training data to optimize the training of the target domain network in the domain adaptive network. The trained domain adaptive network can be used as a defect detection network in the inference stage.
[0164] In some embodiments, the network training method may further include step S85, which can further reduce the feature difference between the source data domain and the target data domain and enhance the cross-domain adaptability of the network.
[0165] Step S85, constructing fourth training data including training samples of different data domains, and optimizing the domain adaptive network using the fourth training data, wherein the training samples belonging to different data domains in the fourth training data meet the second preset condition.
[0166] In some embodiments, the fourth training data may include multiple training samples from the source data domain and multiple training samples from the target data domain. The second preset condition may be that the training samples of the source data domain and the training samples of the target data domain cannot be distinguished based on the domain classification algorithm (the source data domain and the target data domain cannot be classified), and then the fourth training data is used to optimize the training of the domain adaptive network. For example, the fourth training data is used to optimize the training of the target domain network in the domain adaptive network. The trained domain adaptive network can be used as a defect detection network in the reasoning stage.
[0167] In some embodiments, for optimizing the training of the domain adaptation network, one of steps S84 and S85 may be executed, or both steps may be executed. Steps S84 and S85 may be executed in parallel, or step S84 may be executed before or after step S85.
[0168] In some embodiments, a first loss function can be constructed in the process of training the source domain network using the first training data. A second loss function can be constructed in the process of training the target domain network using the second training data, or in the process of training the target domain network using the first training data and the second training data. At the same time, in order to reduce the feature difference between the source data domain and the target data domain, a third loss function can be constructed based on the MMD algorithm to minimize the difference between the feature mean of the training samples of the source data domain in the batch and the feature mean of the training samples of the target data domain. At the same time, a fourth loss function can be constructed based on the domain classification algorithm so that the training samples of the source data domain in the batch and the training samples of the target data domain cannot be domain classified. The total loss function of the training defect detection network can be composed of the above-mentioned first to fourth loss functions.
[0169] In some embodiments, the construction of the first loss function to the fourth loss function can be set according to the actual network training requirements, and the embodiments of the present application are not limited to this. For example, the first loss function, the second loss function and the fourth loss function can be loss functions based on logistic regression, and the third loss function can be an MMD loss function.
[0170] In some embodiments, the defect detection network may include multiple defect detection heads, and each defect detection head may be constructed based on network features of different levels and / or different scales. For example, the source domain network and the target domain network in the defect detection network both include Anchor-based detection heads, Anchor-free detection heads, and Anomaly detection heads. The Anchor-based detection head is suitable for detecting middle frame defects with a moderate aspect ratio. The Anchor-free detection head is suitable for detecting middle frame defects with an unbalanced aspect ratio. The Anomaly detection head is used for binary classification training to identify whether there are defects in the middle frame image, thereby reducing the risk of missing new types of middle frame defects in the detection network.
[0171] The defect detection network can identify the middle frame defect types learned in the training phase through the Anchor-based detection head and the Anchor-free detection head, that is, the outputs of the Anchor-based detection head and the Anchor-free detection head are the specific defect types existing in the middle frame image; the middle frame defect types not learned in the training phase can be identified through the Anomaly detection head, that is, for a new type of middle frame defect, although the Anchor-based detection head and the Anchor-free detection head cannot identify it, the Anomaly detection head can identify that there is a defect in the middle frame image (the defect type is unknown). In this case, it can be manually checked, and if it is determined to be a new type of middle frame defect, a label can be added to the middle frame image. The labeled middle frame image can be input into the defect detection network for training, so that this type of middle frame defect can be subsequently identified by the Anchor-based detection head and / or the Anchor-free detection head.
[0172] After the above training phase, the network training device can be deployed on the defect detection device on the product assembly line. The defect detection device can be a computer (for example, a personal computer, an industrial computer, etc.), a server, and other devices. The network training device and the defect detection device can be the same device or different devices, and the embodiment of the present application does not limit this. The following describes the process of network application, that is, the process of the defect detection device performing reasoning based on the defect detection network.
[0173] 2. Reasoning Stage
[0174] like Fig.12 As shown, a defect detection method provided by an embodiment of the present application, the order of steps in the method can be adjusted according to actual needs, and the embodiment of the present application is not limited to this, the method can be applied to a defect detection device, and the defect detection device can be deployed with a defect detection network trained by the above network training method, the method includes the following steps:
[0175] Step S91, capturing an image of the frame to be detected.
[0176] For example, the image acquisition module 102 described above may be used to acquire the image of the frame to be detected.
[0177] Step S92, inputting the collected image into the defect detection network.
[0178] In some embodiments, since the image size captured by the line scan camera is large, the captured image can be preprocessed before being input into the defect detection network. The preprocessing includes but is not limited to grayscale conversion processing and image cutting and stitching processing. The image cutting and stitching processing may refer to cutting the captured image into multiple sub-images in the length direction of the image, and stitching at least two of the multiple sub-images in the width direction, and then inputting the stitched image into the defect detection network.
[0179] The defect detection network can extract image features and identify defects to obtain the detection results of the middle frame to be detected. The detection results output by the defect detection network can be of two types. The first type can be that the defect detection network can determine whether there are defects in the middle frame to be detected. The detection results may include three types: no defects; defects exist, and the defect type is XX; defects exist, and the defect type is unknown. The defect detection network can output different vectors to represent different detection results. For example, the situation of no defects can be that the Anomaly detection head outputs a vector mark that there are no defects, and the Anchor-based detection head and the Anchor-free detection head both output vector marks that no defects are detected. For example, the situation of defects and the defect type is XX can be that the Anomaly detection head outputs a vector mark that there are defects, and the Anchor-based detection head and / or the Anchor-free detection head outputs a vector mark of the detected defect type. For example, the situation of defects and the defect type is unknown can be that the Anomaly detection head outputs a vector mark that there are defects, and the Anchor-based detection head and the Anchor-free detection head both output vector marks that no defects are detected.
[0180] Another type is that the defect detection network cannot determine whether there is a defect in the middle frame image. This type of detection result can be identified by a preset vector output by the network, indicating that the acquisition parameters of the image acquisition module 102 need to be adjusted (for example, adjusting the illumination intensity of the lighting source, the scanning angle of the line scan camera, etc.) to perform secondary image acquisition and recognition on the middle frame to be inspected.
[0181] Step S93, obtaining the detection result of each defect detection head among the multiple defect detection heads for the image of the middle frame to be detected, and taking the union of the detection results of each defect detection head as the detection result of the middle frame to be detected.
[0182] In some embodiments, after the captured image is input into the defect detection network, each defect detection head of the defect detection network may output a corresponding detection result for the input image.
[0183] In some embodiments, after the captured image is input into the defect detection network, each defect detection head of the defect detection network may output a corresponding detection result and a confidence level of the detection result for the input image.
[0184] In some embodiments, when the confidence level of the detection result of at least one defect detection head is greater than or equal to a preset confidence level, indicating that the defect detection network has detected a credible defect, the union of the detection results of each defect detection head can be used as the detection result of the middle frame to be detected. For example, the defect detection network can calculate the union and output the detection result of each defect detection head, or a computer device can calculate the union and output the detection result of each defect detection head.
[0185] In some embodiments, when the confidence of the detection results of each defect detection head is less than a preset confidence, it indicates that the defect detection network has not detected a credible defect, and the defect detection network can directly determine that the final detection result of the middle frame image is that there is no defect.
[0186] In some embodiments, the defect detection network may also be configured with a first preset confidence and a second preset confidence, and the first preset confidence is greater than the second preset confidence. In the case where the confidence of the detection result of at least one defect detection head is greater than or equal to the first preset confidence, it indicates that the defect detection network has detected that there are credible defects in the product to be detected, and the union of the detection results of each defect detection head can be used as the final detection result of the product to be detected. In the case where the confidence of the detection result of each defect detection head is less than the second preset confidence, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product to be detected can be directly determined as no defect. When the confidence level of the detection result of at least one defect detection head is less than the first preset confidence level and greater than or equal to the second preset confidence level, and there is no defect detection head whose confidence level of the detection result is greater than or equal to the first preset confidence level, it indicates that the defect detection network cannot determine whether there is a defect in the product appearance. The defect detection network can output vector identification information of a detection abnormality, and the computer device can output preset detection abnormality prompt information based on the vector identification information. For example, the preset detection abnormality prompt information is "It is impossible to determine whether there is a defect in the product appearance, please perform secondary image acquisition and inspection". The staff can adjust the acquisition parameters of the image acquisition device (for example, the shooting angle of the line scan camera, the brightness of the ambient light source, etc.) to perform image acquisition and inspection on the product to be inspected again. The value of the preset confidence level can be set according to actual needs.
[0187] In some embodiments, after obtaining the inspection results of the middle frame to be inspected, different processing methods can be performed based on the inspection results of the middle frame to be inspected. For example, a middle frame without defects can be transferred to the next assembly station; for a middle frame with defects, the middle frame can be placed in a waste bin and wait for the operator to process it, for example, to perform defect verification and defect repair.
[0188] In some embodiments, for the middle frames in the waste bin that are verified by the operator as not actually having defects, labeled training samples can be constructed based on these middle frames to optimize the defect detection network.
[0189] The present application also provides a defect detection device. Fig.13 , Fig.13 A structural diagram of a defect detection device provided in an embodiment of the present application, wherein the defect detection network described in the above embodiment may be deployed on the defect detection device 100. Specifically, the defect detection device 100 includes: a receiver 111, a transmitter 112, a processor 113 and a memory 114 (wherein the number of processors 113 in the defect detection device 100 may be one or more, Fig.13In the example of FIG. 1 , the processor 113 may include an application processor 1131 and a communication processor 1132. In some embodiments of the present application, the receiver 111, the transmitter 112, the processor 113 and the memory 114 may be connected via a bus or other means.
[0190] The memory 114 may include a read-only memory and a random access memory, and provides instructions and data to the processor 113. A portion of the memory 114 may also include a non-volatile random access memory (NVRAM). The memory 114 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.
[0191] The processor 113 controls the operation of the defect detection device. In a specific application, the various components of the defect detection device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.
[0192] The above embodiments of the present application Fig.12The disclosed method can be applied to the processor 113, or implemented by the processor 113. The processor 113 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 113. The above processor 113 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 113 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 114, and the processor 113 reads the information in the memory 114 and completes the steps of the above method in combination with its hardware.
[0193] The receiver 111 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the defect detection device. The transmitter 112 can be used to output digital or character information through the first interface; the transmitter 112 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 12 can also include a display device such as a display screen.
[0194] In one embodiment of the present application, the application processor 1131 is used to execute Fig.12 Corresponding to the defect detection method in the embodiment. It should be noted that, for the application processor 1131 to execute Fig.12 The specific implementation methods and beneficial effects of the defect detection methods in the corresponding embodiments can be referred to in Fig.12 The descriptions in the corresponding method embodiments will not be repeated here one by one.
[0195] Please also read Fig.14 , Fig.14 A schematic diagram of the structure of a network training device provided in an embodiment of the present application.
[0196] The network training device 200 can be used to implement Fig.11 Corresponding method embodiments. Specifically, the network training device 200 is implemented by one or more servers or computers. The network training device 200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 210 (for example, one or more processors) and memory 211, and one or more storage media 214 (for example, one or more massive storage devices) storing application programs 212 or data 213. Among them, the memory 211 and the storage medium 214 can be short-term storage or permanent storage. The program stored in the storage medium 214 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the network training device. Furthermore, the central processing unit 210 can be configured to communicate with the storage medium 214 to execute a series of instruction operations in the storage medium 214 on the network training device 200.
[0197] The network training device 200 may also include one or more power supplies 215, one or more wired or wireless network interfaces 216, one or more input and output interfaces 217, and / or one or more operating systems 218, such as Windows Server 200. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.
[0198] In one embodiment of the present application, the central processor 210 may be used to implement Fig.11 Corresponding method embodiments.
[0199] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps executed by the network training device in the above embodiment, or to execute the steps executed by the above defect detection device.
[0200] An embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the steps executed by the above-mentioned network training device, or to execute the steps executed by the above-mentioned defect detection device.
Claims
1. A training method for a defect detection network, It is characterized in that The method comprises: Using a first sample set to train a preset domain adaptation network, wherein the training samples in the first sample set include labeled product appearance image samples; Using a second sample set to train the preset domain adaptive network, a defect detection network for product appearance detection is obtained, wherein the training samples in the second sample set include unlabeled product appearance image samples, and the second sample set and the first sample set belong to different data domains; constructing a third sample set based on the first sample set and the second sample set, and processing the third sample set so that image feature differences of product appearance images belonging to different data domains in the third sample set are minimized, wherein the image feature differences are minimized so that domain classification of the product appearance images belonging to different data domains cannot be performed; The defect detection network is optimized and trained using the third sample set.
2. The method according to claim 1, It is characterized in that The method further comprises: Constructing a fourth sample set based on the first sample set and the second sample set, and processing the fourth sample set so as to minimize the maximum mean difference (MMD) of product appearance images belonging to different data domains in the fourth sample set; The defect detection network is optimized and trained using the fourth sample set.
3. The method according to claim 2, It is characterized in that The training of the defect detection network includes a first loss function, a second loss function, a third loss function and a fourth loss function. The first loss function is a loss function for training the preset domain adaptation network using the first sample set, the second loss function is a loss function for training the preset domain adaptation network using the second sample set, the third loss function is a loss function for minimizing the image feature differences of product appearance images belonging to different data domains in the third sample set, and the fourth loss function is a loss function for minimizing the MMD of product appearance images belonging to different data domains in the fourth sample set.
4. The method according to any one of claims 1 to 3, It is characterized in that The preset domain adaptation network includes a source domain network and a target domain network. The source domain network and the target domain network have the same network structure and share network parameters. The using the first sample set to train the preset domain adaptation network includes: Preprocessing the first sample set, and using the preprocessed first sample set to train the source domain network, wherein the preprocessing includes at least one of grayscale conversion, image cutting and splicing, image data augmentation, and normalization processing; The using the second sample set to train the preset domain adaptation network includes: The second sample set and the first sample set are preprocessed, and the target domain network is trained using the preprocessed second sample set and the first sample set.
5. The method according to claim 4, It is characterized in that The source domain network and the target domain network both include multiple defect detection heads, including anchor-based detection heads, anchor-free detection heads and anomaly detection heads. The Anchor-based detection head is used to detect defect types with an aspect ratio within a preset ratio range, the Anchor-free detection head is used to detect defect types with an aspect ratio outside the preset ratio range, and the anomaly detection head is used to perform binary classification detection of the presence of defects and the absence of defects.
6. The method according to any one of claims 1 to 3, It is characterized in that The using the second sample set to train the preset domain adaptation network includes: Using the preset domain adaptation network trained with the first sample set to add a pseudo label to each training sample in the second sample set; Obtaining the confidence of the pseudo label of each training sample in the second sample set, and setting the training weight of the corresponding training sample based on the confidence of the pseudo label; Based on each training sample and the training weight of each training sample, the preset domain adaptation network is trained.
7. The method according to claim 6, It is characterized in that The setting of the training weight of the corresponding training sample based on the confidence of the pseudo label includes: When the confidence of the pseudo label is greater than a preset threshold, a training weight of the corresponding training sample is set based on the confidence of the pseudo label.
8. A defect detection method, applied to computer equipment, It is characterized in that The computer device is deployed with a defect detection network, the defect detection network is trained according to the defect detection network training method according to any one of claims 1 to 7, the defect detection network includes a plurality of defect detection heads, the plurality of defect detection heads include a defect detection head for detecting a defect whose aspect ratio is within a preset ratio range, a defect detection head for detecting a defect whose aspect ratio is outside the preset ratio range, and a defect detection head for determining whether a defect exists, and the defect detection method includes: Acquire the appearance image of the product to be inspected; Inputting the appearance image into a pre-trained defect detection network; A detection result of each defect detection head among the multiple defect detection heads for the appearance image is obtained, and a union of the detection results of each defect detection head is used as the detection result of the product to be detected.
9. The method according to claim 8, It is characterized in that The step of inputting the appearance image into the pre-trained defect detection network comprises: Preprocessing the appearance image, the preprocessing comprising grayscale conversion processing and image cutting and splicing processing, the image cutting and splicing processing comprising cutting the appearance image into a plurality of sub-images in a length direction, and splicing at least two of the plurality of sub-images in a width direction; The preprocessed appearance image is input into the pre-trained defect detection network.
10. The method according to claim 8, It is characterized in that The obtaining of the detection result of each defect detection head among the plurality of defect detection heads for the appearance image, and taking the union of the detection results of each defect detection head as the detection result of the product to be detected comprises: Obtaining a detection result of each of the multiple defect detection heads for the appearance image and a confidence level of the detection result; If there is at least one defect detection head whose detection result confidence is greater than or equal to a preset confidence, determining that the detection result of the product to be detected is defective, and taking the union of the detection results of each defect detection head as the detection result of the product to be detected; If the confidence level of the detection result of each defect detection head is less than the preset confidence level, it is determined that the detection result of the product to be detected is that there is no defect.
11. The method according to any one of claims 8 to 10, It is characterized in that The computer device is in communication with an image acquisition device, the image acquisition device includes a line scan camera, and before acquiring the appearance image of the product to be inspected, the defect detection method further includes: Obtaining the motion trajectory coordinates of the product to be detected; Based on the rotating tool center point RTCP algorithm and the motion trajectory coordinates, the line scan camera is controlled to perform interpolation motion so that the distance between the line scan camera and the product to be inspected is maintained within a preset distance, and the preset distance is set based on the focal length of the line scan camera.
12. The method according to claim 11, It is characterized in that The method of controlling the line scan camera to perform interpolation motion based on the rotating tool center point RTCP algorithm and the motion trajectory coordinates includes: Calibrate the image acquisition endpoint of the line scan camera as the tool tip following point in the RTCP algorithm, calibrate the product to be inspected as the processed product in the RTCP algorithm, and calibrate the point to be photographed on the product to be inspected as the actual contact point between the tool tip in the RTCP algorithm and the processed product; Based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, the line scan camera is controlled to perform interpolation motion so that the distance between the image acquisition endpoint and the point to be photographed is within the preset distance.
13. The method according to claim 12, It is characterized in that The controlling the line scan camera to perform interpolation motion based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed includes: Constructing a camera coordinate system corresponding to the line scan camera; Convert the motion trajectory coordinates of the point to be photographed into the camera coordinate system to obtain the target trajectory coordinates of the point to be photographed in the camera coordinate system; The line scan camera is controlled to perform interpolation motion based on the RTCP algorithm and the target trajectory coordinates.
14. A computer device, It is characterized in that It comprises a memory and a processor, wherein the memory is used to store computer-readable instructions; the processor is used to read the computer-readable instructions and implement the training method of the defect detection network as described in any one of claims 1 to 7, or the defect detection method as described in any one of claims 8 to 13.
15. A computer storage medium, It is characterized in that Computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the training method of the defect detection network according to any one of claims 1 to 7 or the defect detection method according to any one of claims 8 to 13 is implemented.
