Defect detection methods and related equipment

By controlling the interpolation motion of the line scan camera through a pre-trained defect detection network and the RTCP algorithm, combined with domain adaptive learning and pseudo-label training, the problems of missed detection and low efficiency in manual inspection are solved, and efficient and accurate product appearance defect detection is achieved.

CN120833301BActive Publication Date: 2026-11-10SHENZHEN HONOR SMART MASCH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411632762.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-22
Publication Date
2026-11-10
Estimated Expiration
2044-04-22

AI Technical Summary

Technical Problem

Existing manual inspection methods are prone to missing defects in product appearance and have low inspection efficiency, making it impossible to achieve rapid, real-time inspection.

Method used

A pre-trained defect detection network with multiple defect detection heads is adopted. The interpolation motion of the line scan camera is controlled by image preprocessing and RTCP algorithm. Combined with domain adaptive learning and pseudo-label training, the detection accuracy and robustness are improved.

Benefits of technology

It enables efficient and accurate detection of product appearance defects, reduces the risk of missed detection, adapts to the detection needs of different types of products, and improves the robustness and recognition accuracy of the detection network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120833301B_ABST
    Figure CN120833301B_ABST
Patent Text Reader

Abstract

This application relates to the field of artificial intelligence inspection, providing a defect detection method and related equipment. The defect detection method includes: acquiring an appearance image of a product to be inspected; inputting the appearance image into a pre-trained defect detection network, the defect detection network including multiple defect detection heads, including a detection head for detecting defects with a suitable aspect ratio, a detection head for detecting defects with an unbalanced aspect ratio, and a detection head for determining the presence or absence of a defect; acquiring the detection result of each defect detection head for the appearance image, and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected. Using the above method, appearance defects of the product to be inspected can be accurately detected, and the risk of missing new types of appearance defects can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) detection, and in particular to a defect detection method and related equipment. Background Technology

[0002] During the manufacturing and processing of products, their surfaces are prone to defects such as scratches, dents, bumps, and raised particles. If the quality of the product's appearance cannot be effectively inspected, defective products will enter the market.

[0003] Current product appearance inspection generally relies on manual visual inspection. However, due to the limitations of human beings' physical condition and energy, it is impossible to conduct inspections continuously or quickly. The efficiency and accuracy of inspections are greatly affected by individual physical strength and other physiological factors. Summary of the Invention

[0004] In view of the above, it is necessary to provide a defect detection method and related equipment that can solve the problems of missed detection and low detection efficiency in existing manual inspection.

[0005] In a first aspect, this application provides a defect detection method applied to a computer device. The computer device is equipped with a pre-trained defect detection network. The defect detection network includes multiple defect detection heads, which include defect detection heads for detecting aspect ratios within a preset range, defect detection heads for detecting aspect ratios outside the preset range, and defect detection heads for determining whether a defect exists. The defect detection method includes: acquiring an appearance image of a product to be inspected; inputting the appearance image into the defect detection network; acquiring the detection result of each of the multiple defect detection heads for the appearance image; and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected.

[0006] Using the above technical solution, the pre-trained defect detection network includes multiple detection heads, which can detect defects with abnormal aspect ratios, such as excessively large or small aspect ratios, while ensuring the detection accuracy of defects of regular size. At the same time, by configuring detection heads to judge whether defects exist, it can be used to detect new types of defects that occur during product manufacturing, reducing the risk of missed defects. Furthermore, the final inspection result of the product to be inspected is generated based on the detection results of each defect detection head, which can improve the accuracy of product appearance defect detection. For example, the union of the detection results of each defect detection head can be used as the inspection result of the product to be inspected. For example, the defect detection network can perform the union of the detection results of each defect detection head, or a computer device can perform the union of the detection results of each defect detection head.

[0007] In one possible implementation, inputting the appearance image into the defect detection network includes: preprocessing the appearance image, the preprocessing including grayscale conversion and image cropping and stitching, the image cropping and stitching including cropping the appearance image into multiple sub-images in the length direction and stitching at least two of the multiple sub-images in the width direction; and inputting the preprocessed appearance image into the defect detection network.

[0008] By employing the above technical solution, the image is preprocessed before being input into the network. This preprocessing includes, but is not limited to, grayscale conversion, image cropping, and stitching. This reduces the amount of data processed by the network and ensures that the size of the input image meets the network's input size requirements. This approach guarantees both the accuracy of appearance defect recognition and significantly improves the network's recognition speed. For example, for images captured by a line scan camera, the appearance image can be cropped into multiple sub-images along its length, and at least two of these sub-images can be stitched together along their width.

[0009] In one possible implementation, obtaining the detection result of each of the multiple defect detection heads for the appearance image, and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected, includes: obtaining the detection result of each of the multiple defect detection heads for the appearance image and the confidence level of the detection result; if the confidence level of the detection result of at least one defect detection head is greater than or equal to a preset confidence level, determining that the detection result of the product to be inspected is that there is a defect, and taking the union of the detection results of each defect detection head as the detection result of the product to be inspected; if the confidence level of the detection result of each defect detection head is less than the preset confidence level, determining that the detection result of the product to be inspected is that there is no defect.

[0010] Using the above technical solution, if the confidence level of the detection result of at least one defect detection head is greater than or equal to the preset confidence level, it indicates that the defect detection network has detected a credible defect in the product under test, and the detection result of the product under test can be determined as having a defect. The union of the detection results of each defect detection head can be used as the final detection result of the product under test to avoid missing defects. If the confidence level of the detection result of each defect detection head is less than the preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product under test can be determined as having no defect.

[0011] In one possible implementation, the defect detection network can also be configured with a first preset confidence level and a second preset confidence level. The first preset confidence level is greater than the second preset confidence level. If the confidence level of the detection result of at least one defect detection head is greater than or equal to the first preset confidence level, it indicates that the defect detection network has detected a credible defect in the product under test, and the detection result of the product under test is determined to be defective. The union of the detection results of each defect detection head can be taken as the final detection result of the product under test. If the confidence level of the detection result of each defect detection head is less than the second preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product under test is determined to be defect-free. If the confidence level of the detection result of at least one defect detection head is less than the second preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product under test is determined to be defect-free. If the confidence level of the first preset confidence level is greater than or equal to the second preset confidence level, and the confidence level of the detection result of any defect detection head is greater than or equal to the first preset confidence level, it indicates that the defect detection network cannot determine whether there is a defect in the product appearance. The defect detection network can output a vector identification information of the detection anomaly. The computer device can output a preset detection anomaly prompt information based on the vector identification information. For example, the preset detection anomaly prompt information is "Unable to determine whether there is a defect in the product appearance. Please perform secondary image acquisition and detection." The operator can adjust the acquisition parameters of the image acquisition device (e.g., the shooting angle of the line scan camera, the brightness of the ambient light source, etc.) to re-acquire and detect the product to be inspected. The value of the preset confidence level can be set according to actual needs.

[0012] In one possible implementation, the computer device is communicatively connected to the image acquisition device, which includes a line scan camera. Before acquiring the appearance image of the product to be inspected, the defect detection method further includes: acquiring the motion trajectory coordinates of the product to be inspected; and controlling the line scan camera to perform interpolation motion based on the rotated tool center point (RTCP) algorithm and the motion trajectory coordinates, so that the distance between the line scan camera and the product to be inspected is maintained within a preset distance, which is set based on the focal length of the line scan camera.

[0013] By adopting the above technical solution, the line scan camera has the characteristics of high image clarity and small focal length. By combining RTCP technology with the characteristics of the line scan camera for linked image acquisition, the line scan camera can follow the trajectory of the product with irregular corners. This ensures that the distance between the line scan camera and the product to be inspected is always within the camera's focal length during complex movements, thereby improving the clarity of the image captured and the accuracy of data acquisition.

[0014] In one possible implementation, the line scan camera is controlled to perform interpolation motion based on the RTCP algorithm and motion trajectory coordinates. This includes: calibrating the image acquisition endpoint of the line scan camera as the tool tip following point in the RTCP algorithm, calibrating the product to be inspected as the processed product in the RTCP algorithm, and calibrating the point to be photographed on the product to be inspected as the actual contact point between the tool tip and the processed product in the RTCP algorithm; and controlling the line scan camera to perform interpolation motion based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, so that the distance between the image acquisition endpoint and the point to be photographed is within a preset distance.

[0015] By adopting the above technical solution, the image acquisition endpoint of the line scan camera is calibrated as the tool tip following point in the RTCP algorithm, the product to be inspected is calibrated as the processed product in the RTCP algorithm, and the point to be photographed on the product to be inspected is calibrated as the actual contact point between the tool tip and the processed product in the RTCP algorithm. Based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, the axis of the line scan camera is controlled to perform interpolation motion, thereby realizing the interpolation motion of the line scan camera. This combines RTCP technology with the characteristics of the line scan camera, enabling the line scan camera to perform linked image acquisition. During complex movements, the distance between the line scan camera and the point to be photographed can always be kept within the camera's focal length, improving the clarity of the image captured by the line scan camera and the accuracy of data acquisition.

[0016] In one possible implementation, the line scan camera is controlled to perform interpolation motion based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed. This includes: constructing a camera coordinate system corresponding to the line scan camera; transforming the motion trajectory coordinates of the point to be photographed to the camera coordinate system to obtain the target trajectory coordinates of the point to be photographed in the camera coordinate system; and controlling the line scan camera to perform interpolation motion based on the RTCP algorithm and the target trajectory coordinates.

[0017] By adopting the above technical solution, the motion trajectory coordinates of the point to be photographed are transformed into the camera coordinate system to obtain the target trajectory coordinates. Based on the target trajectory coordinates and the RTCP algorithm, the line scan camera is controlled to perform interpolation motion, so that the distance between the line scan camera and the point to be photographed can always be kept within the camera's focal length during complex motion, thereby improving the clarity of the image captured by the line scan camera and the accuracy of data acquisition.

[0018] In one possible implementation, the defect detection network is trained based on a training sample set and a pre-defined domain adaptive network. The training sample set includes a labeled first sample set and an unlabeled second sample set, with the second sample set belonging to different data domains as well as the first sample set. The training sample set also includes a third sample set, which is constructed based on the first and second sample sets. The third sample set undergoes domain classification processing to minimize the differences in image features between training samples belonging to different data domains in the third sample set.

[0019] Using the above technical solution, the first sample set can be labeled training samples from the source data domain, and the second sample set can be unlabeled training samples from the target data domain. Training the domain adaptive network based on training samples from different data domains can improve the robustness of the trained defect detection network for detecting different types of product appearance defects. Furthermore, the number of samples in the second sample set can be much smaller than the number of samples in the first sample set, facilitating cross-domain detection. A third sample set is constructed through domain classification processing, minimizing the image feature differences of product appearance images belonging to different data domains in the third sample set. For example, minimizing image feature differences can mean that product appearance images belonging to different data domains in the third sample set cannot be classified by domain (it is impossible to distinguish whether they come from the source data domain or the target data domain). Optimizing the defect detection network using the third sample set can further improve the robustness and recognition accuracy of the defect detection network for detecting different types of product appearance defects.

[0020] In one possible implementation, the training sample set further includes a fourth sample set, which is constructed based on the first and second sample sets and undergoes maximum mean discrepancy (MMD) processing to minimize the maximum mean discrepancy of training samples belonging to different data domains in the fourth sample set.

[0021] By adopting the above technical solution, a fourth sample set is constructed through the MMD method, which minimizes the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set. The defect detection network is then optimized and trained using the fourth sample set, which can further improve the robustness and recognition accuracy of the defect detection network for detecting different types of product appearance defects.

[0022] Secondly, this application provides a network training method, the method comprising: training a preset domain adaptive network using a first sample set, wherein the training samples in the first sample set include labeled product appearance image samples; training the preset domain adaptive network using a second sample set to obtain a defect detection network for product appearance inspection, wherein the training samples in the second sample set include unlabeled product appearance image samples, and the second sample set and the first sample set belong to different data domains; constructing a third sample set based on the first sample set and the second sample set, and processing the third sample set to minimize the image feature differences of product appearance images belonging to different data domains in the third sample set; and optimizing the training of the defect detection network using the third sample set.

[0023] Using the above technical solution, the first sample set can be labeled training samples from the source data domain, and the second sample set can be unlabeled training samples from the target data domain. Training the domain adaptive network based on training samples from different data domains can improve the robustness of the detection network to the detection of product appearance defects of different types / models. Moreover, the number of samples in the second sample set can be much smaller than the number of samples in the first sample set, which can facilitate cross-domain detection applications. A third sample set is constructed through domain classification processing, and the differences in image features of product appearance images belonging to different data domains in the third sample set are minimized. For example, minimizing image feature differences can mean that product appearance images belonging to different data domains in the third sample set cannot be classified by domain (it is impossible to distinguish whether they come from the source data domain or the target data domain). The defect detection network is then optimized and trained using the third sample set, which can further improve the robustness and recognition accuracy of the defect detection network to the detection of different types of product appearance defects.

[0024] In one possible implementation, the network training method further includes: constructing a fourth sample set based on the first sample set and the second sample set, and processing the fourth sample set to minimize the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set; and using the fourth sample set to optimize the training of the defect detection network.

[0025] By adopting the above technical solution, a fourth sample set is constructed through the MMD method, which minimizes the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set. The defect detection network is then optimized and trained using the fourth sample set, which can further improve the robustness and recognition accuracy of the defect detection network for detecting different types of product appearance defects.

[0026] In one possible implementation, the training of the defect detection network includes a first loss function, a second loss function, a third loss function, and a fourth loss function. The first loss function is a loss function used to train the preset domain adaptive network using a first sample set. The second loss function is a loss function used to train the preset domain adaptive network using a second sample set. The third loss function is a loss function that minimizes the differences in image features of product appearance images belonging to different data domains in the third sample set. The fourth loss function is a loss function that minimizes the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set.

[0027] By adopting the above technical solution, the total loss function of the defect detection network is configured to consist of a first to a fourth loss function. The first loss function can be the loss function used to train the domain adaptive network using a first sample set. The second loss function can be the loss function used to train the preset domain adaptive network using a second sample set, or the loss function used to train the preset domain adaptive network using a combination of the first and second sample sets. The third loss function can be the loss function that minimizes the differences in image features of product appearance images belonging to different data domains in the third sample set. The fourth loss function is the loss function that minimizes the maximum mean difference of product appearance images belonging to different data domains in the fourth sample set. This enables the construction of a domain-adaptive product appearance defect detection capability, which can improve the robustness and recognition accuracy of the defect detection network for detecting different types of product appearance defects.

[0028] In one possible implementation, the preset domain adaptive network includes a source domain network and a target domain network. The source domain network and the target domain network have the same network structure and share network parameters. Training the preset domain adaptive network using a first sample set includes: preprocessing the first sample set and training the source domain network using the preprocessed first sample set. The preprocessing includes at least one of grayscale conversion, image cropping and stitching, image data augmentation, and normalization. Training the preset domain adaptive network using a second sample set includes: preprocessing both the second sample set and the first sample set, and training the target domain network using the preprocessed second sample set and the first sample set.

[0029] By employing the above technical solution, before training the network using the sample set, preprocessing the sample set (including but not limited to grayscale conversion, image cropping and stitching, image data augmentation, and normalization) can reduce the amount of data processing required by the network and improve the training efficiency and generalization ability of the network. Furthermore, training the source domain network using labeled training samples and the target domain network using unlabeled training samples, or training the target domain network using a combination of unlabeled and labeled training samples, can improve the robustness of the defect detection network for detecting different types of product appearance defects.

[0030] In one possible implementation, both the source domain network and the target domain network include multiple defect detection heads. These multiple defect detection heads include anchor-based detection heads, anchor-free detection heads, and anomaly detection heads. The anchor-based detection heads are used to detect defect types whose aspect ratio is within a preset range. The anchor-free detection heads are used to detect defect types whose aspect ratio is outside the preset range. The anomaly detection heads are used to perform binary classification detection of the presence or absence of defects.

[0031] By adopting the above technical solution and configuring multiple detection heads with different defect detection functions, it is possible to detect defects with different aspect ratios and new types of defects. For example, the network defect detection head can be designed based on anchor-free and anchor-based detection methods. While ensuring the detection accuracy of regular-sized defects, it can also detect defects with abnormal aspect ratios, such as defects with excessively large or small aspect ratios. Furthermore, by setting up a binary classification detection head for judging the presence or absence of defects, it is possible to detect new types of defects and reduce the risk of missed defect detection.

[0032] In one possible implementation, training a pre-defined domain adaptive network using a second sample set includes: adding pseudo-labels to each training sample in the second sample set using the pre-defined domain adaptive network trained on the first sample set; obtaining the confidence level of the pseudo-labels for each training sample in the second sample set, and setting the training weights of the corresponding training samples based on the confidence levels of the pseudo-labels; and training the pre-defined domain adaptive network based on each training sample and the training weights of each training sample.

[0033] By adopting the above technical solution, pseudo-labels are added to unlabeled training samples to enable semi-supervised learning for network training. Based on the confidence level of the pseudo-labels, corresponding training weights are assigned to each training sample with pseudo-labels. This results in training samples with high confidence having higher training weights and training samples with low confidence having lower training weights, thereby improving the training performance and recognition accuracy of the network.

[0034] In one possible implementation, the training weights of the corresponding training samples are set based on the confidence level of the pseudo-label, including: when the confidence level of the pseudo-label is greater than a preset threshold, the training weights of the corresponding training samples are set based on the confidence level of the pseudo-label.

[0035] Using the above technical solution, when the confidence level of the pseudo-label is lower than the preset threshold, the training sample with the added pseudo-label is not used for network training. The preset threshold can be set according to actual training needs. For pseudo-labels with a confidence level greater than or equal to the preset threshold, different training weights are configured for the corresponding unlabeled samples and used for network training. This ensures that training samples with high confidence have higher training weights, and training samples with low confidence have lower training weights, thereby improving the network's training performance and recognition accuracy. As the number of network training iterations increases, the confidence level of the pseudo-labels added to the unlabeled training samples will change. Furthermore, the network can be iteratively trained using unlabeled training samples to fully utilize these samples for network training.

[0036] Thirdly, embodiments of this application provide a computer device including a memory and a processor, wherein the memory is used to store computer-readable instructions; when the computer-readable instructions are executed by the processor, the computer device causes the computer device to implement the method provided in the first aspect and its optional implementations, or to implement the method provided in the second aspect and its optional implementations.

[0037] Fourthly, embodiments of this application provide a computer storage medium storing computer-readable instructions, which, when executed by a processor, implement the method provided by the first aspect and its optional implementations, or implement the method provided by the second aspect and its optional implementations.

[0038] Fifthly, embodiments of this application provide a computer program product containing computer-readable instructions that, when executed by a processor, implement the method provided by the first aspect and its optional implementations, or implement the method provided by the second aspect and its optional implementations.

[0039] The technical effects achieved by the third, fourth and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first or second aspects, and will not be repeated here. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram of an artificial intelligence main framework provided for an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of a mid-frame defect detection system provided in an embodiment of this application;

[0043] Figure 3 A schematic diagram illustrating a type of defect in the middle frame provided in an embodiment of this application;

[0044] Figure 4 A schematic diagram illustrating another type of defect in the mid-frame provided in the embodiments of this application;

[0045] Figure 5 A schematic diagram illustrating another type of defect in the middle frame provided in an embodiment of this application;

[0046] Figure 6A schematic diagram of an image at the corner of the mid-frame acquired by a line scan camera controlled by the RTCP algorithm to follow the movement of the mid-frame, as provided in an embodiment of this application.

[0047] Figure 7 A schematic diagram illustrating the process of controlling a line scan camera to follow the movement of the frame to be detected based on the RTCP algorithm, as provided in an embodiment of this application.

[0048] Figure 8 This is a schematic diagram illustrating the cropping and stitching process of a mid-frame image captured by a line drawing camera, as provided in an embodiment of this application.

[0049] Figure 9 This is a schematic diagram of the network structure of the defect detection network provided in the embodiments of this application;

[0050] Figure 10 This diagram illustrates the different types of defects that can be detected by the Anchor-free detection head and the Anchor-based detection head in the defect detection network provided in this application embodiment;

[0051] Figure 11 A schematic flowchart illustrating the network training method provided in this application embodiment;

[0052] Figure 12 A schematic flowchart illustrating the defect detection method provided in this application embodiment;

[0053] Figure 13 A schematic diagram of the structure of a defect detection device provided in an embodiment of this application;

[0054] Figure 14 This is a schematic diagram of a network training device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0056] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, words such as "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary," "or," and "for example" is intended to present the relevant concepts in a concrete manner.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" means one or more. "More than one" means two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. It should be understood that the order of steps shown in the flowcharts herein can be changed, and some can be omitted.

[0058] To facilitate understanding of the various embodiments of this application, we will first refer to... Figure 1 Describe the overall workflow of the artificial intelligence system. For example... Figure 1 The diagram shown illustrates a general framework for artificial intelligence, which describes the overall workflow of an artificial intelligence system and is applicable to the needs of general artificial intelligence fields.

[0059] The following text will elaborate on the above-mentioned artificial intelligence framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0060] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0061] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0062] (1) Infrastructure

[0063] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then provided to intelligent chips in the distributed computing system on the basic platform for computation.

[0064] (2) Data

[0065] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. The data includes graphics, images, voice, text, as well as IoT data from traditional devices, business data from existing systems, and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0066] (3) Data processing

[0067] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0068] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0069] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0070] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0071] (4) General ability

[0072] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0073] (5) Smart Products and Industry Applications

[0074] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0075] During the manufacturing process, products are prone to surface defects such as scratches, dents, bumps, and raised particles. If the quality of the product's appearance cannot be effectively inspected, defective products will enter the market. Current product appearance inspection generally relies on manual visual inspection. However, due to limitations in human capabilities and energy, continuous and rapid inspection is not possible, and the efficiency and accuracy are significantly affected by individual physical and mental factors.

[0076] The following example uses the mid-frame of a terminal device as an illustration. As the "skeleton" of the terminal device, the mid-frame supports various components and is typically made of metal or plastic. The mid-frame not only serves a decorative purpose but is also a crucial structure connecting different parts of the device, helping to protect internal components from external impacts. The shape, color, size, and material of the mid-frame vary depending on the model and brand of the terminal device. The mid-frame is a major area where the appearance quality of terminal devices is compromised. Various types of defects may exist during the manufacturing process, and if defective mid-frames enter the market, it can lead to customer complaints.

[0077] Current methods for inspecting mid-frames typically rely on manual visual inspection. However, manual inspection generally suffers from the following problems: 1. Human error is prone to occur; inconsistencies in inspection standards, experience, and methods can lead to inconsistent pass / fail criteria for mid-frames. 2. Inspectors are susceptible to fatigue, especially after prolonged periods of performing the same tasks, resulting in visual fatigue and incomplete detection of defective products. 3. Manual inspection is prone to missed detections. 4. Manual inspection is inefficient; due to human limitations, continuous and rapid inspection is not possible, and efficiency is significantly affected by individual energy and physical limitations. Training a defect detection network using deep learning technology requires a large number of labeled mid-frame samples. However, collecting massive amounts of defective mid-frame samples from production lines is difficult, and the models of terminal equipment used on production lines are not fixed. When switching between different models or colors of mid-frames on the production line, the accuracy of the defect detection network drops significantly.

[0078] In view of this, the embodiments of this application provide a defect detection method, which comprehensively collects product appearance data from different angles, uses deep learning detection algorithms to detect product appearance defects of various shapes, and uses domain adaptive learning algorithms to improve the robustness of cross-domain recognition, so that the defect detection network has high defect detection accuracy for different models and types of product appearance detection.

[0079] The defect detection method implemented in this application can be applied to computer equipment, which can be computers (e.g., industrial computers, personal computers, etc.), servers, server clusters, etc. Taking the defect detection method for detecting the appearance of the mid-frame as an example, the mid-frame can be the mid-frame of a terminal device, such as the mid-frame of a smartphone, tablet computer, smartwatch, wearable device, etc., and this application embodiment does not limit this.

[0080] It is understandable that when the sample data used to train the defect detection network is not the appearance data of the mid-frame, but the appearance data of other products (e.g., car doors, battery covers, automotive parts), the defect detection method of this application can also be applied to the appearance inspection of other products.

[0081] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0082] (1) Neural Network

[0083] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0084]

[0085] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0086] (2) Deep Neural Networks

[0087] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0088] Although DNNs appear complex, the operation of each layer can be represented by the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained after processing. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as...

[0089] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0090] (3) Convolutional Neural Network

[0091] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0092] (4) Loss Function

[0093] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0094] (5) Domain-adaptive learning

[0095] Domain-adaptive learning is a method of transfer learning. It is applicable to addressing the challenge of inconsistent distributions between training and testing datasets and real-world data in neural network models. The idea behind domain-adaptive learning is to map data features from different domains to the same feature space, thus leveraging data from other domains to enhance training in a specific domain.

[0096] like Figure 2 The diagram shown is a schematic diagram of the architecture of a frame defect detection system 10 according to an embodiment of this application.

[0097] The mid-frame defect detection system 10 may include a mid-frame feeding module 101, an image acquisition module 102, a mid-frame appearance inspection module 103, and a mid-frame post-processing module 104. For example, the mid-frame loading module 101 can use a robotic arm to grab the mid-frame to be assembled and place it on the mid-frame fixing component, which is used to fix the mid-frame. The image acquisition module 102 can use a motion control mechanism and multiple cameras to acquire mid-frame appearance data from multiple angles. For example, the motion control mechanism can control the rotation of the mid-frame fixing component and the camera. The image acquisition module 102 can transmit the acquired mid-frame appearance data to the mid-frame appearance inspection module 103 for data processing and recognition to determine whether there are defects in the mid-frame appearance, such as whether there are scratches on the outer edge of the mid-frame. The mid-frame appearance inspection module 103 can transmit the appearance inspection results of the mid-frame to the mid-frame post-processing module 104. The mid-frame post-processing module 104 can complete the subsequent operations of the mid-frame based on the appearance inspection results. For example, defect-free mid-frames are sent to the next process for subsequent assembly operations, while defective mid-frames are placed in a scrap bin for subsequent manual processing.

[0098] In some embodiments, the mid-frame defect detection system 10 can adopt a left-in, right-out layout for mid-frame detection and transmission. The mid-frame loading module 101 can automatically load and fix the mid-frame. For example, the mid-frame loading module 101 may include a robotic arm and a mid-frame fixing component. The robotic arm picks up the mid-frame to be assembled from the upper frame of the conveyor belt and places it onto the mid-frame fixing component. The mid-frame fixing component may be equipped with a positioning platform for fixing the mid-frame. This embodiment does not limit the shape of the positioning platform; for example, it may be a trapezoidal positioning platform. The positioning platform can cooperate with the positioning holes of the mid-frame to achieve high-precision positioning of the mid-frame. The positioning holes of the mid-frame can refer to screw holes, mounting holes, etc., provided on the mid-frame itself. To prevent relative displacement between the mid-frame fixing component and the mid-frame during rotation, a vacuum suction cup can also be provided on the mid-frame fixing component to achieve double fixing of the mid-frame.

[0099] In some embodiments, the shapes of defects in the mid-frame vary, with some defects appearing as tiny dots and others as narrow strips. For small-sized mid-frame defects, images captured by low-resolution cameras show indistinct defect features, making them difficult to distinguish and extract features. However, high-resolution line scan cameras / cameras can capture clear defect images, facilitating subsequent feature extraction and network training. The image acquisition module 102 may include one or more two-dimensional line scan cameras / cameras. For defects such as scratches and abrasions, clearer images are required from different perspectives. Therefore, to improve the efficiency of acquiring all mid-frame defects and the comprehensiveness of defect scanning, the image acquisition module 102 may include at least three line scan cameras. These three line scan cameras can be fixedly mounted on camera fixtures. The number of camera fixtures can be one or the same as the number of line scan cameras. The mounting positions of the three line scan cameras can be set according to actual needs, and this embodiment does not limit this. For example, one of the three line scan cameras is positioned to capture images of the outer edge of the mid-frame, while the other two line scan cameras are symmetrically positioned relative to this camera. One camera is tilted at +30 degrees to capture images of the outer edge of the mid-frame, and the other is tilted at -30 degrees to capture images of the outer edge of the mid-frame, thus capturing three images of the mid-frame from the same position at different angles.

[0100] In some embodiments, to enable the acquisition of appearance images of different positions of the mid-frame from different angles, the image acquisition module 102 may further include a motion control device and a motion device. The motion control device controls the movement of the motion device, and the movement dimensions of the motion device can be selected according to actual needs. For example, it can be three-axis motion (X, Y, Z axes), four-axis motion (X, Y, Z axes + one rotation axis), or five-axis motion (X, Y, Z axes + two rotation axes). The motion device may include a motor / electric motor assembly, and the motion control device may include a motor controller / electric motor controller. Existing motion control devices and motion devices can be used. The motion device drives the movement of the mid-frame fixing component and the camera fixing component, so that the mid-frame and the line scan camera can move accordingly.

[0101] In some embodiments, since the middle frame is generally exposed to the user's field of vision with only the four outer edges after product assembly, the image acquisition module 102 can only acquire images of the four outer edges of the middle frame for detection, thereby improving the speed of image acquisition and defect detection of the middle frame.

[0102] For example, a defect image of the mid-frame captured by a line scan camera, such as... Figures 3-5 As shown, Figure 3 This indicates that there are dot-like protrusions in the middle frame. Figure 4 This indicates that the mid-frame has scratches and defects. Figure 5 This indicates that the mid-frame has a jagged edge defect.

[0103] In some embodiments, since the focal length of a 2D line scan camera is generally small, defocusing is prone to occur when capturing images at the corners of the frame. To reduce the probability of defocusing, existing motion control algorithms place high demands on the motion control of the motion device. To solve this problem, this application applies the rotated tool center point (RTCP) algorithm to the linkage motion control between the line scan camera and the frame, combining it with five-axis linkage technology to improve the tracking stability of the line scan camera relative to the frame and the real-time performance of image acquisition, thereby acquiring clear images at the corners of the frame. Motion control based on the RTCP algorithm allows the line scan camera to maintain a distance from the frame within the camera's focal length range during complex movements, improving the image acquisition clarity and the accuracy of the acquired image data. For example, the image at the corner of the frame acquired by the line scan camera controlled by the RTCP algorithm is shown below. Figure 6 As shown.

[0104] Existing RTCP algorithms are mainly used for precise control of the tool tip center point of a five-axis machine tool. In this embodiment, the image acquisition endpoint of the line scan camera is equated to the tool tip following point in the RTCP algorithm, the product to be inspected (e.g., the middle frame) is equated to the product being processed, and the point on the product to be inspected that is directly opposite the image acquisition endpoint is the point to be photographed (equivalent to the actual contact point between the tool tip and the surface of the product being processed). The rotation axis of the line scan camera and the rotation axis of the product to be inspected are equated to the two rotation axes of the five-axis machine tool. By calibrating the rotation center of the rotation axis of the product to be inspected in the camera coordinate system, the motion trajectory coordinates of the product to be inspected are transformed based on the rotation center in the camera coordinate system, and an encoder control signal is generated based on the coordinate transformation result to control the encoder to output pulse signals. The motion device realizes the interpolation motion of the line scan camera based on the pulse signals. The line scan camera can also perform image acquisition based on the pulse signals, so that the distance between the line scan camera and the middle frame is always kept within the camera's focal length range (this can be achieved by keeping the distance between the image acquisition endpoint and the point to be photographed constant, or by keeping it within a preset distance), which can improve the tracking stability and real-time acquisition performance of the line scan camera for the product to be inspected.

[0105] like Figure 7 The diagram shows a flowchart of controlling the motion of a line scan camera based on the RTCP algorithm.

[0106] S41, set the line scan camera and the frame to be detected to a linked motion mode, and set the motion speed of the frame to be detected.

[0107] In some implementations, the linked motion mode can refer to the line scan camera performing interpolation motion along with the movement of the frame to be detected. The movement speed of the frame to be detected can be set according to the actual image acquisition requirements, but this application embodiment does not limit this.

[0108] S42, turn on the line scan camera, enabling the line scan camera to receive pulse signals and acquire images.

[0109] S43, obtain the position and vector of the point to be photographed, and obtain the motion trajectory coordinates of the point to be photographed.

[0110] After obtaining the position and vector of the point to be photographed, the motion trajectory coordinates of the point to be photographed can be obtained based on the position and vector of the point to be photographed.

[0111] In some embodiments, after the middle frame is fixed, a middle frame coordinate system can be constructed based on the rotation axis of the middle frame to obtain the position and vector of the point to be photographed. The motion trajectory coordinates of the point to be photographed can be the motion trajectory coordinates in the middle frame coordinate system.

[0112] S44 transforms the motion trajectory coordinates of the point to be photographed into the camera coordinate system to obtain the target trajectory coordinates.

[0113] In some embodiments, by constructing a camera coordinate system corresponding to the line scan camera, and then transforming the motion trajectory coordinates of the point to be photographed to the camera coordinate system, the target trajectory coordinates are obtained.

[0114] S45 performs interpolation calculations on the image acquisition endpoints of the line scan camera based on the RTCP algorithm and target trajectory coordinates.

[0115] S46 outputs interpolation control commands based on the interpolation calculation results, which are used to control the interpolation motion of the axis where the line scan camera is located.

[0116] S47 outputs a pulse signal according to the interpolation control command. The pulse signal is used to make the axis where the line scan camera is located perform interpolation movement and image acquisition.

[0117] In some embodiments, pulse signals can be used to drive a motion device, causing the motion device to drive the axis of the line scan camera to perform interpolation motion, thereby realizing the interpolation motion of the line scan camera. The line scan camera can also perform image acquisition based on pulse signals, so that the line scan camera can always acquire images of the mid-frame within the focal length range and ensure the real-time performance of image acquisition.

[0118] In some embodiments, the mid-frame appearance inspection module 103 may include a defect identification device that is communicatively connected to a line scan camera. For example, the defect identification device may be a computer device or a module containing a processor and memory. The defect identification device may be deployed with a defect detection network for performing mid-frame defect detection.

[0119] In some embodiments, the mid-frame appearance detection module 103 can be divided into three functional units according to the process of processing the mid-frame image acquired by the image acquisition module 102: a data preprocessing unit, a defect detection unit based on a defect detection network, and a detection result post-processing unit. The data preprocessing unit can be used to preprocess the mid-frame image, and the preprocessing can include grayscale conversion, image cropping and stitching, image data augmentation, normalization, etc.

[0120] For example, the mid-frame images captured by line scan cameras are relatively large. For instance, the resolution of a single long-side image of a mobile phone mid-frame can typically reach 30000*4096 pixels. Such large images cannot be directly input into a defect detection network for recognition. Furthermore, scaling the image before inputting it into the defect detection network would affect the accuracy of identifying small-scale defects. The image cropping and stitching in this embodiment may include cropping the region of interest (ROI) of the mid-frame image, removing non-mid-frame pixel regions, then cropping the cropped image into multiple mid-frame image blocks along the length direction with a certain overlap ratio, and finally stitching the cropped mid-frame image blocks together in the width direction. The pixel size for cropping and stitching the mid-frame image can be set according to the image input size supported by the detection network; this embodiment does not limit this setting.

[0121] like Figure 8 As shown, the image I1 from the line scan camera is cropped along its long side into multiple mid-frame image blocks I2 with a certain overlap ratio. These mid-frame image blocks I2 can have the same length and are stitched together in the width direction to obtain the processed image I3. By cropping and stitching the mid-frame image acquired by the line scan camera, useless image pixels can be removed, reducing the data processing load and noise interference of the network, thus improving the network's processing speed and recognition accuracy. Furthermore, stitching multiple mid-frame image blocks together further improves the network's recognition speed compared to inputting a single mid-frame image block into the network for recognition.

[0122] In some embodiments, the image data augmentation methods used in this application may include, but are not limited to, geometric transformation, color gamut transformation, noise injection, and cross-domain image generation. Cross-domain image generation can refer to using existing image generation algorithms to generate mid-frame images of different scenes / products across domains, thereby increasing the diversity and scale of training data and improving the network's domain adaptability from a data perspective. Mid-frame images of different scenes can refer to mid-frame images with different background colors, different ambient light intensities, different contrasts, etc.

[0123] In some embodiments, the mid-frame image captured by the line scan camera is generally a color image (e.g., an RGB image). To reduce the data processing load on the network, the mid-frame image input to the network for recognition is a grayscale image. Grayscale conversion can refer to converting the color mid-frame image captured by the line scan camera into a grayscale mid-frame image. Normalization processing can refer to mapping grayscale values ​​of 0-255 to the range of 0-1 to simplify the data processing process and improve the efficiency and accuracy of network training.

[0124] The defect detection unit uses a defect detection network to extract features and detect defects in the preprocessed mid-frame image. For example... Figure 9 As shown, the defect detection network includes two sub-networks (hereinafter referred to as the source domain network and the target domain network). The source domain network and the target domain network are twin neural networks, that is, the source domain network and the target domain network have the same network structure and share network parameters. For example, the source domain network and the target domain network share weights with each other.

[0125] A source domain network is typically trained on labeled data from the source data domain (e.g., a labeled border image of product A). After training, the source domain network can be used to detect defects in the border image of product A. Labels can include information about the presence or absence of defects, and, if present, the specific type of defect.

[0126] The target domain network is generally trained on unlabeled data from the target data domain (e.g., an unlabeled border image of product B), or jointly trained on unlabeled data from the target data domain and labeled data from the source data domain. This allows for subsequent defect detection on the border image of product B. The number of samples in the target data domain can be much smaller than the number of samples in the source data domain, enabling the defect detection network to detect defects in the border images of both product A and product B. The source and target data domains can be defined according to actual needs; this application does not limit this. Generally, the source and target data domains involve product images of different models, brands, or series.

[0127] Similarly, a target data domain for product C can be constructed, and the target domain network can be trained using the target data domain of product C, so that the defect detection network can have the ability to detect defects in the bounding box images of product A, product B, and product C.

[0128] Since the source domain network and the target domain network have the same network structure, the following only illustrates the network structure of the source domain network. The source domain network can include a feature extraction layer and a defect detection layer. The feature extraction layer can include multiple network layers Conv1 to ConvN for image feature extraction, where N is a positive integer greater than 1. The image features extracted by the feature extraction layer include, but are not limited to, texture features, brightness features, contrast features, and grayscale features. Each network layer Conv1 to ConvN can perform operations such as convolution, batch normalization (BN), pooling, and non-linear activation. For example, non-linear activation operations can be performed based on non-linear activation functions such as the Sigmoid function, tanh function, ReLU function, and Leaky ReLU function.

[0129] The defect detection layer may include multiple defect detection heads. Each defect detection head may include convolutional layers and fully connected layers, or only fully connected layers. Each defect detection head may be constructed based on network features of different levels and / or different scales to achieve the corresponding defect detection function. The number of defect detection heads can be set according to the actual mid-frame detection requirements, and this application embodiment does not limit this. Figure 9 Three defect detection heads are illustrated as examples: anchor-based detection head, anchor-free detection head, and anomaly detection head. The anchor-based and anchor-free detection heads are built on CNN networks, meaning that both can include multiple convolutional layers, pooling layers, fully connected layers, etc. The anomaly detection head can include multiple fully connected layers.

[0130] Anchor-based detection heads create a series of anchor boxes with different sizes and aspect ratios, either explicitly or implicitly, and then classify or regress the contents within these anchor boxes. This is suitable for detecting mid-frame defects with moderate aspect ratios (such as point-like protrusions), for example, mid-frame defects with aspect ratios between 0.5 and 2. Anchor-free detection heads predict the probability of each pixel in the image belonging to the object to be detected and the object's bounding box information. Based on this information, they generate bounding boxes and classify or regress the contents within them. This is suitable for detecting mid-frame defects with unbalanced aspect ratios (such as narrow scratches), for example, mid-frame defects with aspect ratios less than 0.5 or greater than 2. Anomaly detection heads are used for binary classification training to identify whether mid-frame images contain defects, thereby reducing the risk of the detection network missing new types of mid-frame defects.

[0131] It is understandable that different defect detection heads can be set according to other defect types, sizes, or detection methods. This application embodiment is not limited to setting Anchor-based detection heads, Anchor-free detection heads, and Anomaly detection heads.

[0132] like Figure 10 As shown in (a) and (b), the narrow midframe defect detected by the anchor-free inspection head is illustrated. Figure 10 As shown in (c), this illustrates a mid-frame defect with a suitable aspect ratio detected by an anchor-based inspection head.

[0133] The defect detection network in this application embodiment can identify the types of mid-frame defects learned during the training phase using anchor-based and anchor-free detection heads. That is, the anchor-based and anchor-free detection heads output the specific defect types present in the mid-frame image. The anomaly detection head can identify mid-frame defect types not learned during the training phase. That is, for new types of mid-frame defects, although the anchor-based and anchor-free detection heads cannot identify them, the anomaly detection head can identify that the mid-frame image has a defect (defect type unknown). In this case, manual verification can be performed, and if it is determined to be a new type of mid-frame defect, a label is added to the mid-frame image. The labeled mid-frame image can be input into the defect detection network for training, so that this type of mid-frame defect can be subsequently identified by the anchor-based and / or anchor-free detection heads.

[0134] In some embodiments, training the defect detection network may include: training a source domain network based on training samples from a labeled source data domain, and training a target domain network based on training samples from an unlabeled target data domain (or jointly training the target domain network based on training samples from both the unlabeled target data domain and the labeled source data domain). Furthermore, during network training, the maximum mean discrepancy (MMD) algorithm and domain classification algorithm are used at the feature extraction level to reduce the feature differences between the source and target data domains, thereby further improving the robustness of the defect detection network for cross-scenario deployment.

[0135] For example, suppose a batch of training samples includes m labeled training samples (from the source data domain) and n unlabeled training samples (from the target data domain), where m and n are positive integers and their values ​​can be equal or unequal. A loss function based on the MMD algorithm can be constructed to process the training samples in this batch, minimizing the difference between the feature mean of the m labeled training samples and the feature mean of the n unlabeled training samples. This reduces the feature difference between the source and target data domains. After the mean-minimizing processing, this batch of training samples can be used for network optimization training. Minimizing the difference in feature means based on the MMD algorithm can refer to minimizing the difference between the mean of defect features and the mean of image features for the same defect type, or it can refer to minimizing the difference between the mean of defect features and the mean of image features for training samples randomly selected from the source and target data domains.

[0136] For example, a domain classification algorithm can be defined as an algorithm that makes it impossible to distinguish training samples from the source data domain from training samples from the target data domain. Suppose a batch of training samples includes m labeled training samples (from the source data domain) and n unlabeled training samples (from the target data domain). Each training sample in this batch can be classified (i.e., it can be determined whether it belongs to the source data domain or the target data domain). By constructing a loss function to process the training samples in this batch, the difference between training samples from different data domains can be minimized. For example, by processing the training samples in the batch through a fully connected layer, the training samples in this batch cannot be classified, that is, they cannot be classified as belonging to the source data domain or the target data domain. This reduces the feature difference between the source data domain and the target data domain. After being processed by a domain classification algorithm, the training samples in this batch can be used for network optimization training.

[0137] For example, a loss function can be constructed to measure the difference in defect features or image features among training samples from different domains (source data domain or target data domain) that contain the same defect type. By minimizing this difference, these training samples containing the same defect type cannot be classified into the domain, and can then be used to optimize and train the network. Alternatively, this can be achieved by processing training samples randomly selected from the source data domain and the target data domain that cannot be classified into the domain.

[0138] In some embodiments, training the target domain network using training samples from the unlabeled target data domain can be achieved by: using the target domain network to create pseudo-labels for each unlabeled training sample, and then training the target domain network based on the training samples with added pseudo-labels.

[0139] To improve the training performance and accuracy of the target domain network, training weights can be assigned to each training sample with added pseudo-labels based on the confidence level of the pseudo-labels. This ensures that training samples with high confidence levels have higher training weights, and training samples with low confidence levels have lower training weights. For example, when an unlabeled training sample P1 is input into the target domain network, a detection result and a confidence level are obtained. The detection result can be added to training sample P1 as a pseudo-label, and the confidence level serves as the confidence level of the pseudo-label. The higher the confidence level, the higher the training weight of training sample P1 in the target domain network; the lower the confidence level, the lower the training weight of training sample P1 in the target domain network. The conversion factor between confidence level and training weight can be set according to actual needs, and this embodiment does not limit this.

[0140] In some embodiments, a preset threshold can be set. If the confidence level of the pseudo-label is lower than the preset threshold, the training sample with the added pseudo-label will not be used for network training. The preset threshold can be set according to actual training needs, and this embodiment does not limit it. For example, the preset threshold is 0.3, that is, training samples with pseudo-labels and a confidence level lower than 0.3 are not used for training the target domain network, and training samples with pseudo-labels and a confidence level greater than or equal to 0.3 are configured with different training weights and used for training the target domain network. As the number of training iterations of the target domain network increases, the confidence level of the pseudo-labels added to the unlabeled training samples will change. The target domain network can be trained iteratively multiple times using the training samples of the target data domain to fully utilize the training samples of the target data domain for training the target domain network.

[0141] In some embodiments, the post-processing unit for detection results can be used to combine the detection results of each defect detection head to obtain the final detection result of the mid-frame, such as combining the detection results of the anchor-based detection head, the anchor-free detection head, and the anomaly detection head. For detection results with high confidence, the union method can be used to obtain the final detection result of the mid-frame. For ambiguous detection results, the acquisition parameters of the image acquisition module 102 can be adjusted (e.g., adjusting the illumination intensity of the lighting source, the scanning angle of the line scan camera, etc.) to perform secondary image acquisition and detection.

[0142] For example, each defect detection head can output a detection result and its confidence level. If the confidence level of the detection results from one or more defect detection heads is greater than a preset confidence level, it indicates that the defect detection network has detected a credible defect. In this case, the final detection result of the mid-frame image can be obtained by taking the union of the detection results from all defect detection heads. Conversely, if the confidence level of the detection results from all defect detection heads is less than the preset confidence level, it indicates that the defect detection network has not detected a credible defect. In this case, the final detection result of the mid-frame image can be directly determined as having no defect, and there is no need to take the union. The preset confidence level can be set according to actual needs, and this embodiment does not limit it. For example, the confidence level output by the defect detection head can range from 0 to 1, and the preset confidence level can be set to 0.6.

[0143] For example, for an anchor-based inspection head, an output result of "XX defect, confidence level 0.7" indicates that the anchor-based inspection head detected a reliable XX defect. For an anomaly inspection head, if the output result is "Defect present, confidence level 0.7," it indicates that the anomaly inspection head determines that the product under inspection has a defect, and this result is reliable. If the output result is "Defect present, confidence level 0.4," it indicates that the anomaly inspection head's determination that the product under inspection has a defect is unreliable, and based on this result, it can be concluded that the anomaly inspection head indicates that the product under inspection does not have a defect.

[0144] For example, the defect detection network can also be configured with a first preset confidence level and a second preset confidence level, where the first preset confidence level is greater than the second preset confidence level. If the confidence level of the detection result of at least one defect detection head is greater than or equal to the first preset confidence level, it indicates that the defect detection network has detected a credible defect in the product under test, and the detection result of the product under test can be determined as having a defect. The union of the detection results of each defect detection head can be used as the final detection result of the product under test. If the confidence level of the detection result of each defect detection head is less than the second preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product under test can be determined as not having a defect. If the confidence level of the detection result of at least one defect detection head is less than the first preset confidence level but greater than or equal to the second preset confidence level, and the confidence level of the detection result of no defect detection head is greater than or equal to the first preset confidence level, it indicates that the defect detection network cannot determine whether there is a defect in the product appearance. In this case, the acquisition parameters of the image acquisition module 102 can be adjusted to re-acquire and detect the image of the mid-frame. The values ​​of the first preset confidence level and the second preset confidence level can be set according to actual needs. This application embodiment does not limit this. For example, the confidence level output by the defect detection head is in the range of 0-1, the first preset confidence level can be set to 0.8, and the second preset confidence level can be set to 0.3.

[0145] In some embodiments, the mid-frame post-processing module 104 is used to perform a material unloading operation on the mid-frame based on the final inspection result. For example, the mid-frame post-processing module 104 may include a robotic arm, which controls the robotic arm to complete the material unloading based on the final inspection result of the mid-frame. For defect-free mid-frames, the robotic arm places the mid-frame on a guide rail for transfer to the next assembly station, such as the assembly station of the mid-frame and the touch screen; for defective mid-frames, the robotic arm places the mid-frame in a scrap bin for processing by operators, such as defect verification and defect repair.

[0146] In some embodiments, for mid-frames in the waste bin that are verified by operators to be actually free of defects, labeled training samples can be constructed based on these mid-frames and then input into the defect detection network for optimized training.

[0147] The defect detection network involved in this application includes a training phase and an inference phase, which are described in detail below.

[0148] I. Training Phase

[0149] In this embodiment, the training phase refers to the process by which a network training device trains a domain-adaptive network based on a neural network using training data to obtain a defect detection network. The domain-adaptive network may include a source domain network and a target domain network. The source domain network and the target domain network have the same network structure and share network parameters; for example, the source domain network and the target domain network share weights. The network training device may be a server, a computer, or a server cluster, etc., and this embodiment does not limit this.

[0150] like Figure 11 The diagram illustrates a network training method provided in this application. This method can be applied to network training devices. The order of steps in the method can be adjusted according to actual needs, and this application does not limit this. The method includes the following steps:

[0151] Step S81: Obtain the first training data with labels, and train the domain adaptive network using the first training data.

[0152] In some embodiments, the first training data may be pre-stored in a designated storage area / storage device. For example, the first training data may be a border image of product P1, which may include multiple defective border images and multiple defect-free border images, and is labeled. The first training data can be used to train the source domain network in a domain adaptive network. The first training data can be divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used for network training, the validation set is used to validate the network trained on the training set, and the hyperparameters of the network can be adjusted based on the validation results. The test set is used to test the trained network.

[0153] In some embodiments, after obtaining the first training data, the first training data can be preprocessed first, for example, the first preprocessing includes image data augmentation and normalization, and then the preprocessed first data is divided into a training set, a validation set and a test set, so that the trained network has a stronger generalization ability.

[0154] Step S82: Obtain unlabeled second training data, and use the domain adaptive network trained on the first training data to add pseudo-labels to the second training data. The second training data and the first training data belong to different data domains.

[0155] For example, the first training data is the training samples of the source data domain, and the second training data is the training samples of the target data domain.

[0156] In some embodiments, the second training data is unlabeled training data, which may be bounding box images of product P2 without labels. The amount of data in the second training data can be much smaller than that in the first training data. Adding pseudo-labels to the second training data using a domain adaptive network trained on the first training data can mean: inputting each bounding box image contained in the second training data into the target domain network of the domain adaptive network, and adding the detection results of each bounding box image obtained by the target domain network as labels to the corresponding bounding box images.

[0157] Step S83: Train the domain adaptation network based on the second training data with added pseudo-labels.

[0158] In some embodiments, after adding pseudo-labels to the second training data, the domain adaptive network can be trained based on the pseudo-labeled second training data. For example, the target domain network in the domain adaptive network can be trained, so that the trained domain adaptive network can simultaneously possess the function of detecting bounding box defects in products P1 and P2. The trained domain adaptive network can be used as a defect detection network in the inference stage.

[0159] For example, training weights can be assigned to each training sample with a pseudo-label based on the confidence level of the pseudo-label, so that training samples with high confidence have higher training weights and training samples with low confidence have lower training weights.

[0160] For example, a preset threshold can be set. If the confidence level of the pseudo-label is lower than the preset threshold, the training sample with the pseudo-label is not used for network training. If the confidence level of the pseudo-label is greater than or equal to the preset threshold, corresponding training weights are set for each training sample with the pseudo-label for network training.

[0161] In some embodiments, after adding pseudo-labels to the second training data, the target domain network can be trained by combining the first training data and the second training data with added pseudo-labels, so that the trained domain adaptive network can simultaneously possess the function of detecting bounding box defects in products P1 and P2. The trained domain adaptive network can be used as a defect detection network in the inference stage.

[0162] Before training the target domain network using the second training data with added pseudo-labels, or before training the target domain network using the first training data and the second training data with added pseudo-labels, a first preprocessing can be performed on the second training data with added pseudo-labels, or on the first training data and the second training data with added pseudo-labels. For example, the first preprocessing includes image data augmentation and normalization, followed by the division of the training set, validation set, and test set, so that the trained network has a stronger generalization ability.

[0163] In some embodiments, the network training method may further include step S84, which can reduce the feature differences between the source data domain and the target data domain and improve the cross-domain adaptability of the network.

[0164] Step S84: Construct a third training data containing training samples from different data domains, and use the third training data to optimize and train the domain adaptive network. The training samples belonging to different data domains in the third training data satisfy the first preset condition.

[0165] In some embodiments, the third training data may include multiple training samples from the source data domain and multiple training samples from the target data domain. The first precondition may be to minimize the feature mean of the training samples from the source data domain and the feature mean of the training samples from the target data domain based on the MMD algorithm. Then, the third training data is used to optimize and train the domain adaptive network; for example, the third training data is used to optimize and train the target domain network within the domain adaptive network. The trained domain adaptive network can be used as a defect detection network in the inference stage.

[0166] In some embodiments, the network training method may further include step S85, which can further reduce the feature differences between the source data domain and the target data domain, and improve the network's cross-domain adaptability.

[0167] Step S85: Construct a fourth training data containing training samples from different data domains, and use the fourth training data to optimize and train the domain adaptive network. The training samples belonging to different data domains in the fourth training data satisfy the second preset condition.

[0168] In some embodiments, the fourth training data may include multiple training samples from the source data domain and multiple training samples from the target data domain. The second precondition may be based on a domain classification algorithm that makes the training samples from the source data domain and the target data domain indistinguishable (unable to classify the source and target data domains). The fourth training data is then used to optimize and train the domain adaptive network; for example, the fourth training data can be used to optimize and train the target domain network within the domain adaptive network. The trained domain adaptive network can then be used as a defect detection network in the inference phase.

[0169] In some embodiments, optimizing the training of the domain adaptive network can be performed by executing one of steps S84 and S85, or both steps. The execution order of steps S84 and S85 can be parallel, or step S84 can be executed before or after step S85.

[0170] In some embodiments, a first loss function can be constructed during the training of the source domain network using the first training data. A second loss function can be constructed during the training of the target domain network using the second training data, or during the joint training of the target domain network using the first and second training data. Simultaneously, to reduce the feature differences between the source and target data domains, a third loss function can be constructed based on the MMD algorithm, minimizing the difference between the feature mean of the training samples in the source data domain and the feature mean of the training samples in the target data domain in a batch. Furthermore, a fourth loss function can be constructed based on a domain classification algorithm, preventing the training samples in the source and target data domains from being classified into different domains. The total loss function for training the defect detection network can then be composed of the first to fourth loss functions described above.

[0171] In some embodiments, the construction of the first to fourth loss functions can be set according to the actual network training requirements. This application embodiment does not limit this. For example, the first, second and fourth loss functions can be loss functions based on logistic regression, and the third loss function can be the MMD loss function.

[0172] In some embodiments, the defect detection network may include multiple defect detection heads, each of which can be constructed based on network features at different levels and / or scales. For example, both the source domain network and the target domain network in the defect detection network include anchor-based detection heads, anchor-free detection heads, and anomaly detection heads. Anchor-based detection heads are suitable for detecting mid-frame defects with a moderate aspect ratio. Anchor-free detection heads are suitable for detecting mid-frame defects with an unbalanced aspect ratio. Anomaly detection heads are used for binary classification training to identify whether there are defects in the mid-frame image, thereby reducing the risk of the detection network missing new types of mid-frame defects.

[0173] The defect detection network can identify the types of mid-frame defects learned during the training phase using anchor-based and anchor-free detectors. In other words, the anchor-based and anchor-free detectors output the specific defect types present in the mid-frame image. The anomaly detector can identify mid-frame defect types not learned during training. That is, for new types of mid-frame defects, although the anchor-based and anchor-free detectors cannot identify them, the anomaly detector can identify the presence of defects in the mid-frame image (defect type unknown). In this case, manual verification is possible, and if it is determined to be a new type of mid-frame defect, a label is added to the mid-frame image. This labeled mid-frame image can then be input into the defect detection network for training, enabling the mid-frame defect of this type to be subsequently identified by the anchor-based and / or anchor-free detectors.

[0174] After the aforementioned training phase, the network training equipment can be deployed on a defect detection device on the product assembly line. This defect detection device can be a computer (e.g., a personal computer, industrial computer, etc.), a server, or other similar equipment. The network training equipment and the defect detection device can be the same or different devices; this embodiment does not limit this. The following describes the network application process, that is, the process by which the defect detection device performs inference based on the defect detection network.

[0175] II. Reasoning Stage

[0176] like Figure 12 The diagram illustrates a defect detection method provided in this application. The order of steps in the method can be adjusted according to actual needs, and this application does not limit this. This method can be applied to a defect detection device, which can be equipped with a defect detection network trained using the aforementioned network training method. The method includes the following steps:

[0177] Step S91: Acquire the image of the frame to be detected.

[0178] For example, the image of the frame to be detected can be acquired through the image acquisition module 102 described above.

[0179] Step S92: Input the acquired image into the defect detection network.

[0180] In some embodiments, since the images acquired by the line scan camera are large in size, the acquired images can be preprocessed before being input into the defect detection network. The preprocessing includes, but is not limited to, grayscale conversion processing and image cropping and stitching processing. Image cropping and stitching processing can refer to cropping the acquired image into multiple sub-images in the length direction of the image and stitching at least two of the multiple sub-images in the width direction, and then inputting the stitched image into the defect detection network.

[0181] The defect detection network can extract image features and identify defects to obtain the detection result of the bounding box to be detected. The detection result output by the defect detection network can be of two types. The first type is that the defect detection network can determine whether there is a defect in the bounding box to be detected. The detection result can include three types: no defect; defect exists and the defect type is XX; defect exists and the defect type is unknown. The defect detection network can output different vectors to represent different detection results. For example, in the case of no defect, the Anomaly detection head can output a vector identifier indicating that no defect exists, and both the Anchor-based detection head and the Anchor-free detection head can output vector identifiers indicating that no defect was detected. For example, in the case of defect exists and the defect type is XX, the Anomaly detection head can output a vector identifier indicating that the defect exists, and the Anchor-based detection head and / or the Anchor-free detection head can output vector identifiers indicating the type of defect detected. For example, in the case of defect exists and the defect type is unknown, the Anomaly detection head can output a vector identifier indicating that the defect exists, and both the Anchor-based detection head and the Anchor-free detection head can output vector identifiers indicating that no defect was detected.

[0182] Another type is where the defect detection network cannot determine whether there is a defect in the frame image. In this case, the detection result can be identified by a preset vector output by the network. This indicates that the acquisition parameters of the image acquisition module 102 need to be adjusted (e.g., adjusting the light intensity of the lighting source, the scanning angle of the line scan camera, etc.) to perform secondary image acquisition and recognition of the frame to be detected.

[0183] Step S93: Obtain the detection result of each of the multiple defect detection heads for the image of the bounding box to be detected, and take the union of the detection results of each defect detection head as the detection result of the bounding box to be detected.

[0184] In some embodiments, after the acquired image is input into the defect detection network, each defect detection head of the defect detection network can output the corresponding detection result for the input image.

[0185] In some embodiments, after the acquired image is input into the defect detection network, each defect detection head of the defect detection network can output the corresponding detection result and the confidence level of the detection result for the input image.

[0186] In some embodiments, if the confidence level of the detection result of at least one defect detection head is greater than or equal to a preset confidence level, it indicates that the defect detection network has detected a credible defect. The union of the detection results of each defect detection head can be used as the detection result of the bounding box to be detected. For example, the defect detection network can perform the union of the detection results of each defect detection head and output it, or a computer device can perform the union of the detection results of each defect detection head and output it.

[0187] In some embodiments, if the confidence level of the detection result of each defect detection head is less than the preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the defect detection network can directly determine that the final detection result of the mid-frame image is that there is no defect.

[0188] In some embodiments, the defect detection network may also be configured with a first preset confidence level and a second preset confidence level, wherein the first preset confidence level is greater than the second preset confidence level. If the confidence level of the detection result of at least one defect detection head is greater than or equal to the first preset confidence level, it indicates that the defect detection network has detected a credible defect in the product under test, and the union of the detection results of each defect detection head can be used as the final detection result of the product under test. If the confidence level of the detection result of each defect detection head is less than the second preset confidence level, it indicates that the defect detection network has not detected a credible defect, and the detection result of the product under test can be directly determined as having no defect. If the confidence level of the detection result of at least one defect detection head is less than the first preset confidence level but greater than or equal to the second preset confidence level, and the confidence level of the detection result of any defect detection head is not greater than or equal to the first preset confidence level, it indicates that the defect detection network cannot determine whether there is a defect in the product appearance. The defect detection network can output a vector identification information of the detection anomaly. The computer device can output a preset detection anomaly prompt information based on the vector identification information. For example, the preset detection anomaly prompt information is "Unable to determine whether there is a defect in the product appearance. Please perform secondary image acquisition and inspection." The operator can adjust the acquisition parameters of the image acquisition device (e.g., the shooting angle of the line scan camera, the brightness of the ambient light source, etc.) to re-acquire and inspect the product to be inspected. The value of the preset confidence level can be set according to actual needs.

[0189] In some embodiments, after obtaining the detection result of the mid-frame to be inspected, different processing methods can be performed based on the detection result of the mid-frame to be inspected. For example, for mid-frames without defects, they can be transferred to the next assembly station; for mid-frames with defects, they can be placed in a scrap bin and wait for operators to process them, such as performing defect verification and defect repair.

[0190] In some embodiments, for mid-frames in the waste bin that are verified by operators to be actually free of defects, labeled training samples can be constructed based on these mid-frames to optimize and train the defect detection network.

[0191] This application also provides a defect detection device; please refer to [link to relevant documentation]. Figure 13 , Figure 13 This is a schematic diagram of a defect detection device provided in an embodiment of this application, wherein the defect detection device 100 may be deployed with the defect detection network described in the above embodiments. Specifically, the defect detection device 100 includes: a receiver 111, a transmitter 112, a processor 113, and a memory 114 (wherein, the number of processors 113 in the defect detection device 100 may be one or more). Figure 13(Taking a processor as an example), processor 113 may include application processor 1131 and communication processor 1132. In some embodiments of this application, receiver 111, transmitter 112, processor 113 and memory 114 may be connected via a bus or other means.

[0192] Memory 114 may include read-only memory and random access memory, and provides instructions and data to processor 113. A portion of memory 114 may also include non-volatile random access memory (NVRAM). Memory 114 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0193] Processor 113 controls the operation of the defect detection equipment. In specific applications, the various components of the defect detection equipment are coupled together through a bus system. This bus system includes not only a data bus but also a power bus, control bus, and status signal bus. However, for clarity, all buses are referred to as the bus system in the diagram.

[0194] The above embodiments of this application Figure 12The disclosed method can be applied to or implemented by processor 113. Processor 113 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 113 or by instructions in software form. Processor 113 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 113 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 114, and processor 113 reads the information in memory 114 and, in conjunction with its hardware, completes the steps of the above method.

[0195] Receiver 111 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the defect detection equipment. Transmitter 112 can be used to output digital or character information through the first interface; transmitter 112 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 112 may also include a display device such as a display screen.

[0196] In one embodiment of this application, the application processor 1131 is configured to execute... Figure 12 The defect detection method in the corresponding embodiment. It should be noted that, for application processor 1131 executing... Figure 12 For details on the specific implementation of the defect detection method in the corresponding embodiments and its beneficial effects, please refer to [the relevant documentation / reference]. Figure 12 The descriptions in the corresponding method embodiments will not be repeated here.

[0197] Please refer to the following: Figure 14 , Figure 14 This is a schematic diagram of a network training device provided in an embodiment of this application.

[0198] Network training device 200 can be used to achieve Figure 11 The corresponding method embodiment. Specifically, the network training device 200 is implemented by one or more servers or computers. The network training device 200 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 210 (e.g., one or more processors) and memory 211, and one or more storage media 214 (e.g., one or more mass storage devices) for storing application programs 212 or data 213. The memory 211 and storage media 214 can be temporary or persistent storage. The program stored in the storage media 214 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the network training device. Furthermore, the CPU 210 may be configured to communicate with the storage media 214 and execute the series of instruction operations in the storage media 214 on the network training device 200.

[0199] Network training device 200 may also include one or more power supplies 215, one or more wired or wireless network interfaces 216, one or more input / output interfaces 217, and / or one or more operating systems 218, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0200] In one embodiment of this application, the central processing unit 210 can be used to implement... Figure 11 Corresponding method implementation examples.

[0201] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps performed by the network training device in the above embodiments, or to perform the steps performed by the defect detection device in the above embodiments.

[0202] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the network training device described above, or steps as performed by the defect detection device described above.

Claims

1. A defect detection method applied to computer equipment, characterized in that, The computer device is deployed with a pre-trained defect detection network, which includes multiple defect detection heads. These multiple defect detection heads include a first defect detection head for detecting aspect ratios within a preset range, a second defect detection head for detecting aspect ratios outside the preset range, and a third defect detection head for binary classification (defective or non-defective). The third detection head is also used to detect defects not detected by the first and second detection heads. The defect detection method includes: Acquire an image of the product to be inspected; The appearance image is input into the defect detection network; Obtain the detection result of each of the plurality of defect detection heads for the appearance image and the confidence level of the detection result; If the confidence level of the detection result of at least one defect detection head is greater than or equal to the preset confidence level, the detection result of the product to be tested is determined to be defective, and the union of the detection results of each defect detection head is taken as the detection result of the product to be tested.

2. The defect detection method as described in claim 1, characterized in that, The step of inputting the appearance image into the defect detection network includes: The appearance image is preprocessed, including grayscale conversion and image cropping and splicing. The image cropping and splicing includes cropping the appearance image into multiple sub-images in the length direction and splicing at least two of the multiple sub-images in the width direction. The preprocessed appearance image is input into the defect detection network.

3. The defect detection method as described in claim 1, characterized in that, The method further includes: If the confidence level of the detection result of each defect detection head is less than the preset confidence level, the detection result of the product to be tested is determined to be that there is no defect.

4. The defect detection method according to any one of claims 1 to 3, characterized in that, The computer device is communicatively connected to the image acquisition device, which includes a line scan camera. Before acquiring an image of the appearance of the product to be inspected, the defect detection method further includes: Obtain the motion trajectory coordinates of the product to be tested; Based on the RTCP algorithm of the rotating tool center point and the motion trajectory coordinates, the line scan camera is controlled to perform interpolation motion, so that the distance between the line scan camera and the product to be inspected is maintained within a preset distance, which is set based on the focal length of the line scan camera.

5. The defect detection method as described in claim 4, characterized in that, The method of controlling the line scan camera to perform interpolation motion based on the RTCP algorithm of the rotating tool center point and the motion trajectory coordinates includes: The image acquisition endpoint of the line scan camera is calibrated as the blade tip following point in the RTCP algorithm, the product to be detected is calibrated as the processed product in the RTCP algorithm, and the shooting point on the product to be detected is calibrated as the actual contact point between the blade tip and the processed product in the RTCP algorithm. Based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed, the line scan camera is controlled to perform interpolation motion, so that the distance between the image acquisition endpoint and the point to be photographed is within the preset distance.

6. The defect detection method as described in claim 5, characterized in that, The step of controlling the line scan camera to perform interpolation motion based on the RTCP algorithm and the motion trajectory coordinates of the point to be photographed includes: Construct a camera coordinate system corresponding to the line scan camera; Transform the motion trajectory coordinates of the point to be photographed to the camera coordinate system to obtain the target trajectory coordinates of the point to be photographed in the camera coordinate system. The line scan camera is controlled to perform interpolation motion based on the RTCP algorithm and the target trajectory coordinates.

7. The defect detection method according to any one of claims 1 to 3, characterized in that, The defect detection network is trained based on a training sample set and a preset domain adaptive network. The training sample set includes a labeled first sample set and an unlabeled second sample set, and the second sample set and the first sample set belong to different data domains. The training sample set also includes a third sample set, which is constructed based on the first sample set and the second sample set. The third sample set undergoes domain classification processing to minimize the differences in image features between training samples belonging to different data domains in the third sample set.

8. The defect detection method as described in claim 7, characterized in that, The training sample set also includes a fourth sample set, which is constructed based on the first sample set and the second sample set. The fourth sample set is processed by maximum mean difference (MMD) to minimize the MMD of training samples belonging to different data domains in the fourth sample set.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory is used to store computer-readable instructions; when the computer-readable instructions are executed by the processor, the computer device causes the computer device to implement the defect detection method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The device stores computer-readable instructions that, when executed by a processor, implement the defect detection method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Product identification method and device, model training method and device and electronic equipment

    CN114419035A

  • Scratch detection method and device, equipment and storage medium

    CN114998337A

  • Humanoid three-dimensional scanning motion control method and device based on point cloud

    CN116690557A