A defect detection system and method based on cloud-edge collaboration

Abstract

The present invention discloses a defect detection system and method based on cloud-edge collaboration, the system comprising: a client, a cloud, and an edge; the client is used to obtain the original image uploaded by the user; the edge comprises a first basic defect detection model and at least one second customized defect classification model; the present invention adopts a solution combining a basic large model and a customized small model, and deploys a basic defect detection model including all defect detections and a customized defect classification model composed of multiple classification models on the edge; the edge only needs to arrange the model, perform real-time detection according to the arranged model, upload labeled samples and detection results, and hand over the complex model training work to the cloud, thereby reducing the cost of hardware construction on the edge; and because the model architecture of the present invention is based on a modular design, only the customized defect classification model can be customized and updated according to customer needs, so it has high flexibility.

Description

Technical Field

[0001] The present invention relates to the field of automated quality inspection technology in the field of advanced manufacturing, and more specifically, to a defect detection system and method based on cloud-edge collaboration. Background Art

[0002] In modern industrial production, artificial intelligence technology is increasingly used, especially in the field of industrial automation technology, where artificial intelligence can greatly improve production efficiency and product quality. The rapid development of cloud computing and big data technologies has made real-time data analysis and processing of industrial production possible. Therefore, the hybrid solution of cloud-edge collaboration has become an important trend in the application of industrial artificial intelligence. Cloud-edge collaboration means that edge devices collect industrial production data and transmit the data to the cloud through the network for storage and analysis, thereby achieving efficient data management and model training. The existing hybrid cloud-edge collaboration solution is mainly based on large models for defect detection and classification of industrial production. Large models require a large amount of data during training and have strong generalization capabilities, which can handle a variety of different defects and abnormal situations. However, this solution has some problems in model update and data transmission. For example, when customer demand or sample data changes slightly, the large model needs to be updated, but this may cause the disaster of memory loss and seriously affect the detection effect. In addition, the amount of industrial production data is very large. If all of it is uploaded to the cloud, it will face the problems of real-time data transmission and bandwidth limitation. Summary of the invention

[0003] In response to at least one defect or improvement need in the prior art, the present invention provides a deep learning system and method based on cloud-edge collaboration, which adopts a solution combining a basic large model and a customized small model, and deploys a basic defect detection model including all defect detections and a customized defect classification model composed of multiple classification models at the edge. The edge only needs to receive the model, perform real-time detection according to the received model, upload samples and detection results, and hand over the complex model training work to the cloud, reducing the cost of edge hardware construction. In addition, since the model architecture of the present invention is based on a modular design, it can only customize and update the customized defect classification model according to customer needs, so it has high flexibility.

[0004] To achieve the above-mentioned object, according to a first aspect of the present invention, a defect detection system based on cloud-edge collaboration is provided, the system comprising: a client, a cloud, and an edge;

[0005] The client is used to obtain the original image uploaded by the user;

[0006] The edge end includes a first basic defect detection model and at least one second customized defect classification model, wherein the second customized defect classification model is a defect classification model designed based on a specific defect determination standard;

[0007] The edge end is used to obtain the original image uploaded by the user from the client, and divide the original image into multiple slices and send them to the first basic defect detection model, the first basic defect detection model is used to perform preliminary defect detection on the slices to obtain a first detection result, the first detection result includes multiple first images with defects identified, and a first defect category and a first confidence level corresponding to the first image; the edge end is also used to send the first detection result to the corresponding second customized defect classification model according to the first defect category;

[0008] The second customized defect classification model is used to obtain a first image of a corresponding category, and perform defect classification on the first image to obtain a second classification result, wherein the second classification result includes a second image identified as having defects and a second classification category and a second confidence level corresponding to the second image;

[0009] The cloud is used to obtain the first detection result and / or the second classification result from the edge, train the corresponding first basic defect detection model and / or the second customized defect classification model, and update the first basic defect detection model and / or the second customized defect classification model of the edge.

[0010] According to the defect detection system based on cloud-edge collaboration provided by the present invention, the cloud is used to obtain the first detection result and / or the second classification result from the edge, and train the first basic defect detection model and / or the second customized defect classification model, specifically:

[0011] The cloud is used to obtain a first confidence interval and a second confidence interval;

[0012] The cloud is used to perform sample annotation on the first image whose first confidence does not belong to the first confidence interval to form a first sample for training the first basic defect detection model;

[0013] The cloud is used to perform sample annotation on the second image whose second confidence does not belong to the second confidence interval to form a second sample for training the second customized defect classification model.

[0014] According to the defect detection system based on cloud-edge collaboration provided by the present invention, the cloud is also used to receive the third sample uploaded by the client for training the first basic defect detection model, and the third sample is formed by performing sample annotation on the segmented image corresponding to the missed detection or wrong detection of the first basic defect detection model; wherein the wrong detection refers to the detection that the identified defect belongs to the real defect but the defect category is incorrectly identified; the cloud is also used to receive the fourth sample uploaded by the client for training the second customized defect classification model, and the fourth sample is a customized defect classification model of the basic defect detection model formed by performing sample annotation on the segmented image corresponding to the missed detection, over-detection or wrong detection of the second customized defect classification model.

[0015] According to the defect detection system based on cloud-edge collaboration provided by the present invention, the cloud is used to train the first basic defect detection model when the total amount of the first and third samples received is greater than the first preset value and the first basic defect detection model training instruction is received; the cloud is used to train the second customized defect classification model when the total amount of the second and fourth samples received is greater than the second preset value; wherein the first preset value is greater than the second preset value.

[0016] According to the cloud-edge collaboration-based defect detection system provided by the present invention, the specific defect judgment standard is: judging and classifying point defects based on the brightness level of the defects; the second customized defect classification model judges and classifies point defects in the first image according to the brightness level.

[0017] According to the defect detection system based on cloud-edge collaboration provided by the present invention, the cloud is also used to perform performance testing on the trained first basic defect detection model and / or the second customized defect classification model and obtain performance indicators; the cloud is also used to update the first basic defect detection model and / or the second customized defect classification model at the edge when the performance indicators meet preset standards.

[0018] According to a second aspect of the present invention, a defect detection method based on cloud-edge collaboration is also provided, the method comprising:

[0019] The edge end obtains the original image uploaded by the user from the client, and divides the original image into multiple slices and sends them to the first basic defect detection model; the edge end includes a first basic defect detection model and at least one second customized defect classification model, and the second customized defect classification model is a defect classification model designed based on a specific defect judgment standard;

[0020] The first basic defect detection model at the edge performs preliminary defect detection on the slice to obtain a first detection result, wherein the first detection result includes a plurality of first images with defects identified, and a first defect category and a first confidence level corresponding to the first image; the edge is further configured to send the first detection result to the corresponding second customized defect classification model according to the first defect category;

[0021] The second customized defect classification model at the edge acquires a first image of a corresponding category, and performs defect classification on the first image to obtain a second classification result, where the second classification result includes a second image identified as having defects and a second classification category and a second confidence level corresponding to the second image;

[0022] The edge sends the first detection result and / or the second classification result to the cloud for the cloud to train the corresponding first basic defect detection model and / or the second customized defect classification model, which is used to update the first basic defect detection model and / or the second customized defect classification model of the edge.

[0023] According to the cloud-edge collaboration-based defect detection method provided by the present invention, the method further includes:

[0024] The cloud obtains a first confidence interval and a second confidence interval;

[0025] The cloud performs sample annotation on the first image whose first confidence does not belong to the first confidence interval to form a first sample for training the first basic defect detection model;

[0026] The cloud performs sample annotation on the second image whose second confidence does not belong to the second confidence interval to form a second sample for training the second customized defect classification model.

[0027] The defect detection method based on cloud-edge collaboration provided by the present invention also includes:

[0028] The cloud receives a third sample uploaded by the client for training the first basic defect detection model, wherein the third sample is formed by annotating samples of the segmented images corresponding to missed detection or false detection by the first basic defect detection model; wherein the false detection refers to a detection in which the identified defect belongs to a real defect but the defect category is incorrectly identified;

[0029] The cloud is also used to receive a fourth sample uploaded by the client for training the second customized defect classification model. The fourth sample is formed by annotating samples of the segmented images corresponding to missed detection, over-detection or wrong detection of the second customized defect classification model.

[0030] The defect detection method based on cloud-edge collaboration provided by the present invention also includes:

[0031] The cloud trains the first basic defect detection model when the total amount of the first sample and the third sample received is greater than the first preset value and a first basic defect detection model training instruction is received;

[0032] The cloud trains the second customized defect classification model when the total amount of the second sample and the fourth sample received is greater than a second preset value;

[0033] Wherein, the first preset value is greater than the second preset value.

[0034] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0035] (1) The present invention provides a defect detection system and method based on cloud-edge collaboration, which fully utilizes the computing power and storage resources of the cloud, completes model training in the cloud, and screens and uploads samples on the edge through a two-level model, thus achieving a balance between model update and data transmission. This architecture can not only quickly respond to customer needs, but also avoid data transmission bottlenecks.

[0036] (2) The present invention provides a defect detection system and method based on cloud-edge collaboration, which divides the defect detection model into a first basic defect detection model and a second customized defect classification model. The first basic defect detection model is a large model, which only performs preliminary defect detection and classification. The second customized defect classification model is a small model, which performs further refined classification, thereby ensuring the accuracy of the defect detection results. In addition, the two models can be trained independently. When there is a slight change in customer requirements or a slight change in sample data, only the second customized defect classification model needs to be updated, thereby solving the problem of difficulty in model updating.

[0037] (3) The present invention provides a defect detection system and method based on cloud-edge collaboration, which automatically uploads and labels typical samples with high confidence and difficult samples with low confidence that appear on site to the cloud, and labels samples whose manual review results and test results are inconsistent differently, thus solving the problem of sample labeling selection.

[0038] (4) The present invention provides a defect detection system and method based on cloud-edge collaboration, which can mark different samples according to the results of manual re-judgment and the defect detection results of the two-level model, so that the problem samples can be automatically assigned to different models for training, making the training more targeted. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a framework of a defect detection system based on cloud-edge collaboration provided by an embodiment of the present invention;

[0041] Figure 2 A module schematic diagram of a defect detection system based on cloud-edge collaboration provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a first basic defect detection model training method provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a second customized defect classification model training method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] The terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0046] like Figure 1-2 As shown, as an embodiment of the present invention, a defect detection system based on cloud-edge collaboration is provided. The system is used in the field of industrial inspection and uses image recognition to detect surface defects of industrial products. Therefore, the defect detection system of this embodiment includes an edge terminal, a cloud terminal, and a client terminal.

[0047] The cloud, client, and edge are connected to each other through network transmission, wherein the client is connected to the cloud to realize the interaction between the user and the cloud. For example, the detection results can be read, control instructions can be input, and the original image to be detected can be uploaded. The edge is also connected to the cloud to realize the functions such as defect detection in this embodiment. Network transmission can be realized by switches, routers, etc.

[0048] The cloud, client, and edge are not limited to a one-to-one correspondence. For example, one cloud may correspond to multiple edge. In some embodiments, multiple edge can be arranged on multiple production lines respectively. These edge share the cloud, and after using the computer performance of the cloud to perform model training and other tasks, the trained models are sent to different edge. The client is not limited to one. For example, multiple staff members can use different clients at the same time to connect to the cloud in different occasions, input commands to the cloud, and obtain results.

[0049] The cloud can be a high-performance server or server cluster. In this embodiment, it undertakes complete model-related tasks, such as model training, model pruning, etc., as well as complete data-related tasks, such as data uploading and downloading, data preview and annotation, and load balancing tasks, such as multi-model concurrent training, multi-task concurrent processing, etc.

[0050] The edge end can be a lower-performance edge computing end, which can be a computer containing a low-end graphics card or an embedded edge end. The edge end in the present invention includes a first basic defect detection model and at least one second customized defect classification model, wherein the second customized defect classification model is a defect classification model designed based on a specific defect judgment standard. The edge end can infer the model of the input image, output the result, and upload specific samples to the server according to the set rules and policies. At the same time, the edge end device also supports the function of actively acquiring the image to be detected from the image acquisition device or a specific location.

[0051] The client is mainly composed of some low-performance computers. In an office far away from the production line, any laptop or desktop can be connected to the server to upload the original image to be inspected for defects, manually review the results of terminal inspection on the server, and manually mark samples that are missed by the terminal. The following is a more detailed explanation.

[0052] First, the user uploads the original image to be inspected through the client. In this embodiment, the original image can be collected by an image acquisition device. Generally speaking, the image acquisition device uses a high-resolution industrial camera, so the resolution of the original image is very high, for example, it can reach a resolution of 10000*10000, and the original image can be an image containing various defects, such as scratches of various sizes, dirt, dust, etc.

[0053] Since the defects on the surface of the object to be tested are usually not large-sized defects, in the original image, the defects may only be dozens by dozens of resolutions in size, and the number of defects is small, which makes it inefficient and very demanding on the model to directly use the original image to perform accurate defect detection and defect classification of various defect types in one step. In addition, after the defects are detected, directly uploading the original image as a sample will take up a lot of bandwidth, resulting in the data on the edge end not being uploaded to the cloud service end in real time. Based on this, the edge end needs to cut the original image into multiple slices before performing defect detection. In some embodiments, the slice can be an image with a resolution of 512*512.

[0054] The first basic defect detection model is used to perform preliminary defect detection on the slices to obtain a first detection result, which includes identifying a plurality of first images with defects, and a first defect category and a first confidence level corresponding to the first images.

[0055] The first basic defect detection model in the present invention is a basic large model that can detect all abnormalities and defects of various sizes, and classify the detection results into N major categories, where all abnormalities and defects of various sizes can be pre-trained and set. Since the model needs to be able to detect more types of defects, the model is large and complex, and the model training will take a long time.

[0056] It is understandable that the purpose of the first basic defect detection model is to accurately detect as many defects as possible. There may be a large number of over-inspections, and the defect classification ability is poor, which may easily misclassify the defects. Therefore, it is not enough to use a first basic defect detection model for classification. A second-level model, that is, a second customized defect classification model, is required to perform more detailed classification judgments on the first detection results output by the first basic defect detection model.

[0057] The second customized defect classification model is a defect classification model designed based on specific defect judgment criteria. Different defect judgment criteria will be set in different scenarios. For example, when only line defects need to be concerned, the brightness and direction of line defects can be used as the classification criteria for defects, and line defects can be divided into bright lines, dark lines, horizontal lines and vertical lines; when only point defects are concerned, the brightness of point defects can be used as the judgment criteria to divide point defects into bright point defects and dark point defects, and point defects can also be divided into single point defects and multi-point defects based on the number of adjacent pixels where defects occur. After the defect judgment criteria are determined, it is necessary to organize and divide the samples and perform model training according to these criteria to obtain a defect classification model designed based on specific defect judgment criteria. The second customized defect classification model can be multiple, and can include defect classification models designed for point defects, defect classification models designed for line defects, defect classification models designed for mura defects, etc.

[0058] The edge end is further used to send the first detection result to the corresponding second customized defect classification model according to the first defect category. For example, the edge end can send the first detection result determined as a point defect to the second customized defect classification model designed for point defects, and send the first detection result determined as a line defect to the second customized defect classification model designed for line defects.

[0059] The second customized defect classification model classifies the defects of the acquired first image again to obtain a second classification result, which includes a second image detected with defects and a second classification category, defect level and second confidence corresponding to the second image. The second image is the first image identified as having defects by the second customized defect classification model.

[0060] As an optional embodiment, when the specific defect judgment standard is to judge and classify point defects based on the brightness level of the defects, the second customized defect classification model determines whether the point defects in the first image are defects according to the brightness level of the defects, and divides the determined defects into different defect levels.

[0061] Specifically, the first basic defect detection model may first judge some weak defects as defects. For example, all points with brightness above or below a certain threshold are considered as point defects. However, the defect judgment criteria of different users or the same user will change with demand. For example, different users have different requirements for brightness thresholds. Some users will appropriately relax the requirements for this threshold. As long as it does not affect the use, it will not be considered a defect. At this time, it is necessary to adjust the defect judgment criteria of the second customized defect classification model, and do not regard points with brightness thresholds within a certain range as defects. In this case, when the second customized defect classification model performs defect detection classification, it will output different second classification results due to changes in the defect judgment criteria.

[0062] After the above defect detection is completed, the edge will upload the first detection result and the second classification result to the cloud, and the first detection result and the second classification result will be stored in the cloud. The staff can read the first detection result and the second classification result in the cloud at any time through the client to obtain relevant information about the defect detection. The first image and the second image in the first detection result and the second classification result will be used as candidate samples to train the first basic defect detection model and the second customized defect classification model.

[0063] After uploading, the first image and the second image are usually not directly used as samples to train the first basic defect detection model and the second customized defect classification model in the cloud. There are a large number of images that have no training significance in the first detection results and the second classification results, which will lead to poor results in training the first basic defect detection model and the second customized defect classification model. In this embodiment, suitable images are screened by confidence for sample annotation and used as samples for training. The cloud annotates the first image whose first confidence does not belong to the first confidence interval to form a first sample for training the first basic defect detection model; the cloud annotates the second image whose second confidence does not belong to the second confidence interval to form a second sample for training the second customized defect classification model.

[0064] Specifically, in this embodiment, it is necessary to screen samples with high confidence and low confidence. Samples with high confidence are used as typical samples to consolidate the model's detection capabilities for typical samples and ensure that the model is accurate for such samples. Samples with low confidence are used as difficult samples to improve the model's detection capabilities for difficult samples and improve the generalization of the model. In actual use, "high confidence" and "low confidence" can be modified by staff by sending instructions to the cloud through the client. The cloud will receive a first confidence interval and a second confidence interval, which are used to screen the first image in the first detection result and the second image in the second classification result, respectively. The first confidence interval and the second confidence interval described in this embodiment are generally intervals where the confidence of more samples is concentrated. For example, for the first detection result, the confidence interval can be As the first confidence interval, those skilled in the art can understand that among the first images screened out, samples with a confidence interval less than or equal to 0.1 are the above-mentioned low-confidence samples, and samples with a confidence interval greater than or equal to 0.9 are the above-mentioned high-confidence samples. By labeling the above two types of samples, they are used together as the first samples for training the first basic defect detection model. The second confidence interval can be used to label the second sample for training the second customized defect classification model in the same way, which will not be repeated here.

[0065] In this embodiment, the first confidence interval and the second confidence interval are not required to be exactly the same. As an implementation method, the first confidence interval and the second confidence interval can be set arbitrarily by the staff. For example, the first confidence interval and the second confidence interval can be uploaded to the cloud through the client. In other implementation methods, the cloud can calculate the mean of the first confidence interval when receiving a certain amount of the first detection results and the second classification results. μ 1 and variance σ 1. Calculate the mean μ2 and variance σ2 of the second confidence level, and construct the first confidence interval and the second confidence interval in a combination of the mean and variance. For example, the first confidence interval is constructed as , construct the second confidence interval as ,in m , n They are all positive integers, for example, 1, 2, or 3. After the cloud obtains the first confidence interval and the second confidence interval by internal calculation, the above steps are performed to obtain the first sample and the second sample.

[0066] As an optional embodiment, the edge end can also update the first detection result according to the second classification result. Since the first basic defect detection model has strong versatility, the probability of misjudgment is very high, and the second customized defect classification model has a more accurate classification result than the first basic defect detection model. Therefore, when the second customized defect classification model performs defect detection classification, it is necessary to determine whether the first detection result is a misjudgment. When the second customized defect classification model outputs the second classification result, if it is determined that the first detection result is a misjudgment, the misjudged first detection result can be updated.

[0067] As an optional embodiment, the cloud in the present invention is also used to receive the results of manual re-judgment, construct samples with segmented images that are NG in manual re-judgment but not detected by the first basic defect detection model and the second customized defect classification model, and mark the samples differently for training the first basic defect detection model or the second customized defect classification model. The manual re-judgment step means that after the defect detection system of this embodiment has been detected, a secondary judgment can be made by manual sampling, and pictures with defects can be screened out. If the screened defective pictures are found to have missed detections when performing defect detection in the defect detection system, it means that there are loopholes in the detection of this type of defect by the first basic defect detection model or the second customized defect classification model, and the picture can be used as a training sample to train and update the first basic defect detection model and / or the second customized defect classification model. Table 1 lists the judgment logic of whether to issue a manual command in various situations. The manual command is a command to upload the corresponding segmented image of the object to be tested issued by the client to the edge end when a defect of the object to be tested is found in the manual re-judgment stage and the first detection result or the second classification result does not include the defect.

[0068] Table 1 Relationship between defect detection system and manual review under various circumstances

[0069]

[0070] It can be seen that in the first case, due to the second customized defect classification model, the defects that have been detected by the first basic defect detection model are excluded from the second classification result, resulting in missed detection. At this time, it is necessary to instruct the image to be uploaded to the cloud and mark the image as the fourth sample for training the second customized defect classification model. In the third case, due to the first basic defect detection model, the defect is no longer included in the first detection result, resulting in missed detection. At this time, it is necessary to instruct the image to be uploaded to the cloud and mark the image as the third sample for training the first basic defect detection model.

[0071] As another optional method of manual re-evaluation, the cloud in the present invention is also used to receive a third sample uploaded by the client for training the first basic defect detection model; the third sample is formed by annotating samples of the segmented images corresponding to the missed detection or wrong detection of the first basic defect detection model; wherein the wrong detection refers to a detection in which the identified defect belongs to a real defect but the defect category is incorrectly identified.

[0072] The third samples here include not only samples missed by the first basic defect detection model, but also samples detected by mistake, which can not only improve the generalization ability of the first basic defect detection model, but also improve the defect detection accuracy of the model itself. It should be noted that the defects that have been inspected are not marked during the manual re-judgment process. This is because the purpose of the first basic defect detection model is to detect all defects in the image slice as much as possible, so it is not necessary to mark the samples of the defects that have been inspected to reduce the number of over-inspections. This is a special setting that is conducive to the system to detect more defects with a greater probability.

[0073] The cloud is also used to receive a fourth sample uploaded by the client for training the second customized defect classification model. The fourth sample is a customized defect classification model of a basic defect detection model formed by performing sample annotation on the segmented images corresponding to the missed detection, over-detection or wrong detection of the second customized defect classification model.

[0074] The fourth sample here also includes the sample of defects that have passed the inspection. This is because the second customized defect classification model has high requirements for its own classification accuracy and needs to make a more accurate judgment on the first image output by the first basic defect detection model. Therefore, in the process of manual re-judgment, it is necessary to label samples of various types of misjudgments, so as to obtain models with fewer misjudgments after subsequent model training and updating.

[0075] In this embodiment, the first basic defect detection model is a large model with slow update efficiency and limited number of new samples, which results in that the update effect may not necessarily be better; the second customized defect classification model is a small model with fast update efficiency and may need to be updated at any time according to different detection needs. Figure 3 and Figure 4 As shown, in this embodiment, there are different update strategies for the first basic defect detection model and the second customized defect classification model. Figure 3 The updating method of the first basic defect detection model is shown. The first basic defect detection model needs to accumulate a large number of samples and then perform training updates when an administrator issues a first basic defect detection model training command through a client. Figure 4The updating method of the second customized defect classification model is shown. The second customized defect classification model does not need to accumulate a large number of samples, and can be automatically trained and updated as long as the number of samples is sufficient.

[0076] The cloud is used to train the first basic defect detection model when the number of new labeled samples received (i.e., the total number of the first sample and the third sample) is greater than a first preset value (which can be set to M) and a first basic defect detection model training instruction is received; the cloud is used to train the second customized defect classification model when the number of new labeled samples received (i.e., the total number of the second sample and the fourth sample) is greater than a second preset value (which can be set to N); wherein the first preset value is greater than the second preset value.

[0077] At this time, the first basic defect detection model and the second customized defect classification model can be trained and updated independently. When users may need to update at any time according to different detection needs, they only need to change the specific defect judgment criteria and independently update the second customized defect classification model. This not only meets the stability of the first basic defect detection model, but also meets the personalized needs of the second customized defect classification model, making model updates more flexible.

[0078] After the training is completed, the cloud can also perform performance testing on the trained first basic defect detection model and the second customized defect classification model. The performance test still uses labeled samples to test the first basic defect detection model and the second customized defect classification model respectively. Generally speaking, all labeled samples can be used, or some labeled samples can be used. In this embodiment, the missed detection rate and the false positive rate are used as performance indicators for evaluating the first basic defect detection model and the second customized defect classification model. Among them, the definitions of missed detection and false positive are shown in Table 2. The missed detection rate is the number of missed detection results / total number of samples, and the false positive rate is the number of false positive results / total number of samples. When the above indicators meet the preset standards, it is considered that the trained first basic defect detection model and the second customized defect classification model have passed the test and can be used for model updating.

[0079] Table 2 Correspondence between model detection and real results

[0080]

[0081] It should be noted that the cloud of the present invention can train and update the first basic defect detection model and / or the second customized defect classification model in the following manners:

[0082] The cloud obtains the first basic defect detection model and / or the second customized defect classification model from the edge, completes the training and performance test of the first basic defect detection model and / or the second customized defect classification model in the cloud, and then sends the model update parameters of the first basic defect detection model and / or the second customized defect classification model that meet the performance test requirements to the edge, and uses the model update parameters to complete the update of the first basic defect detection model and / or the second customized defect classification model. This update method greatly reduces the amount of data transmission and is more conducive to the overall defect detection performance of the system.

[0083] Of course, the cloud can also directly send the first basic defect detection model and / or the second customized defect classification model that meets the test requirements to the edge to replace the original first basic defect detection model and / or the second customized defect classification model to achieve model update.

[0084] As another embodiment of the present invention, a defect detection method based on cloud-edge collaboration is provided, the method comprising:

[0085] The edge end obtains the original image uploaded by the user from the client, and divides the original image into multiple slices and sends them to the first basic defect detection model; the edge end includes a first basic defect detection model and at least one second customized defect classification model, and the second customized defect classification model is a defect classification model designed based on a specific defect judgment standard;

[0086] The first basic defect detection model at the edge performs preliminary defect detection on the slice to obtain a first detection result, wherein the first detection result includes a plurality of first images with defects identified, and a first defect category and a first confidence level corresponding to the first image; the edge is further configured to send the first detection result to the corresponding second customized defect classification model according to the first defect category;

[0087] The second customized defect classification model at the edge acquires a first image of a corresponding category, and performs defect classification on the first image to obtain a second classification result, where the second classification result includes a second image identified as having defects and a second classification category and a second confidence level corresponding to the second image;

[0088] The edge sends the first detection result and / or the second classification result to the cloud for the cloud to train the corresponding first basic defect detection model and / or the second customized defect classification model, which is used to update the first basic defect detection model and / or the second customized defect classification model of the edge.

[0089] Furthermore, this embodiment also includes:

[0090] The cloud obtains a first confidence interval and a second confidence interval;

[0091] The cloud performs sample annotation on the first image whose first confidence does not belong to the first confidence interval to form a first sample for training the first basic defect detection model;

[0092] The cloud performs sample annotation on the second image whose second confidence does not belong to the second confidence interval to form a second sample for training the second customized defect classification model.

[0093] Furthermore, this embodiment also includes:

[0094] The cloud receives a third sample uploaded by the client for training the first basic defect detection model, wherein the third sample is formed by annotating samples of the segmented images corresponding to missed detection or false detection by the first basic defect detection model; wherein the false detection refers to a detection in which the identified defect belongs to a real defect but the defect category is incorrectly identified;

[0095] The cloud is also used to receive a fourth sample uploaded by the client for training the second customized defect classification model. The fourth sample is formed by annotating samples of the segmented images corresponding to missed detection, over-detection or wrong detection of the second customized defect classification model.

[0096] Furthermore, this embodiment also includes:

[0097] The cloud trains the first basic defect detection model when the total amount of the first sample and the third sample received is greater than the first preset value and a first basic defect detection model training instruction is received;

[0098] The cloud trains the second customized defect classification model when the total amount of the second sample and the fourth sample received is greater than a second preset value;

[0099] Wherein, the first preset value is greater than the second preset value.

[0100] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A defect detection system based on cloud-edge collaboration, characterized in that: include: Client, cloud, edge; The client is used to obtain the original image uploaded by the user; The edge end includes a first basic defect detection model and at least one second customized defect classification model, wherein the second customized defect classification model is a defect classification model designed based on a specific defect determination standard; The edge end is used to obtain an original image uploaded by a user from the client, and divide the original image into multiple slices and send them to the first basic defect detection model, the first basic defect detection model is used to perform preliminary defect detection on the slices to obtain a first detection result, the first detection result includes multiple first images with defects identified, and a first defect category and a first confidence level corresponding to the first image; The edge end is further used to send the first detection result to the corresponding second customized defect classification model according to the first defect category; The second customized defect classification model is used to obtain a first image of a corresponding category, and perform defect classification on the first image to obtain a second classification result, wherein the second classification result includes a second image identified as having defects and a second classification category and a second confidence level corresponding to the second image; The cloud is used to obtain the first detection result and / or the second classification result from the edge, train the corresponding first basic defect detection model and / or the second customized defect classification model, and update the first basic defect detection model and / or the second customized defect classification model of the edge.

2. The cloud-edge collaboration-based defect detection system according to claim 1, characterized in that: The cloud is used to obtain the first detection result and / or the second classification result from the edge, and train the first basic defect detection model and / or the second customized defect classification model, specifically: The cloud is used to obtain a first confidence interval and a second confidence interval; The cloud is used to perform sample annotation on the first image whose first confidence does not belong to the first confidence interval to form a first sample for training the first basic defect detection model; The cloud is used to perform sample annotation on the second image whose second confidence does not belong to the second confidence interval to form a second sample for training the second customized defect classification model.

3. The cloud-edge collaboration-based defect detection system according to claim 2, characterized in that: The cloud is also used to receive a third sample uploaded by the client for training the first basic defect detection model, wherein the third sample is formed by annotating samples of the segmented images corresponding to missed detection or false detection by the first basic defect detection model; wherein the false detection refers to a detection in which the identified defect belongs to a real defect but the defect category is incorrectly identified; The cloud is also used to receive a fourth sample uploaded by the client for training the second customized defect classification model. The fourth sample is a customized defect classification model of a basic defect detection model formed by performing sample annotation on the segmented images corresponding to the missed detection, over-detection or wrong detection of the second customized defect classification model.

4. The cloud-edge collaboration-based defect detection system according to claim 3, characterized in that: The cloud is used to train the first basic defect detection model when the total amount of the received first sample and the third sample is greater than the first preset value and a first basic defect detection model training instruction is received; The cloud is used to train the second customized defect classification model when the total amount of the received second sample and the fourth sample is greater than a second preset value; Wherein, the first preset value is greater than the second preset value.

5. The cloud-edge collaboration-based defect detection system according to claim 1, characterized in that: The specific defect determination standard is: judging and classifying point defects based on the brightness of the defects; The second customized defect classification model determines and classifies point defects in the first image according to their brightness levels.

6. The cloud-edge collaboration-based defect detection system according to any one of claims 4, characterized in that: The cloud is also used to perform performance testing on the trained first basic defect detection model and / or the second customized defect classification model and obtain performance indicators; The cloud is also used to update the first basic defect detection model and / or the second customized defect classification model of the edge end when the performance index meets the preset standard.

7. A defect detection method based on cloud-edge collaboration, characterized in that: include: The edge end obtains the original image uploaded by the user from the client, and divides the original image into multiple slices and sends them to the first basic defect detection model; the edge end includes a first basic defect detection model and at least one second customized defect classification model, and the second customized defect classification model is a defect classification model designed based on a specific defect judgment standard; The first basic defect detection model at the edge performs preliminary defect detection on the slice to obtain a first detection result, wherein the first detection result includes a plurality of first images with defects identified, and a first defect category and a first confidence level corresponding to the first images; The edge end is further used to send the first detection result to the corresponding second customized defect classification model according to the first defect category; The second customized defect classification model at the edge acquires a first image of a corresponding category, and performs defect classification on the first image to obtain a second classification result, where the second classification result includes a second image identified as having defects and a second classification category and a second confidence level corresponding to the second image; The edge sends the first detection result and / or the second classification result to the cloud for the cloud to train the corresponding first basic defect detection model and / or the second customized defect classification model, which is used to update the first basic defect detection model and / or the second customized defect classification model of the edge.

8. The defect detection method based on cloud-edge collaboration according to claim 7, characterized in that: Also includes: The cloud obtains a first confidence interval and a second confidence interval; The cloud performs sample annotation on the first image whose first confidence does not belong to the first confidence interval to form a first sample for training the first basic defect detection model; The cloud performs sample annotation on the second image whose second confidence does not belong to the second confidence interval to form a second sample for training the second customized defect classification model.

9. The defect detection method based on cloud-edge collaboration according to claim 8, characterized in that: Also includes: The cloud receives a third sample uploaded by the client for training the first basic defect detection model, wherein the third sample is formed by annotating samples of the segmented images corresponding to missed detection or false detection by the first basic defect detection model; wherein the false detection refers to a detection in which the identified defect belongs to a real defect but the defect category is incorrectly identified; The cloud is also used to receive a fourth sample uploaded by the client for training the second customized defect classification model. The fourth sample is formed by annotating samples of the segmented images corresponding to missed detection, over-detection or wrong detection of the second customized defect classification model.

10. The defect detection method based on cloud-edge collaboration according to claim 9, characterized in that: Also includes: The cloud trains the first basic defect detection model when the total amount of the first sample and the third sample received is greater than the first preset value and a first basic defect detection model training instruction is received; The cloud trains the second customized defect classification model when the total amount of the second sample and the fourth sample received is greater than a second preset value; Wherein, the first preset value is greater than the second preset value.

Images