Defect detection method and related equipment

Abstract

The invention provides a defect detection method and related equipment, and belongs to the technical field of computers and defect detection. The method comprises the steps of obtaining a to-be-processed image; a to-be-processed image is processed through a feature extraction module in a deep learning defect detection model, the feature extraction module comprises a middle feature layer, the middle feature layer comprises a discarding layer, and the discarding layer is used for randomly discarding features extracted after the middle feature layer processes the to-be-processed image. After the to-be-processed image is processed by the feature extraction module, a plurality of features after random discarding processing are obtained; processing the plurality of features after random discarding processing through a detection head in a deep learning flaw detection model, and obtaining a plurality of category prediction results and a plurality of position prediction results of flaws in the to-be-processed image; and obtaining a category detection result, a position detection result, category uncertainty and position uncertainty of the flaw in the to-be-processed image according to the plurality of category prediction results and the plurality of position prediction results of the flaw.

Description

Technical Field

[0001] The present disclosure relates to the field of computers and defect detection technology, and in particular, to a defect detection method, device and system, electronic equipment, computer-readable storage medium and computer program product. Background Art

[0002] In real life, products or items may have various defects. Before these products or items leave the factory, they need to be tested for defects to prevent defective products or items from being circulated in the market. Summary of the invention

[0003] The present disclosure provides a defect detection method, comprising: obtaining an image to be processed; processing the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discard layer, the discard layer being used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding after being processed by the feature extraction module; processing the plurality of features after random discarding by a detection head in the deep learning defect detection model, to obtain a plurality of category prediction results and a plurality of position prediction results of defects in the image to be processed; obtaining a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed according to the plurality of category prediction results and the plurality of position prediction results of the defect.

[0004] The present disclosure provides a defect detection device, comprising: an image acquisition module, for acquiring an image to be processed; an image processing module, for processing the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discarding layer, the discarding layer being used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; the image processing module is also used to process the plurality of features after random discarding processing by a detection head in the deep learning defect detection model, so as to obtain a plurality of category prediction results and a plurality of position prediction results of defects in the image to be processed; the image processing module is also used to obtain a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed according to the plurality of category prediction results and the plurality of position prediction results of the defect.

[0005] The present disclosure provides a defect detection system, comprising: an image acquisition component, used to obtain the image to be processed of the object to be detected in the production process; a defect detection device, connected to the image acquisition component, used to obtain the image to be processed from the image acquisition component; wherein the defect detection device comprises: an image processing module, used to process the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discarding layer, the discarding layer being used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; the plurality of features after random discarding processing are processed by the detection head in the deep learning defect detection model to obtain a plurality of category prediction results and a plurality of position prediction results of the defects in the image to be processed; the category detection result, the position detection result, the category uncertainty and the position uncertainty of the defects in the image to be processed are obtained according to the plurality of category prediction results and the plurality of position prediction results of the defects.

[0006] An embodiment of the present disclosure provides a defect detection system, comprising: an image capture device for collecting the image to be processed of the object to be detected in the production process; a processor and a memory for storing computer programs; when the processor executes the computer program, the steps in the method described in any embodiment of the present disclosure are implemented.

[0007] An embodiment of the present disclosure provides an electronic device, including: one or more processors; and a memory configured to store one or more programs, so that when the one or more programs are executed by the one or more processors, the electronic device implements the method described in any embodiment of the present disclosure.

[0008] An embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on a computer, the computer implements the method described in any embodiment of the present disclosure.

[0009] The embodiments of the present disclosure provide a computer program product, including a computer program. When the computer program is executed by a computer, the method described in the embodiments of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of a defect detection method according to an exemplary embodiment of the present disclosure.

[0011] Figure 2 is a schematic diagram of a deep learning defect detection model according to an exemplary embodiment of the present disclosure.

[0012] Figure 3 is a schematic diagram of a deep learning defect detection model according to another exemplary embodiment of the present disclosure.

[0013] Figure 4 yes Figure 3 Schematic diagram of the first cross-stage convolution block and the second cross-stage convolution block shown.

[0014] Figure 5 yes Figure 4 Schematic diagram of the first intermediate layer and the second intermediate layer shown.

[0015] Figure 6 yes Figure 5 Schematic diagram of the first depth-wise separable convolutional layer and the second depth-wise separable convolutional layer shown.

[0016] Figure 7 yes Figure 3 Schematic diagram of the third cross-stage convolutional block shown.

[0017] Figure 8 is a schematic diagram of a deep learning defect detection model according to yet another exemplary embodiment of the present disclosure.

[0018] Fig. 9 is a schematic diagram of a deep learning defect detection model according to yet another exemplary embodiment of the present disclosure.

[0019] Fig.10 is a schematic diagram of a spatial pyramid pooling block according to an exemplary embodiment of the present disclosure.

[0020] Fig.11 is a schematic diagram of an alternative embodiment of the spatial pyramid pooling block in the embodiment of the present disclosure.

[0021] Fig.12 yes Fig. 9 Schematic diagram of the attention block shown.

[0022] Fig.13 is a schematic diagram of a defect detection system according to an exemplary embodiment of the present disclosure.

[0023] Fig.14 is a schematic diagram of a defect detection system according to another exemplary embodiment of the present disclosure.

[0024] Fig.15 is a schematic diagram of a defect detection system according to yet another exemplary embodiment of the present disclosure.

[0025] Fig.16 It is a schematic structural diagram of a defect detection device according to an exemplary embodiment of the present disclosure.

[0026] Fig.17 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described here are merely illustrative and should not be interpreted as limiting the scope of the present disclosure.

[0028] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0029] In the present disclosure, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise expressly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances.

[0030] In the description of the present disclosure, it is necessary to understand that the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, should not be understood as a limitation on the present disclosure.

[0031] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0032] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0033] Figure 1 The defect detection method provided in the embodiment can be executed by any electronic device, such as a terminal and / or a server, and the present disclosure does not limit this. Figure 1 As shown, the method provided by the embodiment of the present disclosure includes the following steps.

[0034] In S110, an image to be processed is obtained.

[0035] The image to be processed in the disclosed embodiment can be any image of an object to be detected that needs to be detected to see if it contains defects or flaws. The object to be detected can be, for example, a sanitary and personal care product, such as medical and sanitary products, such as surgical gowns, protective clothing, disinfection wraps, masks, diapers, civilian rags, wipes, wet face towels, magic towels, soft towel rolls, beauty cosmetics, etc.; it can also be a home decoration product, such as wall coverings, tablecloths, bed sheets, bedspreads, etc.; it can also be a clothing product, such as lining, adhesive interlining, flakes, shaping cotton, various synthetic leather base fabrics, etc.; it can also be an industrial product, such as filter materials, insulating materials, cement packaging bags, geotextiles, covering fabrics, etc.; it can also be an agricultural product, such as crop protection fabrics, seedling cloths, irrigation fabrics, thermal insulation curtains, etc.; it can also be other products, such as space cotton, thermal insulation and sound insulation materials, oil-absorbing felt, cigarette filters, tea bags in bags, etc.

[0036] In the following embodiments, the object to be detected is a mask (including a mask that has been sewn and a mask in the cutting process), and the corresponding image to be processed is a mask image. In other embodiments, the object to be detected is a mask cloth in high-speed motion in a production line (hereinafter referred to as a production line), and the corresponding image to be processed is a mask cloth image. For example, the mask cloth can be a non-woven fabric, but the present disclosure is not limited to this, and other suitable materials can also be used as the mask cloth. At this point, the present disclosure relates to the field of non-woven material detection technology, mainly for the mask cloth defect detection subdivision field, and the method can also be referred to as a non-woven mask cloth defect detection method or a mask cloth defect detection method.

[0037] For example, the process of making masks with non-woven fabrics mainly includes material preparation, cutting, sewing and finishing. After the raw materials are prepared, cutting is carried out. The non-woven fabric is laid flat and cut with a cutting machine to cut out the main part of the mask. Next is the sewing process, in which the non-woven fabric, nose bridge strips, ear straps and other parts are sewn together to form a complete mask. Finally, finishing and quality inspection are required to ensure that each mask meets the quality standards. The relevant technology mainly involves the defect detection of finished masks, and does not involve the defect detection of mask cloth on the mask production line, and quality management and control cannot be achieved in the production stage.

[0038] Due to the global epidemics (such as covid-19) that have broken out in recent years, which have posed a huge threat to public safety, people's demand for masks has surged. The quality of masks plays an important role in the prevention and control of epidemics. Therefore, mask manufacturers need to improve the production efficiency of masks while ensuring quality. In the related art, manual detection is mainly used. The method based on manual detection to achieve mask quality control lacks efficiency, and there are problems such as inconsistent standards and low precision. The method provided in the embodiment of the present disclosure uses machine vision technology to realize automated defect detection of mask cloth on the production line, which can greatly improve production efficiency while ensuring the quality of masks, and can effectively reduce labor costs.

[0039] In S120, the image to be processed is processed by a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, the intermediate feature layer includes a discard layer, and the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module.

[0040] The deep learning defect detection model in the embodiments of the present disclosure is a model based on deep learning that can process images, detect whether there are defects in the images, and predict the defect information of the defects when defects are detected. The defect information includes, for example, any one or more of the size, location, category, etc. of the defects. For example, the deep learning defect detection model can be based on a target detection model, such as any one or more of the YOLO (You Only Look Once) series (such as YOLOv3, YOLOv4, Yolov5, etc.), Faster R-CNN, SSD, CNN (convolutional neural network), recurrent neural network (RNN) and transformer (Transformer). In the following embodiments, the YOLO model is used as an example, but the present disclosure is not limited to this.

[0041] The deep learning defect detection model in the disclosed embodiment is a real-time target detection model, which divides the input image to be processed into grids, each grid is responsible for detecting a target (i.e., defect), and predicting the bounding box (also referred to as anchor box) and category of the target. The deep learning defect detection model includes a feature extraction module, which is the backbone network (Backbone) in the deep learning defect detection model, and is mainly responsible for extracting the features of the input image to be processed. The backbone network may include convolution layers, pooling layers, and other possible network layers, etc., which can gradually extract deep features in the image to be processed, and are used to convert the image to be processed into multi-layer feature maps. These feature maps are then used for target detection tasks, such as bounding box prediction and category classification. For example, the backbone network may include a Conv module (convolution module), a C3 module (composed of three standard convolution layers, each convolution layer having a specific convolution kernel size and step size) and an SPPF module (spatial pyramid pooling layer, used to aggregate features at multiple scales), etc. These modules extract and fuse feature information at different levels through operations such as convolution and pooling.

[0042] Backbone's intermediate feature layer refers to the hidden layer (excluding feature pyramid and detection head) located after the input layer (if there is an output layer, it is located between the input layer and the output layer) in the Backbone network, which is used to generate feature maps of the image to be processed. These intermediate feature layers capture the information of the image to be processed at different levels and scales. The information in the intermediate feature layer can not only be used to generate the final detection results, but sometimes it can also be used for other tasks, such as feature fusion or further analysis. For example, the intermediate feature layer of Backbone can be fused with other network parts (such as FPN, i.e., feature pyramid network / module) to improve detection performance.

[0043] In the disclosed embodiment, a dropout layer is added to the intermediate feature layer of the feature extraction module of the deep learning defect detection model. Dropout simulates the integrated learning between different sub-networks by randomly "closing" or "erasing" a part of the neurons (or features) in the hidden layer so that the deleted neurons no longer transmit signals. This can effectively prevent the model from overfitting and improve its generalization ability on unseen data (i.e., data not included in the training data set). Through dropout, the model is forced to learn a more robust feature representation because it cannot rely on any specific combination of neurons. Dropout can be regarded as a Bayesian approximation method to a certain extent. By randomly "closing" a part of the neurons, the sparsity of the model parameters is simulated, which has certain similarities with the posterior distribution of parameters in Bayesian inference. Dropout does not directly simulate the posterior distribution of Bayes, but provides a regularization effect through its randomness, which helps to reduce overfitting. In the disclosed embodiment, dropout is added to the intermediate feature layer of Backbone, and its ability to prevent overfitting can be used to improve the generalization performance of the model. At the same time, the complexity and number of parameters of the neural network can also be reduced. If dropout is only enabled during the training phase and disabled during the inference phase, it is used to prevent the model from overfitting. In the disclosed embodiment, dropout is enabled during both the training phase and the inference phase to simulate Bayesian uncertainty.

[0044] In S130, the detection head in the deep learning defect detection model processes the multiple features after random discarding to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed.

[0045] Under the effect of dropout, the disclosed embodiment can randomly characterize the generation process of the network structure of the hidden layer by randomly closing the neurons in the hidden layer (making the node values ​​of a part of the hidden layer 0), such as the number of layers of the neural network, the connection mode of the neurons in each layer, etc., and can obtain multiple different network structures of the intermediate feature layer, so as to make multiple predictions for the same image to be processed (the same target sample image in the training process) through multiple different network structures of the intermediate feature layer, thereby obtaining multiple category prediction results and multiple position prediction results for each defect in the image to be processed. Uncertainty estimation is to calculate the distribution index of the results of multiple predictions.

[0046] In the disclosed embodiments, the category prediction result refers to the category result to which the defect belongs, as predicted and output by the detection head of the deep learning defect detection model (e.g., the depth-separable convolution block therein). The position prediction result refers to the position of the defect in the image (including the image to be processed and / or the target sample image), as predicted and output by the detection head of the deep learning defect detection model (e.g., the depth-separable convolution block therein), which may be an absolute position or a relative position. The category prediction result may include the predicted probability that the defect belongs to each category, and the position prediction result may include the predicted information such as the center coordinates, width, and height of the bounding box of the defect in the image.

[0047] In S140, a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the defect in the image to be processed are obtained according to the multiple category prediction results and the multiple position prediction results of the defect.

[0048] In the target detection task, the detection head of the deep learning defect detection model in the disclosed embodiment is mainly responsible for predicting the category of defects in the image to be processed, the location of the bounding box, and the confidence, and can also predict the category uncertainty of the predicted category of the defect and the location uncertainty of the location. In some embodiments, the detection head can use multiple convolutional layers to further extract and integrate features, and then output multiple category prediction results and multiple location prediction results of the defect, and based on the multiple category prediction results and multiple location prediction results of the defect, obtain the category detection result, location detection result, category uncertainty, and location uncertainty of the defect in the image to be processed.

[0049] In the disclosed embodiment, the category detection result and the position detection result refer to the category result of the defect output by the detection head of the deep learning defect detection model and the position of the defect in the image, respectively.

[0050] In the disclosed embodiments, category uncertainty refers to an indicator used to evaluate the degree of certainty of whether the predicted category result matches the true category of the defect when the deep learning defect detection model is used to predict the category prediction result of the defect in the image (including the image to be processed and / or the target sample image). The higher the category uncertainty, the more uncertain the category prediction result of the deep learning defect detection model for the defect is, and the lower the category uncertainty, the more certain the category prediction result of the deep learning defect detection model for the defect is.

[0051] In the embodiments of the present disclosure, position uncertainty refers to an indicator of the degree of certainty of whether the predicted position result matches the actual position of the defect when the deep learning defect detection model is used to predict the position prediction result of the defect in the image (including the image to be processed and / or the target sample image). The higher the position uncertainty, the more uncertain the position prediction result of the defect by the deep learning defect detection model is, and the lower the position uncertainty, the more certain the position prediction result of the defect by the deep learning defect detection model is.

[0052] In the disclosed embodiment, in the deep learning defect detection model, in addition to predicting the position and category of the bounding box, the confidence of the output bounding box can also be predicted. The confidence of the predicted output is a value between 0 and 1, which is used to indicate the degree of certainty of the model about the detected defect. The confidence can indicate the probability (also called target confidence) of whether there is a defect (also called a defect) in the bounding box, that is, whether the bounding box is just the background or contains defects. Optionally, it can also indicate the degree of match between the bounding box predicted by the model and the real box of the defect when the bounding box has defects; it can also indicate the degree of match between the category result predicted by the model and the real category of the defect when the bounding box has defects. For example, the confidence can be obtained by calculating the IoU (Intersection over Union) between the predicted bounding box and the actual defect. If the IoU is greater than a certain set threshold, it is considered that the defect is detected, and the confidence is the IoU score of the bounding box; if the IoU is less than the set threshold, it is considered that the defect is not detected, and the confidence is 0.

[0053] Uncertainty estimation is an effective method to measure the credibility of detection results. The uncertainty in target detection tasks can be divided into data uncertainty and model uncertainty. Data uncertainty mainly comes from the inherent noise in real data (such as the input image to be processed or the target sample image), and model uncertainty comes from the uncertainty of model parameters. The category uncertainty in the disclosed embodiment may include both data uncertainty and model uncertainty, and the position uncertainty may also include both data uncertainty and model uncertainty. Among them, the proportion of data uncertainty is relatively small, which is related to the noise and the annotation quality of the sample.

[0054] When prediction uncertainty is insufficient, the model may be overconfident and give wrong category detection results and position detection results. For example, in quality inspection on a production line, the model may mistakenly classify a defective / defective product as non-defective, or vice versa. This misclassification may lead to inefficient production lines and even affect the overall quality of the product. For production processes that rely on model prediction results for quality control, if the model cannot provide reliable and accurate predictions, then these predictions may not be trusted by end users.

[0055] In an exemplary embodiment, the mean of multiple category prediction results of a defect may be calculated as the category detection result of the defect. The category prediction result indicates the probability of the detected defect being a certain type of defect. The category prediction result may also be referred to as the category confidence, and thus the category detection result may also be referred to as the category confidence mean. The mean of multiple position prediction results of the defect may be used as the position detection result of the defect. When coordinates are used to represent the position, the position detection result may also be referred to as the coordinate mean.

[0056] In the disclosed embodiments, category uncertainty is the uncertainty of category confidence, which is a probability distribution index of category confidence, that is, the probability distribution of category confidence represents category uncertainty, so it can also be called category confidence uncertainty. For example, it can be calculated using the variance of category confidence, which can be called category confidence variance. Position uncertainty is a probability distribution index of a bounding box, and the probability distribution of a bounding box represents position uncertainty. For example, the variance of the position prediction result can be calculated to obtain position uncertainty. When coordinates are used to represent position, and the variance of the coordinates is calculated as position uncertainty, position uncertainty can also be called coordinate variance.

[0057] For example, the model output includes [coordinate mean, coordinate variance (position uncertainty), category confidence mean, category confidence variance (category uncertainty), target confidence]. Among them, the target confidence indicates the probability of whether the detection is a defect. The category confidence mean indicates the probability of which type of defect the detection is.

[0058] In the embodiment of the present disclosure, a confidence threshold can be set. If the target confidence of the model output is less than the confidence threshold, the defect with lower target confidence is detected, for example, prompting manual judgment. The confidence threshold is a real number greater than 0 and less than 1, for example, it can be set to 0.3, but the present disclosure is not limited to this.

[0059] Assuming that the position or coordinates of each bounding box are represented by (x, y, w, h), where x, y, w, h represent the central abscissa and ordinate, width, and height of the bounding box, respectively, the coordinate mean is expressed as: The coordinate variance is expressed as: For example, the mean and variance of category confidence are: Then the category confidence p(c|Object) can be expressed as (c represents the current category, k represents traversing all categories):

[0060]

[0061] However, the present disclosure is not limited to this. For example, the category prediction result with the highest frequency among the multiple category prediction results of the defect may be selected as the category detection result of the defect, and the position prediction result with the highest frequency among the multiple position prediction results of the defect may be selected as the position detection result of the defect. Since the category detection result and position detection result of the defect in the embodiment of the present disclosure comprehensively consider the multiple category prediction results and multiple position prediction results of the defect, the detection result (including the category detection result and the position detection result) outputted by the defect has higher accuracy.

[0062] It should be noted that in the disclosed embodiment, the category prediction result output by the deep learning defect model is the category confidence mean. The category confidence mean output by the model is subjected to the system post-processing process using the above formula (1). The post-processing process actually adds a softmax process after the model output to map the category confidence to a real number between [0,1]. Optionally, the post-processing process can also protect the bounding box NMS (non-maximum suppression) and the like.

[0063] In an exemplary embodiment, the variance of multiple category prediction results of the defect may be calculated as the category uncertainty of the defect, and the variance of multiple position prediction results of the defect may be calculated as the position uncertainty of the defect.

[0064] The present disclosure is not limited to directly using variance as uncertainty. For example, uncertainty can also be calculated by the following formula:

[0065] Uncertainty=exp(-λ·variance) (2)

[0066] Where λ represents the adjustment term, which is a real number greater than 0 and less than 1. Variance represents the variance of multiple category prediction results or the variance of multiple location prediction results. Uncertainty represents category uncertainty or location uncertainty.

[0067] In some embodiments, an uncertainty threshold may be set to filter prediction results with higher uncertainty.

[0068] Sensitivity to environmental factors may affect the detection performance of the model, such as environmental factors such as light, shadows, and cluttered backgrounds. The disclosed embodiment optimizes the model to improve its stability and accuracy in the actual environment. For example, Bayesian estimation / Bayesian prediction is introduced to improve the generalization ability and robustness of the model to the natural environment. Bayesian estimation simulates randomness by adding a dropout layer to randomly erase the output of a part of the features. At the same time, the output of the detection head of the model is changed, and the output is interpreted as a probability distribution, not just a regression problem. Interpreting the output as a probability distribution can better capture the uncertainty of the location and category of the defect. In this way, the model can not only give a deterministic estimate of the prediction, but also provide uncertainty measures about these estimates. This is particularly important for dealing with complex natural environments, because in such environments, the location, size, and appearance of the target may change due to a variety of factors. By combining Bayesian estimation and dropout layers and interpreting the output as a probability distribution, a more robust and more generalized deep learning defect detection model can be constructed. Such a model can better adapt to changes in the natural environment and improve the accuracy and reliability of detection.

[0069] The defect detection method provided by the disclosed embodiment simplifies the approximation of the posterior probability distribution by adding a discard layer to the intermediate feature layer in the feature extraction module of the deep learning defect detection model, so that it can not only quickly and effectively predict the category and position of the defects in the processed image, but also evaluate the category uncertainty and position uncertainty of the prediction results. The disclosed embodiment has developed a new deep learning defect detection model that is more accurate, efficient and has good generalization ability. It can improve the accuracy of defect recognition without sacrificing the detection speed, and can adapt to the needs of various production environments and types of objects to be detected. It has broad application prospects. For example, it can be applied to the mask fabric defect detection system to identify defects in different mask types.

[0070] like Figure 2 As shown, the deep learning defect detection module 1 provided in the embodiment of the present disclosure includes a feature extraction module 11 and a detection head 12. The feature extraction module 11 includes an input block 111 and an intermediate feature layer 112. The input block 111 is used to receive an image to be processed and send the image to be processed to the intermediate feature layer 112. The intermediate feature layer 112 includes a discard layer 1121. The input block 111 can also be used to pre-process the received input image to be processed, such as resizing and normalizing, so as to adapt to the input requirements of the model.

[0071] The image to be processed is processed by the feature extraction module 11 in the deep learning defect detection model 1, and the discarding layer 1121 in the intermediate feature layer 112 of the feature extraction module 11 is used to randomly discard the features extracted after the intermediate feature layer 112 processes the image to be processed, so that the image to be processed obtains multiple features (the specific number of samples can be set according to actual needs) after being processed by the feature extraction module 11. Then, the multiple features after the random discarding process are processed separately by the detection head 12 in the deep learning defect detection model 1 to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed. The detection head 12 also obtains the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed based on the multiple category prediction results and multiple position prediction results of the defects.

[0072] In an exemplary embodiment, the intermediate feature layer includes a first cross-stage convolution block, and the discard layer includes a first discard layer. The first cross-stage convolution block includes a first cross-stage input layer, a first convolution layer, a first intermediate layer, a second convolution layer, a first fusion layer, a third convolution layer, and the first discard layer. The first convolution layer is used to perform convolution processing on the first cross-stage input feature map input by the first cross-stage input layer, and input the first convolution feature map to the first intermediate layer. The first intermediate layer is used to process the first convolution feature map, and input the first feature map to be fused to the first fusion layer. The second convolution layer is used to perform convolution processing on the first cross-stage input feature map, and input the second feature map to be fused to the first fusion layer. The first fusion layer is used to perform fusion processing on the first feature map to be fused and the second feature map to be fused, and input the first fused feature map to the third convolution layer. The third convolution layer is used to perform convolution processing on the first fused feature map, and input the second convolution feature map to the first discard layer. The first discard layer is used to randomly discard features in the second convolutional feature map to obtain multiple first discarded feature maps. The multiple features obtained after the image to be processed is processed by the feature extraction module and subjected to random discarding processing include multiple first discarded feature maps.

[0073] The cross-stage input layer (including the first to third cross-stage input layers), the intermediate input layer (including the first to third intermediate input layers), and the depth separation input layer (including the first to third depth separation input layers) in the embodiments of the present disclosure are all input layers of the corresponding modules in the model. Their names are only used to distinguish which module the input layer belongs to. For example, the cross-stage input layer indicates that it is the input layer of the cross-stage convolution block. The input layer is used to preprocess the input feature map or image data, such as adjusting the size, so as to adapt to the input requirements of the corresponding module in the model.

[0074] The convolution feature map in the embodiment of the present disclosure (for example, including the first to sixth convolution feature maps) refers to the feature map obtained after the convolution process; the feature map to be fused (for example, including the first to sixth feature maps to be fused) refers to the feature map to be fused, and the fusion process here can be, for example, splicing or cascading, for example, the first feature map to be fused and the second feature map to be fused are cascaded to obtain the first fused feature map. The discarded feature map (for example, including the first to third discarded feature maps) refers to the feature map generated by randomly discarding features through the corresponding dropout layer. The spatial pooling feature map (for example, including the first to third spatial pooling feature maps) refers to the feature map generated after being processed by the spatial pyramid pooling block (for example, including the first to third spatial pyramid pooling blocks).

[0075] In an exemplary embodiment, the intermediate feature layer further includes a second cross-stage convolution block and a third cross-stage convolution block, and the dropout layer further includes a second dropout layer and a third dropout layer.

[0076] The input end of the second cross-stage convolution block is connected to the output end of the first cross-stage convolution block to receive a plurality of the first discarded feature maps respectively, process the plurality of the first discarded feature maps, and input a plurality of second discarded feature maps to the third cross-stage convolution block through the second discard layer. The input end of the third cross-stage convolution block is connected to the output end of the second cross-stage convolution block to receive a plurality of the second discarded feature maps respectively, process the plurality of the second discarded feature maps, and output a plurality of third discarded feature maps through the third discard layer.

[0077] The second cross-stage convolution block includes a second cross-stage input layer, a fourth convolution layer, a second intermediate layer, a fifth convolution layer, a second fusion layer, a sixth convolution layer and the second discard layer. The second cross-stage input layer is connected to the first discard layer and is used to respectively use the plurality of the first discard feature maps as the second cross-stage input feature maps. The fourth convolution layer is used to perform convolution processing on the second cross-stage input feature map input by the second cross-stage input layer and input the third convolution feature map to the second intermediate layer. The second intermediate layer is used to process the third convolution feature map and input the third feature map to be fused to the second fusion layer. The fifth convolution layer is used to perform convolution processing on the second cross-stage input feature map and input the fourth feature map to be fused to the second fusion layer. The second fusion layer is used to perform fusion processing on the third feature map to be fused and the fourth feature map to be fused and input the second fused feature map to the sixth convolution layer. The sixth convolution layer is used to perform convolution processing on the second fused feature map and input the fourth convolution feature map to the second discard layer. The second discard layer is used to randomly discard the features in the fourth convolutional feature map to obtain multiple second discarded feature maps. The multiple features obtained after the image to be processed is processed by the feature extraction module and processed by random discard also include multiple second discarded feature maps.

[0078] The cross-stage convolution blocks in the embodiments of the present disclosure (for example, including the first to third cross-stage convolution blocks) adopt the idea of ​​cross-stage partial connection, and enhance the diversity and robustness of the features by fusing the feature maps of different stages. This design helps to improve the performance of the model, especially when dealing with complex target detection tasks. Specifically, the cross-stage convolution block adopts a staged processing approach, dividing the input feature map into multiple stages, each of which contains a series of convolution operations. At the end of each stage, the cross-stage convolution block fuses the feature map of the current stage with the feature map of the previous stage to obtain a richer feature representation. This cross-stage connection method helps the model capture feature information of different scales and levels, and improves the model's ability to perceive the target.

[0079] In some other embodiments, the cross-stage convolutional block can also adopt a partial connection strategy, that is, instead of fusing all the feature maps of all stages, it selectively fuses some feature maps. This design can reduce the amount of calculation of the model while maintaining high performance.

[0080] In an exemplary embodiment, the deep learning defect detection model further includes a feature pyramid module, and the feature pyramid module includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block. The first spatial pyramid pooling block is connected to the first cross-stage convolution block, and is used to receive a plurality of the first discarded feature maps. The second spatial pyramid pooling block is connected to the second cross-stage convolution block, and is used to receive a plurality of the second discarded feature maps, and processes the plurality of the second discarded feature maps respectively to obtain a plurality of second spatial pooling feature maps. The third spatial pyramid pooling block is connected to the third cross-stage convolution block, and is used to receive a plurality of the third discarded feature maps, and processes the plurality of the third discarded feature maps respectively to obtain a plurality of third spatial pooling feature maps. The first spatial pyramid pooling block is also connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and obtain a plurality of first spatial pooling feature maps according to the plurality of the first discarded feature maps and the plurality of the second spatial pooling feature maps. The second spatial pyramid pooling block is also connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and obtain a plurality of second spatial pooling feature maps according to a plurality of the second discarded feature maps and a plurality of the third spatial pooling feature maps.

[0081] In the disclosed embodiment, a feature pyramid module can be added between the backbone network (e.g., feature extraction module 11) and the detection head 12 to further process and enhance the features extracted by the backbone network. In an exemplary embodiment, the feature pyramid module can include one or more of FPN (Feature Pyramid Networks) and PANet (Path Aggregation Network), which combine feature information at different levels through multi-scale feature fusion to improve the detection performance of the model. In the following embodiments, the feature pyramid module includes a spatial pyramid pooling block for illustration, but the present disclosure is not limited to this.

[0082] The Spatial Pyramid Pooling block (SSP block for short) uses the technique of spatial pyramid pooling. The idea of ​​spatial pyramid pooling comes from the application of pyramid-shaped grid cells that cover receptive fields of different sizes. Each grid cell will perform the same pooling operation, and eventually all feature maps of different sizes will be connected together to form a feature vector of fixed length. Specifically, the spatial pyramid pooling block scales the input feature map and performs pooling operations at each scale. This multi-scale pooling method enables the network to process inputs of any size and generate outputs of a fixed size. It improves the flexibility and adaptability of the model by splicing feature maps of different scales together to obtain a multi-scale feature representation. Since the spatial pyramid pooling block uses this multi-scale pooling technique, it can flexibly process inputs of different sizes and generate feature representations of fixed length, so it is called the spatial pyramid pooling block.

[0083] In an exemplary embodiment, the detection head includes a first depthwise separable convolution block, a second depthwise separable convolution block and a third depthwise separable convolution block. The defect includes a first defect, a second defect and a third defect; the first depthwise separable convolution block is connected to the first spatial pyramid pooling block, and is used to receive a plurality of the first spatial pooling feature maps, and respectively process the plurality of the first spatial pooling feature maps to obtain a plurality of first category prediction results and a plurality of first position prediction results of the first defect. The second depthwise separable convolution block is connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and respectively process the plurality of the second spatial pooling feature maps to obtain a plurality of second category prediction results and a plurality of second position prediction results of the second defect. The third depthwise separable convolution block is connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and respectively process the plurality of the third spatial pooling feature maps to obtain a plurality of third category prediction results and a plurality of third position prediction results of the third defect.

[0084] The detection head in the embodiment of the present disclosure may include two or more depth-separable convolution blocks (for example, including first to third depth-separable convolution blocks), and the number of depth-separable convolution blocks in the detection head corresponds to the number of cross-stage convolution blocks in the intermediate feature layer. When the model also includes a feature pyramid module, the number of depth-separable convolution blocks also corresponds to the number of spatial pyramid pooling blocks in the feature pyramid module. In the following embodiments, the detection head is respectively illustrated as including 2 or 3 depth-separable convolution blocks, but the present disclosure is not limited to this. For example, more depth-separable convolution blocks may be included to extract defects of more scales.

[0085] In the disclosed embodiment, each depth-separable convolution block decomposes a complete convolution operation into two steps, namely Depthwise Convolution and Pointwise Convolution. In the depth convolution step, for example, a feature map with a size of 64×64 pixels and three channels of color (for example, it can come from the output of the corresponding cross-stage convolution block or the output of the spatial pyramid pooling block) first undergoes the first convolution operation. This convolution is completely performed in a two-dimensional plane, and the number of filters is the same as the Depth (depth, here refers to the number of channels rather than the network level) of the previous layer. So a three-channel feature map generates 3 feature maps after the operation. One of the filters contains a kernel of size 3×3. The number of feature maps after Depthwise Convolution is the same as the Depth of the input layer. Then, through Pointwise Convolution, the information of different feature maps at the same spatial position is used to combine these feature maps to generate a new feature map. The size of the convolution kernel of PointwiseConvolution is 1×1×M, where M is the Depth of the previous layer. The convolution operation here will perform a weighted combination of the feature mapp in the depth direction of the previous step to generate a new feature map. After PointwiseConvolution, 4 feature maps are also output, which is the same as the output dimension of the regular convolution. The same input also gets 4 feature maps. The number of parameters of the depthwise separable convolution is about 1 / 3 of that of the regular convolution, and the amount of calculation can be reduced by 8-9 times. Therefore, under the premise of the same number of parameters, the number of neural network layers using depthwise separable convolution can be made deeper.

[0086] In the disclosed embodiment, the structure of the depth-separable convolution block in the detection head can refer to the depth-separable convolution layer in the intermediate feature layer (for example, the first to third depth-separable convolution layers), and the convolution kernel size of the depth-separable convolution block can be different from that of the depth-separable convolution layer, and the convolution method can be the same. For example, the convolution kernel sizes in the detection head can be 1×1 and 3×3, respectively. The convolution kernel sizes of the depth-separable convolution layer in the intermediate feature layer can be 1×1, 3×3, 5×5, and 7×7, respectively. The role of different convolution kernels is to have different feature receptive fields. Depth-separable convolution can effectively reduce the amount of convolution parameters and thus enhance the processing speed. For example, for normal convolution, such as 128x3x3x256 (c1 wh c2), 128 represents the number of input channels, 3x3 represents the convolution kernel size, 256 represents the number of output channels, and the parameter amount is 294912. Depthwise separable convolution first convolves each input channel with a 3x3 depthwise convolution, with a parameter of 128x3x3=1152, and then convolves each sub-region with a 1x1 point convolution, with a parameter of 128x1x1x256=32768. The total number of parameters is 1152+32768=33930. Compared with ordinary convolution, the number of parameters is reduced by 88%. However, there is only a slight loss in accuracy. Depthwise separable convolution can completely replace ordinary convolution, with the same activation method and connection method.

[0087] In some other embodiments, the depthwise separable convolution block may not be used in the detection head, that is, the detection head may include using normal convolution operations to achieve prediction.

[0088] Wherein, based on multiple category prediction results and multiple position prediction results of the defect, the category detection result, position detection result, category uncertainty and position uncertainty of the defect in the image to be processed are obtained, including: based on multiple first category prediction results and multiple first position prediction results of the first defect, the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect are obtained; based on multiple second category prediction results and multiple second position prediction results of the second defect, the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect are obtained; based on multiple third category prediction results and multiple third position prediction results of the third defect, the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect are obtained.

[0089] In the embodiment of the present disclosure, the first defect, the second defect, the third defect, etc. refer to defects of different scales in the image to be processed, rather than being used to limit the number of defects.

[0090] In an exemplary embodiment, the first intermediate layer includes a first intermediate input layer, a first depth-separable convolution layer and a first accumulation layer. The first intermediate input layer is used to receive the first convolution feature map output by the first convolution layer, and input the first convolution feature map to the first depth-separable convolution layer and the first accumulation layer respectively. The first depth-separable convolution layer is used to perform depth-separable convolution processing on the first convolution feature map, and input the first depth-separated convolution feature map to the first accumulation layer. The first accumulation layer accumulates the first convolution feature map and the first depth-separated convolution feature map to obtain the first feature map to be fused.

[0091] In an exemplary embodiment, the first depth-separable convolution layer includes a first depth-separation input layer, a first depth-convolution layer, a first point-convolution layer, and a first activation function layer. The first depth-separation input layer is used to receive the first convolution feature map output by the first intermediate input layer, and input the first convolution feature map to the first depth-convolution layer and the first activation function layer, respectively. The first depth-convolution layer is used to perform depth-convolution processing on the first convolution feature map, and input the first depth-convolution feature map to the first point-convolution layer. The first point-convolution layer is used to perform point-convolution processing on the first depth-convolution feature map, and input the first point-convolution feature map to the first activation function layer. The first activation function layer is used to process the first point-convolution feature map and the first convolution feature map to obtain the first depth-separated convolution feature map.

[0092] In an exemplary embodiment, the feature extraction module further includes an attention block connected to the intermediate feature layer. The attention block is used to process the image to be processed, obtain a downsampled feature map of the image to be processed, and input the downsampled feature map to the intermediate feature layer.

[0093] In an exemplary embodiment, the deep learning defect detection model further includes a background removal module. The background removal module is used to remove background information in the image to be processed, and input the image to be processed after the background information is removed into the feature extraction module.

[0094] In the disclosed embodiment, background removal can be used to remove or separate parts that are not related to the object to be detected from the image to be processed, so as to more accurately focus on the object to be detected itself and its possible defects. These backgrounds may include factors such as factory production lines, conveyor belts, support structures, ambient lighting, etc., which may introduce noise, interfere with edges, or affect the extraction of features such as color and texture during image processing and analysis. By removing the background, you can focus more on the object to be detected itself, so as to more accurately identify and locate defects. Background removal may involve background modeling, threshold segmentation, morphological operations, etc., to separate the foreground (i.e., the object to be detected) and the background from the original image to be processed. It should be noted that background removal does not mean completely eliminating all traces of the background in the image. On the contrary, its purpose is to reduce the interference of the background on the defect detection process while retaining sufficient contextual information so that the state and quality of the object to be detected can be accurately judged.

[0095] In the disclosed embodiment, the intermediate feature layer may include 3 or more cross-stage convolution blocks, and the number of depth-separable convolution blocks in the detection head is the same as the number of cross-stage convolution blocks in the intermediate feature layer. When the model includes a feature pyramid module, the number of spatial pyramid pooling blocks included in the feature pyramid module is the same as the number of depth-separable convolution blocks in the detection head.

[0096] Combine the following Figures 3 to 11 The deep learning defect detection model provided in the embodiment of the present disclosure is illustrated by way of example, but the present disclosure is not limited thereto.

[0097] Figure 3 is a schematic diagram of a deep learning defect detection model according to another exemplary embodiment of the present disclosure. Figure 3 As shown, Figure 2 The intermediate feature layer 112 in the image processing unit includes a first cross-stage convolution block, a second cross-stage convolution block and a third cross-stage convolution block. Optionally, the detection head 12 includes a first depth-separable convolution block, a second depth-separable convolution block and a third depth-separable convolution block. The input block 111 inputs the received image to be processed into the first cross-stage convolution block. After the first cross-stage convolution block processes the first image to be processed, a plurality of first discarded feature maps of the image to be processed are obtained, and the plurality of first discarded feature maps are respectively input into the first depth-separable convolution block for processing. The first depth-separable convolution block processes the plurality of first discarded feature maps respectively, and predicts a plurality of first category prediction results and a plurality of first position prediction results of the first defect in the image to be processed based on each first discarded feature map.

[0098] It should be noted that the backbone network structure in the embodiments of the present disclosure is not limited to the cross-stage convolutional blocks listed above, and can also be replaced by at least one of RepVGG, EfficientNet, ConvNeXt, etc.

[0099] Figure 3 In an embodiment, the first cross-stage convolution block further inputs a plurality of first discarded feature maps to a second cross-stage convolution block, respectively. The second cross-stage convolution block processes each first discarded feature map respectively and outputs a plurality of second discarded feature maps.

[0100] The second cross-stage convolution block inputs each second discarded feature map to the second depth-separable convolution block. The second depth-separable convolution block receives multiple second discarded feature maps, processes each second discarded feature map, and predicts multiple second category prediction results and multiple second position prediction results of the second defect in the image to be processed.

[0101] The second cross-stage convolution block also inputs the plurality of second discarded feature maps to the third cross-stage convolution block, respectively. The third cross-stage convolution block processes each second discarded feature map and outputs a plurality of third discarded feature maps.

[0102] The third cross-stage convolution block inputs each third discarded feature map to the third depth-separable convolution block. The third depth-separable convolution block receives multiple third discarded feature maps, processes each third discarded feature map, and predicts multiple third category prediction results and multiple third position prediction results of the third defect in the image to be processed.

[0103] Continue to refer Figure 3 The detection head 12 is also used to perform Bayesian prediction / analysis based on multiple category prediction results and multiple position prediction results predicted by each depth-separable convolution block to obtain the category detection result, position detection result, category uncertainty and position uncertainty of each defect.

[0104] For example, Bayesian prediction processing (such as averaging and variance calculation) is performed on multiple first category prediction results and multiple first position prediction results of the first defect predicted output by the first depthwise separable convolution block to obtain the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect.

[0105] For another example, Bayesian prediction processing is performed on multiple second category prediction results and multiple second position prediction results of the second defect predicted output by the second depth-separable convolution block to obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect.

[0106] For another example, Bayesian prediction processing is performed on multiple third category prediction results and multiple third position prediction results of the third defect predicted output by the third depth-separable convolution block to obtain the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect.

[0107] In the disclosed embodiment, the obtained multiple category prediction results of the defects in the image to be processed include multiple first category prediction results of the first defect, multiple second category prediction results of the second defect, and multiple third category prediction results of the third defect. The obtained multiple position prediction results of the defects in the image to be processed include multiple first position prediction results of the first defect, multiple second position prediction results of the second defect, and multiple third position prediction results of the third defect. The obtained category detection result of the defects in the image to be processed includes the first category detection result of the first defect, the second category detection result of the second defect, and the third category detection result of the third defect. The obtained position detection result of the defects in the image to be processed includes the first position detection result of the first defect, the second position detection result of the second defect, and the third position detection result of the third defect. The obtained category uncertainty of the defects in the image to be processed includes the first category uncertainty of the first defect, the second category uncertainty of the second defect, and the third category uncertainty of the third defect. The obtained position uncertainty of the defects in the image to be processed includes the first position uncertainty of the first defect, the second position uncertainty of the second defect, and the third position uncertainty of the third defect.

[0108] Figure 3 In the example, the image to be processed is an image of a mask cloth on a production line. The category detection result, i.e., the classification result, can be any one of adhesive tape / sticker, stains, printing errors, creases (mask fabric wrinkles), etc. It is understandable that these four classification results (e.g., C=4) are only used for illustration and are not limited to one time, and can be set according to the actual scene. The position detection result is the location / position information of the defect in the image to be processed.

[0109] like Figure 4 As shown, the dropout layer 1121 includes a first dropout layer and a second dropout layer. The first dropout layer is located in the first cross-stage convolution block, and the second dropout layer is located in the second cross-stage convolution block.

[0110] The input connection of the first cross-stage input layer in the first cross-stage convolution block Figure 3The output end of the input block 111 in the embodiment. The output end of the first cross-stage input layer is connected to the input end of the first convolutional layer and the second convolutional layer respectively. The output end of the first convolutional layer is connected to the input end of the first intermediate layer. The output end of the first intermediate layer is connected to the first input end of the first fusion layer. The output end of the second convolutional layer is connected to the second input end of the first fusion layer. The output end of the first fusion layer is connected to the input end of the third convolutional layer. The input end of the third convolutional layer is connected to the input end of the first drop layer. The output end of the first drop layer is respectively connected to the input end of the second cross-stage input layer in the second cross-stage convolutional block and the input end of the first depth-separable convolutional block.

[0111] The output end of the second cross-stage input layer in the second cross-stage convolution block is connected to the input ends of the fourth convolution layer and the fifth convolution layer respectively. The output end of the fourth convolution layer is connected to the input end of the second intermediate layer. The output end of the second intermediate layer is connected to the first input end of the second fusion layer. The output end of the fifth convolution layer is connected to the second input end of the second fusion layer. The output end of the second fusion layer is connected to the input end of the sixth convolution layer. The input end of the sixth convolution layer is connected to the input end of the second dropout layer. The output end of the second dropout layer is connected to the input end of the second depthwise separable convolution block.

[0112] Figure 4 In an embodiment, the convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer may all be 1×1.

[0113] like Figure 5 As shown, the first intermediate layer includes a first depth-wise separable convolution layer, and the second intermediate layer includes a second depth-wise separable convolution layer.

[0114] The input terminal of the first intermediate input layer of the first intermediate layer is connected Figure 4 The output end of the first convolutional layer in the embodiment. The output end of the first intermediate input layer is connected to the input end of the first depth-separable convolutional layer and the input end of the first accumulation layer respectively. The output end of the first accumulation layer is connected to Figure 4 The input of the first fusion layer in an embodiment.

[0115] The input terminal of the second intermediate input layer of the second intermediate layer is connected Figure 4 The output end of the fourth convolutional layer in the embodiment. The output end of the second intermediate input layer is connected to the input end of the second depth-separable convolutional layer and the input end of the second accumulation layer respectively. The output end of the second accumulation layer is connected to Figure 4 The input of the second fusion layer in an embodiment.

[0116] like Figure 6As shown, the first depth-separable convolution layer includes a first depth-separated input layer, a first depth-convolution layer, a first point-convolution layer, and a first activation function layer. The second depth-separable convolution layer includes a second depth-separated input layer, a second depth-convolution layer, a second point-convolution layer, and a second activation function layer.

[0117] The input connection of the first depth separation input layer Figure 5 The output end of the first intermediate input layer in the embodiment. The output end of the first depth separation input layer is connected to the input end of the first depth convolution layer and the input end of the first activation function layer respectively. The output end of the first depth convolution layer is connected to the input end of the first point layer. The output end of the first point convolution layer is connected to the first activation function layer.

[0118] The input connection of the second depth separation input layer Figure 5 The output end of the second intermediate input layer in the embodiment. The output end of the second depth separation input layer is connected to the input end of the second depth convolution layer and the input end of the second activation function layer respectively. The output end of the second depth convolution layer is connected to the input end of the second point layer. The output end of the second point convolution layer is connected to the second activation function layer.

[0119] Figure 7 In an embodiment, the discard layer further includes a third discard layer. Figure 7 As shown, the third cross-stage convolution block includes a third cross-stage input layer, a seventh convolution layer, a third intermediate layer, an eighth convolution layer, a third fusion layer, a ninth convolution layer and the third drop layer. The input end of the third cross-stage input layer is connected to the output end of the second cross-stage convolution block, for example, connected to the output end of the second drop layer, and the plurality of second drop feature maps output by the second drop layer are respectively used as the third cross-stage input feature maps.

[0120] The seventh convolutional layer is used to perform convolution processing on the third cross-stage input feature map input by the third cross-stage input layer, and input the fifth convolutional feature map to the third intermediate layer. The third intermediate layer is used to process the fifth convolutional feature map, and input the fifth feature map to be fused to the third fusion layer. The eighth convolutional layer is used to perform convolution processing on the third cross-stage input feature map, and input the sixth feature map to be fused to the third fusion layer. The third fusion layer is used to perform fusion processing on the fifth feature map to be fused and the sixth feature map to be fused, and input the third fused feature map to the ninth convolutional layer. The ninth convolutional layer is used to perform convolution processing on the third fused feature map, and input the sixth convolutional feature map to the third discard layer. The third discard layer is used to randomly discard the features in the sixth convolutional feature map to obtain multiple third discarded feature maps. After the image to be processed is processed by the feature extraction module, the multiple features obtained after random discarding include multiple third discarded feature maps.

[0121] Continue to refer Figure 7 , the third intermediate layer includes a third intermediate input layer, a third depth-separable convolution layer and a third accumulation layer. The third intermediate input layer is used to receive the fifth convolution feature map output by the seventh convolution layer, and input the fifth convolution feature map to the third depth-separable convolution layer and the third accumulation layer respectively. The third depth-separable convolution layer is used to perform depth-separable convolution processing on the fifth convolution feature map, and input the third depth-separable convolution feature map to the third accumulation layer. The third accumulation layer accumulates the fifth convolution feature map and the third depth-separable convolution feature map to obtain the fifth feature map to be fused.

[0122] Continue to refer Figure 7 , the third depth-separable convolution layer includes a third depth-separated input layer, a third depth-convolution layer, a third point-convolution layer and a third activation function layer. The third depth-separated input layer is used to receive the fifth convolution feature map output by the third intermediate input layer, and input the fifth convolution feature map to the third depth-convolution layer and the third activation function layer respectively. The third depth-convolution layer is used to perform depth-convolution processing on the fifth convolution feature map, and input the third depth-convolution feature map to the third point-convolution layer. The third point-convolution layer is used to perform point-convolution processing on the third depth-convolution feature map, and input the third point-convolution feature map to the third activation function layer. The third activation function layer is used to process the third point-convolution feature map and the fifth convolution feature map to obtain the third depth-separated convolution feature map.

[0123] In the embodiments of the present disclosure, the activation function used by the activation function layer (for example, including the first to third activation function layers) can be any appropriate activation function. In the following embodiments, SiLU (Swish activation function) is used as an example, but the present disclosure is not limited to this. The SiLU function is differentiable in the entire domain, so the gradient calculation during the back propagation process is more stable. The SiLU function has the characteristics of no upper bound, lower bound, smoothness, non-monotonicity, and a larger and more stable gradient in the deep model.

[0124] Figure 7 In an embodiment, the convolution kernel sizes of the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer may all be 1×1.

[0125] A depthwise separable convolution layer is introduced into the cross-stage convolution block of the embodiment of the present disclosure. The depthwise separable convolution layer reduces the number of parameters required for convolution calculation and improves the efficiency of the use of convolution kernel parameters by splitting the correlation between the spatial dimension and the channel (depth) dimension. The convolution calculation process of the depthwise separable convolution layer is divided into two parts: depthwise convolution, which performs spatial convolution on each channel separately, that is, a single convolution kernel is used to perform convolution operation on each input channel, and then the outputs are spliced; pointwise convolution, which uses a unit convolution kernel (i.e., a 1x1 convolution kernel) to perform channel convolution to integrate the information of each channel and obtain the final feature map. The advantage of depthwise separable convolution is that it reduces the computational complexity and the number of model parameters while maintaining the expressiveness of the model. Depthwise separable convolution is a lightweight convolution method that improves the efficiency and performance of the model by reducing parameters and computation.

[0126] Figure 8 Example and Figure 3 The difference between the embodiments is that the feature extraction module 11 may further include a fourth cross-stage convolution block, and the detection head 12 may further include a fourth depth-separable convolution block. The structure of the fourth cross-stage convolution block can refer to the above Figures 4 to 7 . The fourth cross-stage convolution block is connected to the third cross-stage convolution block and the fourth depthwise separable convolution block in the detection head 12, and is used to respectively receive multiple third discarded feature maps, and process the multiple third discarded feature maps to generate multiple fourth discarded feature maps. The fourth cross-stage convolution block may include a fourth discard layer to output multiple fourth discarded feature maps to the fourth depthwise separable convolution block. The fourth depthwise separable convolution block may predict and output multiple fourth category prediction results and multiple fourth position prediction results of a fourth defect. The detection head 12 may obtain a fourth category detection result and a fourth category uncertainty of a fourth defect based on the multiple fourth category prediction results, and obtain a fourth position detection result and a fourth position uncertainty of a fourth defect based on the multiple fourth position prediction results.

[0127] Fig. 9 Example and Figure 3 One of the differences between the embodiments is that the deep learning defect detection model 1 further includes a feature pyramid module 13 located between the feature extraction module 11 and the detection head 12 .

[0128] Fig. 9The embodiment introduces a feature pyramid module 13 between the feature extraction module 11 and the detection head 12 to solve the multi-scale problem in defect detection. Defects of different sizes present different features in the image, and the feature pyramid can extract and fuse these features at different scales. By changing the network connection, the feature pyramid greatly improves the performance of small defect detection without substantially increasing the amount of calculation of the original model. Specifically, the feature pyramid fuses feature layers of different sizes through upsampling and downsampling operations. In this way, the model can simultaneously obtain high-level semantic information and low-level detail information, thereby extracting better features. This fused feature not only helps to improve the detection performance of the model, but also enables the model to better adapt to defect detection tasks of different sizes. First, the intermediate feature layer in the feature extraction module 11 generates feature maps of different scales. These feature maps contain increasingly abstract information from the bottom layer to the top layer in the pyramid structure, corresponding to defects of different sizes. The bottom feature map usually has a higher resolution and more detail information, which is suitable for detecting small defects; while the high-level feature map has a lower resolution and more abstract semantic information, which is suitable for detecting large defects. Next, the feature pyramid fuses these feature maps of different scales through a series of upsampling and downsampling operations. Specifically, the feature maps of the lower layers are upsampled to match their resolution with the feature maps of the higher layers, and then fused with the feature maps of the higher layers by splicing or adding them. In this way, the fused feature maps contain both low-level detail information and high-level semantic information, which helps improve the model's detection performance for defects of different sizes. By connecting feature maps of different scales, multiple prediction results of different scales can be generated. These prediction results correspond to defects of different sizes, thereby realizing multi-scale defect detection. This mechanism enables the model to pay attention to both global information and local details in the image, thereby improving the accuracy and robustness of defect detection.

[0129] For example, Fig. 9 As shown, the feature pyramid module 13 includes a first spatial pyramid pooling block, a second spatial pyramid pooling block and a third spatial pyramid pooling block. In the disclosed embodiment, the cross-stage convolution block can be set to implement downsampling of the feature map. Fig. 9 In the embodiment, feature maps of three different scales are input to the feature pyramid module 13 as input to the YOLO multi-scale detection head 12. Because the cross-stage convolution block can enable the downsampling function by defining parameters, more than three cross-stage convolutions can be used to input feature maps of three different scales to the feature pyramid module 13.

[0130] The first input end of the first spatial pyramid pooling block is connected to the output end of the first cross-stage convolution block, for example, connected to the output end of the first discarding layer, to receive multiple first discarding feature maps output by the first discarding layer. The second input end of the first spatial pyramid pooling block is connected to the output end of the second spatial pyramid pooling block to receive multiple second spatial pooling feature maps output by the second spatial pyramid pooling block. The first spatial pyramid pooling block processes each of the multiple first discarding feature maps and a corresponding second spatial pooling feature map in the multiple second spatial pooling feature maps, and outputs multiple first spatial pooling feature maps to the first depthwise separable convolution block in the detection head 12 through its output end. The first depthwise separable convolution block in the detection head 12 processes each of the first spatial pooling feature maps, and predicts multiple first category prediction results and multiple first position prediction results of the first defect. The detection head 12 performs Bayesian prediction on the multiple first category prediction results and multiple first position prediction results of the first defect to obtain the first category detection result, the first position detection result, the first category uncertainty and the first position uncertainty of the first defect.

[0131] The first input end of the second spatial pyramid pooling block is connected to the output end of the second cross-stage convolution block, for example, connected to the output end of the second discarding layer, to receive multiple second discarding feature maps output by the second discarding layer. The second input end of the second spatial pyramid pooling block is connected to the output end of the third spatial pyramid pooling block to receive multiple third spatial pooling feature maps output by the third spatial pyramid pooling block. The second spatial pyramid pooling block processes each second discarding feature map in the multiple second discarding feature maps and a corresponding third spatial pooling feature map in the multiple third spatial pooling feature maps, and outputs multiple second spatial pooling feature maps to the second depth separable convolution block in the detection head 12 through its output end. The second depth separable convolution block in the detection head 12 processes each second spatial pooling feature map respectively, and predicts multiple second category prediction results and multiple second position prediction results of the second defect. The detection head 12 performs Bayesian prediction on the multiple second category prediction results and multiple second position prediction results of the second defect to obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect.

[0132] The input end of the third spatial pyramid pooling block is connected to the output end of the third cross-stage convolution block, for example, connected to the output end of the third discard layer, to receive multiple third discard feature maps output by the third discard layer. The third spatial pyramid pooling block processes each of the multiple third discard feature maps respectively, and outputs multiple third spatial pooling feature maps to the third depth-separable convolution block in the detection head 12 through its output end. The third depth-separable convolution block in the detection head 12 processes each third spatial pooling feature map respectively, and predicts multiple third category prediction results and multiple third position prediction results of the third defect. The detection head 12 performs Bayesian prediction on the multiple third category prediction results and multiple third position prediction results of the third defect, and obtains the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect. Fig. 9 In an embodiment, the defect in the image to be processed further includes a third defect, the multiple category prediction results of the defect in the image to be processed obtained further include a third category prediction result of the third defect, and the multiple position prediction results further include a third position prediction result of the third defect. The category detection result, position detection result, category uncertainty, and position uncertainty of the defect in the image to be processed obtained further include a third category detection result, a third position detection result, a third category uncertainty, and a third position uncertainty of the third defect, respectively.

[0133] In the disclosed embodiment, the feature pyramid module 13 can fuse convolutional features of receptive fields of different sizes under the original feature maps of three scales, realize multi-scale feature fusion, and enhance the robustness of the features.

[0134] Fig. 9 Example and Figure 3 The second difference between the embodiments is that the deep learning defect detection model 1 may further include a background removal module located before the feature extraction module 11. The background removal module is used to remove background information in the image to be processed, and input the image to be processed after the background information is removed into the feature extraction module 11. Here, the image to be processed without removing the background information is taken as a mask cloth image for example, and the mask cloth image with the background information removed is referred to as the image to be processed.

[0135] Fig.10 Schematic diagram of a spatial pyramid pooling block of an exemplary embodiment of the present disclosure. Fig.10 As shown, each spatial pyramid pooling block (e.g. Fig. 9 The first to third spatial pyramid pooling blocks in the spatial pyramid pooling block may have the same structure. The spatial pyramid pooling block may include an input layer, a first convolutional layer connected to the input layer, three parallel maximum pooling layers connected to the first convolutional layer, a fusion layer connected to the three maximum pooling layers respectively, and a second convolutional layer connected to the fusion layer.

[0136] Each maximum pooling layer in the spatial pyramid pooling block can be used to reduce the number of parameters in the feature map, improve the calculation speed, and increase the receptive field. It is a downsampling operation. Through pooling, the model can pay more attention to global features rather than local locations. This dimensionality reduction process can retain some important feature information, improve fault tolerance, and also prevent overfitting to a certain extent.

[0137] Fig.10 In an embodiment, the convolution kernel sizes of the first convolution layer and the second convolution layer may both be 1×1.

[0138] Continue to refer Fig.10 , the convolution kernel sizes of the three parallel maximum pooling layers can be 3×3, 5×5, and 7×7 respectively.

[0139] It should be noted that the feature pyramid module in the embodiment of the present disclosure is not limited to using Fig. 9 and Fig.10 The spatial pyramid pooling block exemplified in the example may be replaced by other structures. Fig.10 In the embodiment, for the same feature map, convolution kernels of different sizes are used for parallel processing, and then the feature map after direct fusion processing is obtained. Fig.11 As shown in the figure, the feature maps of the input value feature pyramid module can be processed serially by convolution kernels of different sizes. For example, three convolution layers with convolution kernel sizes of 3×3, 5×5, and 7×7 are used in series, and then the feature maps of different convolution kernel outputs are jumped to achieve fusion by shortcut (direct connection or shortcut).

[0140] Fig. 9 Example and Figure 3The third difference between the embodiments is that the input block 111 can be an attention block 111'. The attention block 111' is a slice operation of the image to be processed (b1, c1, w1, h1 respectively represent the batch size, number of channels, width, and height of the image to be processed) before entering the feature extraction module 11. The specific operation is to get a value for every other pixel in an image to be processed, similar to neighboring downsampling, so that four images are obtained, and the four images are complementary, and there is no information loss. In this way, the w1 / h1 information is concentrated in the channel space, and the input channel is expanded by 4 times (the input of (b1, 4*c1, w1, h1) is obtained), that is, the spliced ​​image becomes 4*c1 channels relative to the original c channels. Finally, the new image is subjected to a convolution operation (for example, the convolution kernel size is 1x1), and finally a double down-sampled feature map without information loss is obtained, which is input to the first cross-stage convolution block. After such processing, the attention block 111' obtains a two-fold downsampled feature map without information loss, which can reduce the number of parameters, the number of network layers, the gradient and the number of calculations, while improving the precision and recall rate of the model. This enables the model to more accurately detect and identify defects in the image while maintaining high efficiency, especially those with small size and unclear features.

[0141] Fig.12 The embodiment uses a spatial attention mechanism to design the attention block. It is understandable that the attention block in the embodiment of the present disclosure is not limited to using Fig.12 The structure shown can also implement the channel attention block through the Squeeze-Excitation (compression-excitation, explicitly modeling the interdependence between feature channels, obtaining the importance of each channel through learning, and then weighting the features on each channel according to this importance, highlighting important features and suppressing unimportant features) operation. In addition, the spatial attention block can also be implemented through dynamic convolution.

[0142] It is understandable that the model provided by the embodiment of the present disclosure is not limited to the above-mentioned exemplified structure. The embodiment of the present disclosure provides adding a discard layer to the intermediate feature layer, adding a Bayesian loss function (such as the second loss function described below) during the training process of the model, and the method provided by the embodiment of the present disclosure can adapt to various network structures, realize the integration of uncertainty estimation while maintaining compatibility with various network architectures, provide flexibility in application, and further improve performance.

[0143] In the disclosed embodiments, through the use of convolutional layers, the model can be adapted to different image resolutions and blemish / defect detection scenarios. The convolution operation allows the network to focus on local features in the image rather than the entire image. Due to this property of the convolutional layer, the model is able to process input images of different sizes without having to make major modifications to the model structure. This makes the model highly flexible and can adapt to a variety of image resolutions. In the field of defect detection, this means that regardless of whether the input image is high-definition or low-resolution, or how the size and position of the defects in the image vary, the model using convolutional layers can effectively perform defect detection. Therefore, this model can easily adapt to different defect detection scenarios, improving the versatility and practicality of the model.

[0144] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: obtaining a target training data set, which includes a target sample image and annotated categories and annotated positions of target sample defects in the target sample image; inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defects; obtaining category detection results, position detection results, category uncertainty and position uncertainty of the target sample defects based on the multiple category prediction results and multiple position prediction results of the target sample defects; constructing a loss function based on the annotated category and category detection result, annotated position and position detection result, and category uncertainty and position uncertainty of the target sample defects to train the deep learning defect detection model.

[0145] In the disclosed embodiment, a data set for defect detection can be obtained, and divided into a target training data set and a target test data set according to a predetermined ratio, for example, a ratio of 9:1. During model training, a dropout operation is performed during the forward propagation process. Under the effect of dropout, multiple different network structures can be obtained by randomly closing neurons, thereby performing multiple predictions on the same target sample image to obtain multiple category prediction results and multiple position prediction results. The mean of multiple category prediction results and the mean of multiple position prediction results are calculated to represent the category detection result and position detection result of the target sample defect, respectively. The mean of multiple category prediction results and the variance of multiple position prediction results are calculated to represent the category uncertainty and position uncertainty of the target sample defect, respectively.

[0146] For example, the training parameters of the model can be set as follows: batch size is set to 64, number of epochs is set to 100, initial learning rate is set to 0.001, decay coefficient is set to 0.005, and dropout probability is set to 0.3. After dropout, the number of neurons in each layer becomes about half of the original number.

[0147] In an exemplary embodiment, the loss function includes a first loss function and a second loss function; when the difference between the labeled category and the category detection result of the target sample defect is greater, and the difference between the labeled position and the position detection result is greater, the value of the first loss function is greater; when the category uncertainty and the position uncertainty of the target sample defect are greater, the value of the second loss function is greater.

[0148] The disclosed embodiment introduces the Bayesian method, and adds dropout in the model to create random uncertainty to realize the Bayesian posterior distribution. At the same time, the detection head of the model is optimized, and the mean and variance of the anchor box (i.e., the position prediction result) and the category (i.e., the category prediction result) are added to represent their distribution. For example, the second loss function can be a negative log-likelihood loss function to optimize the uncertainty estimation, which not only promotes the evolution of the model towards accurate prediction, but also encourages the model to accurately estimate the uncertainty of its prediction, thereby improving the detection accuracy and robustness of the model.

[0149] For example, the first loss function L1 is as follows:

[0150]

[0151] MSE is the bounding box regression calculation, y i is the coordinate of the i-th bounding box in the real data (i.e., the marked position of the i-th target sample defect), is the i-th bounding box coordinate predicted by the model (i.e., the position detection result of the i-th target sample defect), N is the number of bounding boxes (i.e., the number of candidate defects in the target sample), N is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N. CE represents the multivariate cross entropy calculation, Represents the category confidence of category c in the predicted category detection result (i.e., the probability of belonging to category c). c The confidence of category c in the real data (i.e., the category confidence of category c in the labeled category). C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C. BCE represents the binary cross entropy calculation, o pre represents the predicted target confidence, and o represents the confidence of the target in the real data, that is, the confidence of the target marked in the target sample defect.

[0152] For another example, the second loss function L2 is as follows:

[0153]

[0154] μ i is the predicted position detection result of the defect of the i-th target sample, for example, the mean value of the i-th bounding box coordinates predicted by the model.i is the labeled position of the defect of the i-th target sample, for example, the i-th bounding box coordinates in the real data. is the position uncertainty of the predicted i-th target sample defect, for example, the variance of the i-th bounding box coordinates predicted by the model. N is the number of target sample defects, N is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N. μ c Represents the mean category confidence of the predicted category c, that is, the category detection result. c is the category confidence of category c in the real data. is the category confidence variance of the predicted category c, that is, the category uncertainty. C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C.

[0155] In an exemplary embodiment, obtaining a target training data set includes: obtaining an initial sample image and the annotated category and annotated position of an initial sample defect in the initial sample image; performing any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an augmented sample image and the annotated category and annotated position of an extended sample defect in the extended gain image. The target sample image includes the initial sample image and the augmented sample image.

[0156] In an exemplary embodiment, the initial sample image is subjected to defect synthesis processing to generate an augmented sample image, including: extracting initial sample defects from the initial sample image; synthesizing the initial sample defects with an augmented background image to generate the augmented sample image; and / or, learning the initial sample image through a deep learning generation model to generate the augmented sample image.

[0157] The quality of the dataset affects the accuracy and robustness of the model. For example, if we take the detection of defects in mask cloth images as an example, in order to train an efficient and accurate model, a large number of high-quality mask cloth images with defects are required to form a dataset. If the image quality in the dataset is low or the sample size is insufficient, the accuracy and robustness of the model may be affected.

[0158] In actual situations, the number of defective mask cloth images is less than the number of defective mask cloth images, that is, the number of positive and negative samples in the training data set is unbalanced. The present disclosure solves the problem of data set quality by implementing a multi-stage data enhancement strategy. First, a large number of original mask cloth images are collected, and initial training data (including initial sample images) are generated through expert annotation. Then, one or more of image processing techniques such as random cropping, rotation, scaling and illumination change simulation, as well as synthetic defect generation methods, are used to expand the data set to obtain an amplified sample image.

[0159] In the disclosed embodiment, the defective mask cloth image can be increased by using any one or more of the data enhancement methods of random cropping, rotation, scaling, illumination change simulation (such as brightness adjustment), defect synthesis, adding Gaussian noise, random occlusion, etc. The illumination change simulation is to randomly adjust one or more of the saturation, brightness, contrast, etc. of the original defective mask cloth image, which can simulate the image being easily affected by changes in light and weather; random occlusion is to block random areas of the defective mask cloth image with black, which can simulate the situation that it is easy to be blocked by other objects during shooting; Gaussian noise is to add noise whose probability density function obeys Gaussian distribution to the original defective mask cloth image, which can better simulate unknown real noise.

[0160] It is difficult for the models in the related art to achieve a good balance between detection speed and accuracy. This may lead to unsatisfactory detection performance of small and micro defects in practical applications, because small and micro defects require processing of high-resolution images, which increases the computing power burden and reduces the detection speed. The disclosed embodiment uses data enhancement to randomly scale and crop multiple initial sample images for splicing, which greatly increases the number of small and micro defects in the training data set and improves the effect of small and micro defect detection.

[0161] The disclosed embodiment can also generate new defect samples based on real defect samples by introducing AIGC (stable diffusion) to increase data diversity. Any one or two of the following two methods of synthesizing defects can be used: the first is to extract the defect foreground and replace the background of other fabrics. The second is to learn the real defect image through stable diffusion and then generate a fake defect image. By introducing AIGC into the field of defect sample generation, new defect samples can be generated based on real defect samples, which can enrich the data set and help train a more accurate model. The generated defect samples can be used to simulate various possible defect situations, so as to more comprehensively test the performance of the defect detection system in practical applications.

[0162] In an exemplary embodiment, obtaining a target training data set includes: obtaining candidate sample images and the annotated categories and annotated positions of candidate sample defects in the candidate sample images; obtaining output features of the candidate sample images through the feature extraction module; obtaining feature distances between the candidate sample images based on the output features of each candidate sample image; and selecting candidate sample images whose feature distances are greater than a first distance threshold among candidate sample images with the same annotated category, and / or candidate sample images whose feature distances are less than a second distance threshold among candidate sample images with different annotated categories as target sample images in the target training data set.

[0163] In the embodiment of the present disclosure, Euclidean distance may be used as the feature distance, but the present disclosure is not limited thereto.

[0164] The disclosed embodiment may also apply an adaptive sample selection mechanism to preferentially select those samples that are most effective in improving model performance for training. For example, during the training process, the features of the backbone network output of multiple samples are extracted to capture the similarities and differences between different samples. The feature distance of each sample is then calculated.

[0165] For example, the i-th sample (x i ) and the jth sample (x j ) between the characteristic distance d ij :

[0166]

[0167] F k (x i ) is the sample x i The eigenvalue in the kth dimension, F k (x j ) is the sample x j The eigenvalue of the kth dimension, D is the dimension of the sample feature, D is a positive integer greater than or equal to 1, and k is a positive integer greater than or equal to 1 and less than or equal to D. i and j are both positive integers greater than or equal to 1, and i is not equal to j.

[0168] Calculating the characteristic distance of samples helps to understand the distribution of samples in the feature space. For example, you can select the farthest samples among the same type of samples (images corresponding to the same type of defects, that is, different samples with the same label, such as samples all labeled as "tape") and the closest samples among the different types of samples (images corresponding to different types of defects, such as one sample labeled as "stain" and another sample labeled as "crease") to participate in back propagation, which is equivalent to using the most error-prone (most difficult to judge) samples to optimize the model weights to improve the accuracy and robustness of the model.

[0169] In the disclosed embodiment, the sample selection criteria of the adaptive sample selection mechanism is to first count the feature distance, and then divide the first distance threshold and the second distance threshold according to the statistical value of the feature distance, for example, the first distance threshold (also called intra-class distance) is selected to be greater than 0.3, and the second distance threshold (also called inter-class distance) is selected to be less than 0.2. The adaptive sample selection mechanism is a training strategy that is added after conventional training and is used for secondary training to solve the hard sample problem. After secondary training, the feature distribution of intra-class samples is more concentrated, and the feature distribution of inter-class samples is more discrete, thereby enhancing classification accuracy.

[0170] Based on the calculation results of feature distance, selecting the farthest samples among the samples of the same type means that these samples have the greatest difference in feature space, and even if they belong to the same category, it is difficult for the model to distinguish them correctly. By letting the model learn these difficult-to-distinguish samples, the model can be prompted to pay more attention to the detailed differences within the category, thereby improving the accuracy of the model. At the same time, selecting the closest samples among samples of different types means that these samples have a high similarity in feature space, but in fact they belong to different categories. These samples are the most error-prone samples for the model because they have similar features but different labels. By letting the model learn these samples, the model's ability to distinguish between different categories can be strengthened, thereby improving the robustness of the model. Using this strategy for backpropagation is equivalent to using the most error-prone (most difficult to judge) samples to optimize the model's weights. In this way, the model will pay more attention to these challenging samples during training, thereby improving its generalization ability and robustness when facing complex and changing environments.

[0171] In an exemplary embodiment, during the training process of the deep learning defect detection model: extract the weight tensor of each convolutional layer in the deep learning defect detection model; obtain the Euclidean norm of each channel in the weight tensor; sort each convolutional layer according to the Euclidean norm of its channel, and prune the corresponding channel according to a pruning threshold or a pruning ratio.

[0172] Channel pruning is an effective model compression and acceleration technology, especially suitable for the needs of lightweight model design. Channel pruning in the model can significantly reduce the number of model parameters and computational complexity while maintaining the performance of the model.

[0173] In the disclosed embodiment, the contribution of the convolution channel to the network output is measured based on L2-norm (Euclidean norm), and channel pruning is performed according to the degree of contribution. First, the contribution of the convolution channel is measured based on L2-norm: L2-norm (Euclidean norm) can be used to evaluate the importance or contribution of the convolution channel. For each convolution layer in the model, the L2-norm value of each convolution channel weight can be calculated. This value reflects the influence and contribution of the channel in the network. Specifically, for each convolution layer in the model, its weight tensor is extracted. The L2-norm value of each channel in the weight tensor is calculated. For example, operations such as squaring, summing and square rooting are performed on the weight tensor. The calculated L2-norm value is saved for subsequent channel pruning decisions. Then, channel pruning is performed according to the degree of contribution: According to the L2-norm value calculated in the previous step, the importance of each convolution channel can be evaluated, and channel pruning can be performed according to a certain strategy. The goal of channel pruning is to remove channels that contribute less to the network output, thereby reducing the number of parameters and computational complexity of the model. Specifically, a pruning threshold or pruning ratio is set. The pruning threshold or pruning ratio can be determined according to specific needs and experiments. For each convolutional layer, sort them according to the L2-norm value of its channels. According to the set pruning threshold or pruning ratio, select channels with smaller contributions for pruning. For example, set the weights of the corresponding channels to zero or directly remove these channels from the model. After pruning, retrain and fine-tune the model to restore or maintain its performance. The Euclidean norm reflects the response weight of the channel to the output, and channels with lower response weights are pruned first.

[0174] In some embodiments, in order to achieve a balance between detection speed and accuracy, the deep learning defect detection model is lightweight designed, the number of feature channels is reduced through channel pruning, and the original convolution layer is replaced with a depthwise separable convolution to reduce the parameters and computational complexity of the model, thereby achieving faster detection while maintaining high accuracy.

[0175] In an exemplary embodiment, the target sample images include mask images (or mask cloth images) having different materials, textures, colors, and prints.

[0176] In order to increase the generalization ability of the model, the embodiment of the present disclosure designs a mixed data set. Taking the mask cloth image as an example, the mixed data set contains multiple types of mask samples, such as mask images with different mask materials, colors, textures, and print patterns.

[0177] For certain types of mask materials, such as mask cloth with printed patterns, the pattern types may vary greatly, and the training samples cannot contain mask sample images of all patterns, resulting in the model being unable to achieve high detection accuracy. In the disclosed embodiments, transfer learning and domain adaptation techniques can be used, including one or more of feature space alignment, fine-tuning, metric learning, etc., to enable the model to adapt to new materials, colors, textures, printed patterns, etc.

[0178] In some embodiments, the model is periodically retrained on newly collected data to maintain its generalization performance.

[0179] In an exemplary embodiment, obtaining an image to be processed includes: obtaining the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is used to obtain the image to be processed of the object to be detected in the production process. The method further includes: obtaining the current production speed of the object to be detected and the current processing speed of the deep learning defect detection model for the image to be processed; and adjusting the number of parallel threads or parallel processes used by the deep learning defect detection model to process the image to be processed according to the current production speed and the current processing speed.

[0180] A real-time detection feedback mechanism is introduced in the disclosed embodiment to dynamically adjust the model inference time (i.e., the current processing speed of the model) by changing the number of parallel threads of model inference according to the current production line speed (or current production speed), thereby ensuring that the required detection speed is achieved while maintaining high accuracy.

[0181] For example, the model is deployed using TensorRT after pruning, and the encapsulated TensorRT architecture supports multi-process and multi-threaded concurrency. At the same time, considering that long-term high load on the hardware will affect its lifespan, a dynamic adjustment mechanism is designed. In the disclosed embodiment, a thread pool can be used to manage the concurrent execution of reasoning tasks. The thread pool can ensure the effective use of thread resources and avoid the overhead caused by frequent creation and destruction of threads. Design a task scheduler to assign reasoning tasks to idle threads. A load balancing strategy can be used to ensure that the loads of each thread are relatively balanced to avoid situations where some threads are overloaded while other threads are idle.

[0182] TensorRT natively supports multi-threaded reasoning, which can be achieved through execute_async_v2 of IExecutionContext. In addition, the disclosed embodiment also takes advantage of the multi-core CPU (Central Processing Unit) to serialize the TensorRT engine to disk, and then loads this serialized engine in multiple processes, so that each process can perform reasoning tasks independently, and maximize the resources of the CPU and GPU (Graphics Processing Unit).

[0183] The disclosed embodiments can use sensors or other monitoring devices to obtain the current production speed in real time; the current reasoning speed of the monitoring model, that is, the time required to process each image is used as its current processing speed. According to the comparison between the current production speed and the current processing speed of the model, the number of parallel reasoning threads of the model is adjusted. If the current production speed is accelerated, the number of parallel threads needs to be increased to speed up the reasoning speed; conversely, if the current production speed is slowed down, the number of parallel threads can be reduced to save resources. By combining the real-time detection feedback mechanism, dynamically adjusting the number of parallel threads of model reasoning, and the efficient deployment of TensorRT, it can be ensured that the defect detection tasks on the production line are both accurate and efficient.

[0184] In an exemplary embodiment, obtaining an image to be processed includes: obtaining the image to be processed from an image capturing device in an image acquisition component; if the grayscale value and grayscale distribution of the image to be processed do not meet preset conditions, sending a control signal to adjust the exposure parameters of the image capturing device; obtaining a new image to be processed from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset conditions is obtained.

[0185] In order to improve the robustness of the model to environmental factors, an automatic exposure module is designed in the embodiment of the present disclosure. Each time the system is turned on, an image in the current environment is captured by an image capture device such as a camera, and its grayscale distribution is analyzed. If the grayscale value and grayscale distribution of the image do not meet the preset conditions, the exposure parameters are adjusted, and the image is captured again for analysis. This cycle continues until the grayscale value and grayscale distribution of the captured image meet the requirements, and then it is used as the image to be processed for the input model. The automatic exposure module can monitor the lighting conditions in real time, calculate the latest exposure parameters of the camera in real time, and suppress the influence of external light.

[0186] For example, the desired preset condition is histogram equalization, the grayscale mean is between 130 and 140, and the smaller the grayscale variance, the better. The grayscale histogram, grayscale mean, and grayscale variance of the image captured by the camera are used to determine whether the preset condition is met. If not, the exposure is increased or decreased based on the target difference.

[0187] In an exemplary embodiment, the method provided by the embodiment of the present disclosure further includes: if the category uncertainty and / or location uncertainty of the defect is greater than an uncertainty threshold, sending prompt information for the defect.

[0188] For example, during online operation, the entire system is end-to-end. It takes the acquired mask cloth image as input, and the model first deletes irrelevant background to obtain the image to be processed. Then, it performs feature extraction, and finally outputs the category detection result, position detection result, category uncertainty and position uncertainty of the defect. The system can determine that the defect with category uncertainty and / or position uncertainty greater than the uncertainty threshold (whose value can be set according to actual needs) has a high uncertainty defect, and a prompt message can be triggered at this time to prompt the user that the prediction of the defect has high uncertainty. Further, the user can determine how to deal with the defect, such as whether to manually mark the location and category of the defect. That is, the disclosed embodiment can also use the category uncertainty and / or position uncertainty predicted by the model to assist the user in identifying the most informative data for marking, thereby improving the efficiency of the annotation process.

[0189] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: counting pixel quantity information of each defect detected in the image to be processed; determining the defect level of each defect based on the pixel quantity information of each defect; and performing corresponding processing based on the defect level of each defect.

[0190] The defect detection system provided by the embodiment of the present disclosure includes an algorithm layer and a data service layer / data service component. The algorithm layer is deployed with the above-mentioned deep learning defect detection model to process the image to be processed and obtain defect data, such as whether the image to be processed has defects. If there are defects, it contains defect information such as the category detection result, position detection result, category uncertainty and position uncertainty of the defect. The data service layer is responsible for analyzing and processing the defect data provided by the algorithm layer to determine the severity of the defect. For example, by counting the number of pixels of the defect (such as the number of pixels contained in the long side of the defect), the defects are divided into four levels: the first level is a slight defect, such as a defect with less than 20 pixels on the long side; the second level is a general defect, such as a defect with more than or equal to 20 pixels on the long side and less than 60 pixels; the third level is a more serious defect, such as a defect with more than or equal to 60 pixels on the long side and less than 100 pixels; and the fourth level is a serious defect, such as a defect with more than 100 pixels on the long side. Based on this grading system, the data service layer can formulate and execute corresponding processing strategies for defects of different levels. For example, when a serious defect is detected, an alarm is directly triggered to prompt the user (such as a worker) to handle it immediately. Minor defects may not trigger an alarm, but only prompt on the interactive interface, and the user can decide whether to handle it. This ensures the efficiency and reliability of defect detection and processing.

[0191] In an exemplary embodiment, corresponding processing is performed according to the defect level of each defect, including: in the image to be processed, defects at different defect levels are marked with frames with different thicknesses.

[0192] In the disclosed embodiment, the data service layer uses the thickness of the border of the defect area to assist in determining the classification level of the defect. The thicker the border, the higher the defect level, and the thinner the border, the lower the defect level. For example, the data service layer first counts the number of pixels of each defect, and then divides the defects into four levels according to the number of defect pixels. The first level of defects are then marked with the thinnest border in the image to be processed, and the fourth level of defects are marked with the thickest border in the image to be processed. The marked defect image is then provided to the user and displayed on the interactive interface, so that the user can immediately identify the severity of the defect and quickly decide whether to stop the machine for processing. This visual identification allows the operator to quickly and intuitively understand the severity of the defect during the subsequent manual inspection process.

[0193] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: saving the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed; receiving and retrieving corresponding defect information based on one or more of the input image features, detection time and detection serial number of the defect area.

[0194] The defect detection system provided by the embodiment of the present disclosure may also include a defect recording module. When a defect is detected, the defect recording module will automatically record the level of the defect (i.e., defect level), the exact location (i.e., location detection result), and the detailed regional image of the defect point (i.e., defect or defect point). In order to further improve efficiency, the data service layer may also provide a defect point information retrieval function. The image features of the defect / defect area (for example, the features extracted by the feature extraction module), the detection time, and the detection sequence number will be stored in the database and can be retrieved through the above three data. Through this function, historical defect records can be quickly retrieved and reviewed to support continuous improvement of quality according to the production process.

[0195] When the defect detection method and system are applied to the defect detection of non-woven mask cloth, it can also be called a non-woven mask cloth defect detection method and system. When the defect detection method and system are applied to the defect detection of hygienic and personal care product materials, it can also be called a hygienic and personal care product material intelligent defect detection method and system.

[0196] The embodiment of the present disclosure also provides a defect detection system, including: an image acquisition component, used to obtain the image to be processed of the object to be detected in the production process; a defect detection device, connected to the image acquisition component, used to obtain the image to be processed from the image acquisition component.

[0197] Among them, the defect detection device includes: an image processing module, which is used to process the image to be processed through a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, and the intermediate feature layer includes a discard layer, and the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding after being processed by the feature extraction module; the multiple features after random discarding are processed by the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; according to the multiple category prediction results and multiple position prediction results of the defects, the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed are obtained.

[0198] In an exemplary embodiment, the image acquisition component includes: an image capture device, wherein the image capture device includes an area array camera. The area array camera is used to capture an original image to be processed and perform pixel merging processing on the original image to be processed to generate the image to be processed.

[0199] An area array camera is a camera used to obtain images of the surface of an object to be inspected. It can use different types of sensors, such as CCD (Charge-Coupled Device) or array camera or CMOS area array camera, which consists of a fixed array of pixels, each pixel unit can capture an image at the same time, so that the entire image can be collected at one time. Area array cameras can provide higher resolution and sensitivity, especially in low light conditions.

[0200] In some embodiments, binning can be used to optimize the basic camera algorithm to fuse multiple real pixels of an array camera such as a CCD camera, thereby achieving a clear image with a very short exposure time, which is an effective high-speed image capture method. During the pixel merging process, the pixels in the image are merged into "super pixels" or "virtual pixels", for example, 2×2 or 3×3 pixels are merged into one virtual pixel to enhance the image, so that the camera can shoot at a very low exposure time without underexposure, thereby avoiding blur and smearing caused by long exposure time.

[0201] In CCD cameras, the pixel binning process is performed in hardware or "on chip". The merged super-pixels have higher signal strength, which helps to obtain clearer images with very short exposure times. Due to the shortened exposure time, the camera can capture images faster and continuously, thus achieving high-speed image capture.

[0202] For example, the area array camera achieves pixel fusion through binning technology, thereby improving the initial brightness of the image at the hardware layer, ensuring that the camera can capture high-speed moving mask fabric within a shorter exposure time and avoiding motion blurred images.

[0203] In an exemplary embodiment, the image acquisition component further includes: a bar light source, located at one side of the image capture device, and irradiating the object to be inspected moving on the production line at a predetermined angle.

[0204] A bar light source is a special light source with a narrow and long shape, usually using LED or CCFL as the light source. The brightness of a bar light source is more uniform than that of an ordinary light source, which makes it very suitable for lighting or display, and can provide more uniform lighting within a certain space. Bar light sources can be customized according to specific usage scenarios, including adjusting their length, brightness, color and other parameters to meet different lighting needs and achieve the best lighting effect. Bar light sources have high durability and long life, and can maintain stable light output for a long time. Bar light sources can not only adjust brightness and color, but also meet the application needs of various scenarios by changing the shape and direction of the light beam. Bar light sources use LED technology, which is highly efficient and energy-saving, and is more energy-efficient than traditional fluorescent or incandescent lamps. At the same time, it also has better environmental performance, which helps to reduce energy consumption and environmental pollution.

[0205] It should be noted that in the defect detection system of the embodiment of the present disclosure, the camera may also have its own light source. In other embodiments, other light sources may also be used, such as a surface light source, a ring light source, etc.

[0206] In an exemplary embodiment, the predetermined angle is 45 degrees.

[0207] For example, a bar light source is located on one side of the camera and illuminates the mask fabric moving on the production line at a 45-degree angle. This specific lighting angle can maximize the visual contrast of defects, making them more obvious in the image captured by the camera. First, the oblique lighting method can produce shadows and reflections, which help to highlight the unevenness or defects on the surface of the mask fabric. When light is shone on the fabric at a certain angle, the defective part may produce different reflection or absorption effects due to different shapes, textures or depths. In this way, the camera can more easily distinguish the difference between defects and normal parts when capturing the image. The 45-degree angle ensures that the light can fully cover the surface of the fabric, while avoiding the light being too direct or too oblique, resulting in heavy or light shadows. Appropriate shadows and reflections help enhance the contrast of defects and make them more prominent in the image. The narrow and long shape of the bar light source allows the light to be concentrated in a narrow area, thereby improving the uniformity and consistency of lighting. This is especially important for inspecting mask fabrics moving on the assembly line, because the movement of the fabric may cause changes in lighting conditions. The stability and consistency of the bar light source helps ensure that the image quality captured by the camera is stable and reliable. By properly setting the position and angle of the bar light source, the visual contrast of the defects can be maximized, and the quality and accuracy of the images captured by the camera can be improved, thereby achieving effective detection of mask fabric defects.

[0208] It should be noted that the above 45 degrees does not require strict accuracy, and may have a certain error range, for example, greater than ±3 degrees.

[0209] In an exemplary embodiment, the system includes an industrial computer component, the industrial computer component includes an industrial computer, and the defect detection device is deployed on the industrial computer.

[0210] In an exemplary embodiment, a data service component is also deployed on the industrial computer, and the data service component is used to record and count the defect information detected in the production process.

[0211] In some embodiments, the data service component manages all defect data (defect information, such as whether there is a defect in the image, the defect category, location, category uncertainty and location uncertainty, defect level, and any other data related to the defect) through MySQL.

[0212] In other embodiments, the data service component can upload the detected defect information to the cloud server in real time, so as to centrally manage the data and facilitate remote monitoring, analysis and optimization of the production process.

[0213] In an exemplary embodiment, the system provided by the embodiment of the present disclosure further includes: an alarm component connected to the defect detection device, configured to receive a trigger signal sent by the defect detection device when the defect detection device detects a defect, and send an alarm message according to the trigger signal. For example, the alarm component may include an alarm, but the present disclosure is not limited thereto.

[0214] In an exemplary embodiment, the system provided by the embodiment of the present disclosure further includes: a display connected to the defect detection device, and used to display defect information detected by the defect detection device.

[0215] Combine the following Figures 13 to 15 The system provided in the embodiment of the present disclosure is illustrated by way of example, but the present disclosure is not limited thereto.

[0216] like Fig.13 As shown, the defect detection system 120 may include an image acquisition component 121 and a defect detection device 122. For example, the defect detection device 122 may be set in a high-performance industrial computer component. The defect detection device 122 may include an image processing module 1221. The image acquisition component 121 includes an image capture device 1211 (e.g., a surface array camera) and a light source 1212 (e.g., a strip light source). The image capture device 1211 is connected to the industrial computer component. The industrial computer component includes an industrial computer.

[0217] Optionally, the defect detection system 120 may further include an alarm component 123, and the alarm component 123 may be connected to the industrial computer component.

[0218] For example, a high-speed array camera is connected to a high-performance industrial computer, and the collected images are transmitted back to the industrial computer in real time. The alarm component 123 is connected to a high-performance industrial computer. When the industrial computer detects that there are defects / defects in the cloth image / mask cloth image, it sends a trigger signal to the alarm, and the alarm sends an audible and visual signal as an alarm information to prompt the worker to handle the corresponding defective mask. Among them, the algorithm recognition component / algorithm layer and the data service component are all run on a high-performance industrial computer.

[0219] Fig.14 An exemplary scenario of a defect detection system for mask production is shown. Fig.14 As shown, the hardware of the defect detection system 120 may include an image acquisition component / system (including a high-speed camera 1211' and a strip light source 1212', responsible for collecting image information, such as images to be processed), a high-performance industrial control computer (IPC, an abbreviation of industrial control computer, responsible for processing the collected image information and generating detection results, such as category detection results and position detection results as well as category uncertainty and position uncertainty) 122', and optionally, an alarm may also be included. There is a camera link (Camera Link interface) 124 between the high-performance industrial control computer 122' and the high-speed camera 1211'. The high-performance industrial control computer 122' can be connected to the strip light source 1212' via RS232. The high-speed camera 1211' and the strip light source 1212' are located before the cutter 125 of the mask material. It is assumed here that the mask material is non-woven fabric. The mask material or mask cloth moves toward the cutter 125 on the cloth guide roller 126 along the movement direction of the mask non-woven fabric. An angle θ between the light emitted by the bar-shaped light source 1212 ′ toward the mask cloth on the cloth guide roller 126 and the mask cloth may be, for example, 45 degrees.

[0220] refer to Fig.14 The mask cloth image / material image captured by the camera 1211' (e.g., a camera) is processed by the IPC for defect / flaw identification. If a defect is identified in the image, the IPC sends an audible and visual signal through an alarm to notify the workers on the assembly line to reject the defective product.

[0221] from Fig.14It can be seen that the high-speed camera 1211' and the light source (e.g., a bar light source 1212') are installed in a position before the mask material cutting machine 125. Since the space in the detection environment is very limited and the speed of moving the mask material on the production line is very fast, the camera needs to capture images in a short time (e.g., at least 25 frames per second). However, when the camera captures images at such a high speed, the image becomes blurred and distorted due to the limitation of the exposure time. Therefore, the embodiment of the present disclosure uses binning to fuse 2*2 pixels into a virtual pixel to enhance the image, so that the camera can shoot at a very low exposure time without underexposure, thereby avoiding blur and smear caused by long exposure.

[0222] The detection system proposed in the embodiment of the present disclosure overcomes the limitations of the related art that it relies too much on mechanical structures and does not adequately describe visual recognition methods. The system uses advanced image capture equipment, intelligent recognition technology combined with deep learning algorithms, and efficient data service support to achieve real-time and accurate detection of defects in high-speed sports mask fabrics.

[0223] like Fig.15 As shown, the defect detection system (also referred to as an intelligent defect detection system) provided by the embodiment of the present disclosure includes hardware, an algorithm layer / component (also referred to as a detection algorithm layer) and a data service layer / component. The hardware components of the system include an industrial computer, a high-speed camera (high-speed camera), a light source and an alarm, which work together to capture images and issue warnings when defects are detected. The algorithm layer includes technologies such as preprocessing, channel pruning, deep separable convolution and Bayesian optimization, which effectively improve the detection speed and accuracy. The data service component includes image and data storage, defect data analysis, defect data reporting and automated defect labeling. The detection algorithm is the core part of the detection system, and the hardware and data services make the defect detection system faster, more accurate and easier to use.

[0224] The camera may be, for example, a CIS camera (CMOS Image Sensor, complementary metal oxide semiconductor image sensor), a CCD camera, or the like.

[0225] For example, the camera and light source are responsible for collecting image information on the surface of the mask cloth to obtain the image of the mask cloth; the industrial computer is responsible for processing the collected image and identifying defects on the surface of the mask cloth; the alarm is used to remind workers to deal with defective masks that do not meet quality standards. When the mask cloth image collected by the camera is processed by the industrial computer for defect identification, if a defect is identified in the image, the industrial computer will send an audible and visual signal through the alarm to notify the workers on the production line to remove the defective products.

[0226] The algorithm layer is the core of the system intelligence and is responsible for defect location and classification. The algorithm layer includes image preprocessing, channel pruning, deep separable convolution processing, and Bayesian optimization modules. The role of preprocessing is to preprocess the image before defect detection, and improve the difference between defects and background through image enhancement and filtering, so as to better identify defects; channel pruning is based on the needs of lightweight model design, and the channel pruning of the model is carried out. The contribution of the convolution channel to the network output is measured by L2-norm, and the channel is pruned according to the contribution degree to reduce its model parameters and improve the processing speed; deep separable convolution processing is also a model lightweight means. Through Deepwise Conv and Pointwise Conv to simplify the convolution operation process, deep separable convolution can reduce 85% of the operation parameters compared with ordinary convolution operations, while ensuring that the accuracy is not lost; Bayesian optimization is to optimize the model using the Bayesian method. First, add a dropout layer to the model to create randomness to simulate the Bayesian posterior distribution to enhance the accuracy and reliability of defect detection. The Bayesian method can quantify the uncertainty of the prediction. In a detection system, the model not only predicts outcomes, but also provides information about how confident those predictions are. This quantification of uncertainty allows the system to process information more intelligently, such as taking a more cautious approach to predictions that the model is not confident about.

[0227] In this embodiment, the detection algorithm can predict four main types of mask material defects: stickers (stickers refer to the connection positions between each roll of mask cloth, which are pasted together on the assembly line by adhesive tape, so they can also be called adhesive tape), stains (stains), printing problems (some masks are patterned and printed, and there will be printing errors) and mask cloth creases. During online operation, the entire system is end-to-end. It takes the acquired image as input, and the model first removes irrelevant background, which refers to the area other than the mask cloth captured in the image. Efficient image feature extraction is achieved through channel pruning and depth-separable convolution. Then, Bayesian methods for random and epistemic uncertainty are incorporated into the detection architecture. Random uncertainty stems from the randomness of the training data itself. In the context of mask cloth defect detection, this may include uncertainty caused by natural variation in mask cloth, slight differences in the production process, or noise in the image acquisition process. Random uncertainty is usually irreducible because it reflects the inherent variation in the data itself. Epistemic uncertainty stems from the uncertainty of model parameters and is about the uncertainty in the model's understanding and inference of data. In the defect detection of mask cloth, cognitive uncertainty may come from the uncertainty of the model in identifying specific types of defects, or the model's adaptability to new materials and new types of defects. The Bayesian method includes: adding a dropout layer to the intermediate feature layer of the backbone to simulate the effect of the Bayesian posterior distribution to obtain different outputs about the features. The distribution of these outputs provides an approximation of the prediction uncertainty, where the variability of the prediction results reflects the uncertainty of the model; modifying the detection head, adding the mean and variance of each parameter on the basis of the model output including the coordinates, category and confidence of the bounding box to represent the measure of prediction uncertainty; adjusting the loss function, adding a negative log-likelihood loss, and penalizing excessive uncertainty and inaccurate predictions. Specifically, a negative log-likelihood loss function of the detection head is added to calculate the Bayesian probability loss. There are two functions: first, punishing excessive uncertainty (because if the model is very uncertain about its own predictions, the loss will increase). Second, penalizing inaccurate predictions (large prediction errors will increase losses).

[0228] The disclosed embodiment optimizes the defect category, location and uncertainty prediction process of the mask fabric defect detection system. The output of the improved detection head is a probability distribution, and a new Bayesian loss, such as a negative log-likelihood loss, is added to increase the penalty for uncertain results, thereby optimizing back propagation. Dropout is added to the backbone intermediate feature layer to simulate the uncertainty of the model parameters and realize the Bayesian posterior distribution. Data uncertainty is determined by data noise, annotation variance, and sample quality (such as lighting, resolution, and contrast). Through the above improvements, the model estimates the uncertainty of its predictions, so that the model improves its robustness and reliability. The implementation of these innovations is important for improving the accuracy and efficiency of the mask fabric defect detection system, especially in a high-speed production environment. Through the combination of these technologies, effective detection of high-speed motion mask fabrics can be achieved, while improving the robustness and reliability of the system.

[0229] The data service layer is the information processing and decision support center in the mask fabric defect detection system. It is responsible for integrating the output of the algorithm layer and performing subsequent data management and decision making. The data service layer includes defect and image data storage, defect data aggregation, defect data transmission, and automatic decision-making modules. Image and defect data storage is responsible for collecting and saving all the original image data obtained through the image acquisition system and the result data processed by the algorithm layer (including one or more of the location of the defect, the defect type of the defect, the uncertainty of each defect, and the image characteristics of the defect area). Through efficient data organization and indexing mechanisms, the queryability and accessibility of the data are ensured. Among them, the defect data analysis will unify and analyze the various types of defect data detected, and summarize the scattered defect information into an intuitive report to facilitate monitoring and evaluation of production quality. Defect data transmission ensures that the detected defect information can be quickly and accurately transmitted to the downstream links or quality control departments of the production line, providing support for real-time response and handling of defects. Automatic decision making is based on aggregated analysis of defect data and the actual situation of the production line. This module uses an intelligent decision-making algorithm to analyze the distribution of defect detection and uncertainty probability, and generates reports to help producers understand which types of defects are most common in the current mask production, the frequency of serious defects, etc., so that producers can understand the defects of the production process from a more macro perspective to optimize the production process.

[0230] The data service layer is responsible for storing, analyzing and transmitting inspection data, and implementing corresponding processing strategies according to the severity of defects. In addition, the data service layer also enhances the efficiency of manual inspection, intuitively displays the defect level / defect level through the thickness of the defect border, and provides the function of retrieving defect information / defect information, supporting quality tracking and production process optimization.

[0231] The disclosed embodiments relate to a mask cloth defect detection system in a high-speed production environment. The system integrates high-performance image capture hardware (camera and light source), advanced deep learning algorithms (including the deep learning defect detection model mentioned in the above embodiments), and a comprehensive data service layer to achieve real-time defect detection and classification of high-speed moving mask fabrics. Overall, the present disclosure provides an efficient and reliable solution to meet the demand for high-quality masks on fast production lines.

[0232] The mask cloth defect detection system provided by the embodiment of the present disclosure, by setting an image acquisition component, an algorithm layer and a data service component, can efficiently and accurately detect defects on the mask cloth passing through without affecting the speed of the mask cloth production line, and automatically mark the defects when defects are detected, greatly improving production efficiency and product quality. Through the combination of manual and automated detection, the system also ensures a high level of detection reliability. For example, in combination with the Bayesian method, an uncertainty score will be given for each predicted defect point (for example, refer to the above formula (2)). If the uncertainty score of a defect point is high, the defect point will be sent to manual marking.

[0233] "High speed" in the embodiments of the present disclosure means satisfying the global shutter, and the maximum frame rate is, for example, more than 60fps. High speed means, for example, that when shooting mask cloth at a mask production speed of 500 pieces / min on an assembly line, the image does not appear to be dynamic blurred, and the camera frame sampling interval does not lose the mask cloth area information.

[0234] According to an implementation aspect of the present disclosure, a mask cloth defect detection system is provided, which includes: an image acquisition component, including a high-speed array camera, a strip light source, and a camera and light source bracket. The high-speed array camera is used to collect the surface image of the mask cloth, and the light source is used to highlight the visual difference between the defect and the background; a high-performance industrial computer component, including an industrial computer equipped with a high-performance GPU and a display, the industrial computer is used to detect the defects in the image collected by the camera in real time, and the display is installed at the end of the mask production line to facilitate workers to confirm the image information of the mask defect; an alarm component is installed at the end of the mask production line. When a defect is detected, the alarm triggers to prompt the worker to deal with it in time; a defect recognition algorithm component improves the model by analyzing the defect characteristics and image characteristics, and optimizes its calculation speed and accuracy. Among them, the image characteristics may include one or more of the changes in the background in the image, the changes in the surface texture of the detected object, the interference of the surrounding ambient light on the image, etc. The processing involving image characteristics is to adjust the camera exposure and remove the image background.

[0235] In the disclosed embodiment, the defect point characteristics are the anchor box distribution characteristics of the defect points and the Euclidean distance between the defect point feature classes. For example, the anchor box distribution characteristics of the defect points and the Euclidean distance between the defect point feature classes are analyzed. Clustering is performed according to the anchor box distribution characteristics to generate the most prioritized anchor box, and sorting and comparison are performed according to the Euclidean distance between the defect point feature classes to optimize the defect point classification accuracy.

[0236] In the embodiments of the present disclosure, there are two ways to analyze defect characteristics / flaw properties:

[0237] One is to generate the best prior anchor frame: for example, cluster the labeled defect anchor frames through k-means to find the best prior anchor frame, so as to analyze the length-width ratio distribution of the defects.

[0238] The other is hard sample mining and optimization: first, according to the distribution of anchor boxes, find defects of different proportions. For example, for defects with an aspect ratio of 1:1, for defect A, calculate the Euclidean distance between A and other types of defects with an aspect ratio of 1:1. Then find the closest defect of a different type and mark it as N (Negative), and find the farthest defect of the same type and mark it as P (Positive). Then these three defect samples constitute a hard sample triplet. By constructing multiple hard sample triplets of defects with different aspect ratios, a hard sample dataset is finally constructed. Then, the model is deeply trained on the hard sample dataset to make the model more suitable for this hard sample, thereby achieving better detection results. That is, the most difficult defects to distinguish are selected for targeted optimization.

[0239] The advantages of bar light sources are high power and high light intensity. Because the camera needs to shoot highly moving objects, the exposure time cannot be too long, so a strong enough light source is needed to provide brightness. The illumination width and length of the selected bar light source are sufficient to cover the camera shooting area.

[0240] The data service component records and counts all defects detected on the assembly line, updates them to the database in real time, performs data analysis, and helps workers analyze the causes of defects, thereby optimizing the process.

[0241] The purpose of the present disclosure is to provide a mask cloth defect detection system, which aims to at least to some extent solve the problems encountered in the prior art in mask cloth defect detection, such as insufficient detection accuracy. The system significantly improves the accuracy and efficiency of defect identification by introducing advanced image acquisition and processing technology, thereby improving the process flow in the relevant technical field.

[0242] The disclosed embodiments provide a complete set of online high-speed defect detection systems, which is a systematic improvement and innovation for specific business scenarios (mask cloth in high-speed movement on a production line). The system lightweights the model as a whole, introduces Bayesian estimation into the model, enhances training samples through AIGC technology, solves the hard sample problem through quantitative learning to enable the model to converge better, improves image acquisition quality through binning, designs a camera automatic exposure algorithm, and designs human-computer interaction logic to optimize the defect decision-making process.

[0243] In an exemplary embodiment, the present disclosure also provides a defect detection system, comprising: an image capture device for collecting the image to be processed of the object to be inspected in the production process; a processor and a memory for storing computer programs; when the processor executes the computer program, the steps of the method described in any of the above embodiments are implemented.

[0244] In an exemplary embodiment, the memory includes a memory for storing a defect detection program (implementing the functions of the above algorithm layer) and a memory for storing a service data program (implementing the functions of the above data service layer).

[0245] In the above description, the intelligent defect detection system and method are listed as preferred examples. Those skilled in the art will understand that modifications, including additions and / or substitutions, may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the purpose of this disclosure is to enable those skilled in the art to practice the teachings herein without excessive experimentation.

[0246] Fig.16 FIG. 1 is a schematic diagram of a defect detection device according to an exemplary embodiment of the present disclosure. Fig.16 As shown, the defect detection device 1500 provided in the embodiment of the present disclosure includes an image acquisition module 1510 and an image processing module 1520 .

[0247] The image acquisition module 1510 can be used to obtain an image to be processed.

[0248] The image processing module 1520 can be used to process the image to be processed through the feature extraction module in the deep learning defect detection model, the feature extraction module includes an intermediate feature layer, the intermediate feature layer includes a discard layer, the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module.

[0249] The image processing module can also be used to process multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of defects in the image to be processed.

[0250] The image processing module may be further configured to obtain a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed according to the multiple category prediction results and the multiple position prediction results of the defect.

[0251] Among them, the image processing module 1520 is also used to: obtain the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect according to multiple first category prediction results and multiple first position prediction results of the first defect; obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect according to multiple second category prediction results and multiple second position prediction results of the second defect.

[0252] In an exemplary embodiment, the image acquisition module 1510 is further used to: obtain the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is used to obtain the image to be processed of the object to be inspected in a production process.

[0253] The defect detection device 1500 may further include: a speed acquisition module, which may be used to obtain the current production speed of the object to be detected and the current processing speed of the deep learning defect detection model on the image to be processed. The image processing module 1520 may also be used to adjust the number of parallel threads or parallel processes used by the deep learning defect detection model to process the image to be processed according to the current production speed and the current processing speed.

[0254] In an exemplary embodiment, the image acquisition module 1510 is also used to: obtain the image to be processed from the image capturing device in the image acquisition component; if the grayscale value and grayscale distribution of the image to be processed do not meet the preset conditions, send a control signal to adjust the exposure parameters of the image capturing device; obtain a new image to be processed from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset conditions is obtained.

[0255] In an exemplary embodiment, the defect detection device 1500 may further include: a sending module, configured to send prompt information for the defect if the category uncertainty and / or position uncertainty of the defect is greater than an uncertainty threshold.

[0256] In an exemplary embodiment, the defect detection device 1500 may also include: a data processing module, which is used to count the pixel number information of each defect detected in the image to be processed; determine the defect level of each defect based on the pixel number information of each defect; and perform corresponding processing based on the defect level of each defect.

[0257] In an exemplary embodiment, the data processing module may also be used to: in the image to be processed, mark defects at different defect levels with frames of different thicknesses.

[0258] In an exemplary embodiment, the defect detection device 1500 may further include: a data processing module, which can be used to save the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed; receive and retrieve corresponding defect information based on one or more of the input image features, detection time and detection serial number of the defect area.

[0259] In an exemplary embodiment, the defect detection device 1500 may also include: a model training module, which can be used to obtain a target training data set, wherein the target training data set includes a target sample image and annotated categories and annotated positions of target sample defects in the target sample image; inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defects; obtaining category detection results, position detection results, category uncertainty and position uncertainty of the target sample defects based on the multiple category prediction results and multiple position prediction results of the target sample defects; constructing a loss function based on the annotated category and category detection results, annotated position and position detection results, and category uncertainty and position uncertainty of the target sample defects to train the deep learning defect detection model.

[0260] In an exemplary embodiment, the model training module can also be used to: obtain the initial sample image and the annotated category and annotated position of the initial sample defect in the initial sample image; perform any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an expanded sample image and the annotated category and annotated position of the extended sample defect in the expanded gain image. Wherein, the target sample image includes the initial sample image and the expanded sample image.

[0261] In an exemplary embodiment, the model training module can also be used to: extract initial sample defects in the initial sample image; synthesize the initial sample defects with the augmented background image to generate the augmented sample image; and / or, learn the initial sample image through a deep learning generation model to generate the augmented sample image.

[0262] In an exemplary embodiment, the model training module can also be used to: obtain candidate sample images and the labeled categories and labeled positions of candidate sample defects in the candidate sample images; obtain the output features of the candidate sample images through the feature extraction module; obtain the feature distances between the candidate sample images based on the output features of each candidate sample image; select candidate sample images whose feature distances are greater than a first distance threshold among candidate sample images with the same labeled categories, and / or candidate sample images whose feature distances are less than a second distance threshold among candidate sample images with different labeled categories, as target sample images in the target training data set.

[0263] In an exemplary embodiment, the model training module can also be used in the training process of the deep learning defect detection model to: extract the weight tensor of each convolutional layer in the deep learning defect detection model; obtain the Euclidean norm of each channel in the weight tensor; sort each convolutional layer according to the Euclidean norm of its channel, and prune the corresponding channel according to a pruning threshold or a pruning ratio.

[0264] Fig.16 For other contents of the embodiment, reference may be made to the other embodiments mentioned above.

[0265] Fig.17 A schematic structural diagram of an electronic device 1600 according to an embodiment of the present disclosure is schematically shown. Fig.17 The electronic device 1600 shown includes a processor 1610, which can call and run a computer program from a memory to implement the method in the embodiment of the present disclosure.

[0266] Alternatively, if Fig.17 As shown, the electronic device 1600 may further include a memory 1620. The processor 1610 may call and run a computer program from the memory 1620 to implement the method in the embodiment of the present disclosure.

[0267] The memory 1620 may be a separate device independent of the processor 1610 , or may be integrated into the processor 1610 .

[0268] Alternatively, if Fig.17As shown, the electronic device 1600 may further include a transceiver 1630, and the processor 1610 may control the transceiver 1630 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0269] Optionally, the processor 1610 , the memory 1620 , and the transceiver 1630 may implement bidirectional communication with each other via the communication bus 1640 .

[0270] It should be understood that the processor of the embodiment of the present disclosure may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form.

[0271] The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in combination with its hardware.

[0272] It can be understood that the memory in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be understood that the above memory is an example and not a limitation.

[0273] The embodiments of the present disclosure also provide a computer-readable storage medium for storing a computer program. The computer program enables a computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, it is not repeated here. The embodiments of the present disclosure also provide a computer program product, including computer program instructions. The computer program instructions enable a computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, it is not repeated here. The embodiments of the present disclosure also provide a computer program. When the computer program is run on a computer, it enables the computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, it is not repeated here.

[0274] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0275] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

Claims

1. A defect detection method, characterized in that: include: Obtaining an image to be processed; Processing the image to be processed by a feature extraction module in a deep learning defect detection model, wherein the feature extraction module includes an intermediate feature layer, and the intermediate feature layer includes a discarding layer, and the discarding layer is used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; Processing the multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; According to the multiple category prediction results and the multiple position prediction results of the defect, the category detection result, the position detection result, the category uncertainty and the position uncertainty of the defect in the image to be processed are obtained.

2. The method according to claim 1, characterized in that The intermediate feature layer includes a first cross-stage convolution block, and the dropout layer includes a first dropout layer; The first cross-stage convolution block includes a first cross-stage input layer, a first convolution layer, a first intermediate layer, a second convolution layer, a first fusion layer, a third convolution layer and the first discard layer; The first convolution layer is used to perform convolution processing on the first cross-stage input feature map input by the first cross-stage input layer, and input the first convolution feature map to the first intermediate layer; The first intermediate layer is used to process the first convolution feature map and input the first feature map to be fused into the first fusion layer; The second convolution layer is used to perform convolution processing on the first cross-stage input feature map, and input the second feature map to be fused into the first fusion layer; The first fusion layer is used to fuse the first feature map to be fused and the second feature map to be fused, and input the first fused feature map to the third convolution layer; The third convolutional layer is used to perform convolution processing on the first fusion feature map, and input the second convolutional feature map to the first discard layer; The first discard layer is used to randomly discard features in the second convolutional feature map to obtain multiple first discarded feature maps. The multiple features obtained after the image to be processed is processed by the feature extraction module and subjected to random discarding processing include multiple first discarded feature maps.

3. The method according to claim 2, characterized in that The intermediate feature layer further includes a second cross-stage convolution block and a third cross-stage convolution block, and the dropout layer further includes a second dropout layer and a third dropout layer; The input end of the second cross-stage convolution block is connected to the output end of the first cross-stage convolution block to respectively receive a plurality of the first discarded feature maps, process the plurality of the first discarded feature maps, and input the plurality of second discarded feature maps to the third cross-stage convolution block through the second discard layer; The input end of the third cross-stage convolution block is connected to the output end of the second cross-stage convolution block to respectively receive multiple second discarded feature maps, process the multiple second discarded feature maps, and output multiple third discarded feature maps through the third discard layer.

4. The method according to claim 3, characterized in that The deep learning defect detection model also includes a feature pyramid module, and the feature pyramid module includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block; The first spatial pyramid pooling block is connected to the first cross-stage convolution block and is used to receive a plurality of the first discarded feature maps; The second spatial pyramid pooling block is connected to the second cross-stage convolution block, and is used to receive a plurality of the second discarded feature maps, and process the plurality of the second discarded feature maps respectively to obtain a plurality of second spatial pooling feature maps; The third spatial pyramid pooling block is connected to the third cross-stage convolution block, and is used to receive the plurality of third discarded feature maps, and process the plurality of third discarded feature maps respectively to obtain a plurality of third spatial pooling feature maps; The first spatial pyramid pooling block is also connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and obtain a plurality of first spatial pooling feature maps according to a plurality of the first discarded feature maps and a plurality of the second spatial pooling feature maps; The second spatial pyramid pooling block is also connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and obtain a plurality of second spatial pooling feature maps according to a plurality of the second discarded feature maps and a plurality of the third spatial pooling feature maps.

5. The method according to claim 4, characterized in that The detection head includes a first depth-separable convolution block, a second depth-separable convolution block and a third depth-separable convolution block; the defect includes a first defect, a second defect and a third defect; The first depthwise separable convolution block is connected to the first spatial pyramid pooling block, and is used to receive a plurality of the first spatial pooling feature maps, and process the plurality of the first spatial pooling feature maps respectively to obtain a plurality of first category prediction results and a plurality of first position prediction results of the first defect; The second depthwise separable convolution block is connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and process the plurality of the second spatial pooling feature maps respectively to obtain a plurality of second category prediction results and a plurality of second position prediction results of the second defect; The third depthwise separable convolution block is connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and process the plurality of the third spatial pooling feature maps respectively to obtain a plurality of third category prediction results and a plurality of third position prediction results of the third defect; Wherein, obtaining the category detection result, position detection result, category uncertainty and position uncertainty of the defect in the image to be processed according to the multiple category prediction results and the multiple position prediction results of the defect includes: Obtaining a first category detection result, a first position detection result, a first category uncertainty, and a first position uncertainty of the first defect according to a plurality of first category prediction results and a plurality of first position prediction results of the first defect; Obtaining a second category detection result, a second position detection result, a second category uncertainty, and a second position uncertainty of the second defect according to the plurality of second category prediction results and the plurality of second position prediction results of the second defect; According to the multiple third category prediction results and the multiple third position prediction results of the third defect, a third category detection result, a third position detection result, a third category uncertainty and a third position uncertainty of the third defect are obtained.

6. The method according to claim 2, characterized in that The first intermediate layer includes a first intermediate input layer, a first depth-separable convolutional layer and a first accumulation layer; The first intermediate input layer is used to receive the first convolution feature map output by the first convolution layer, and input the first convolution feature map into the first depthwise separable convolution layer and the first accumulation layer respectively; The first depth-separable convolution layer is used to perform depth-separable convolution processing on the first convolution feature map, and input the first depth-separable convolution feature map into the first accumulation layer; The first accumulation layer performs accumulation processing on the first convolutional feature map and the first depth-separated convolutional feature map to obtain the first feature map to be fused.

7. The method according to claim 6, characterized in that The first depthwise separable convolutional layer comprises a first depthwise separation input layer, a first depthwise convolutional layer, a first pointwise convolutional layer and a first activation function layer; The first depth separation input layer is used to receive the first convolution feature map output by the first intermediate input layer, and input the first convolution feature map to the first depth convolution layer and the first activation function layer respectively; The first depth convolution layer is used to perform depth convolution processing on the first convolution feature map, and input the first depth convolution feature map to the first point convolution layer; The first point convolution layer is used to perform point convolution processing on the first depth convolution feature map, and input the first point convolution feature map to the first activation function layer; The first activation function layer is used to process the first point convolution feature map and the first convolution feature map to obtain the first depth separation convolution feature map.

8. The method according to claim 1, characterized in that The feature extraction module also includes an attention block connected to the intermediate feature layer; The attention block is used to process the image to be processed, obtain a downsampled feature map of the image to be processed, and input the downsampled feature map into the intermediate feature layer.

9. The method according to claim 1, characterized in that The deep learning defect detection model also includes a background removal module; The background removal module is used to remove background information in the image to be processed, and input the image to be processed after the background information is removed into the feature extraction module.

10. The method according to claim 1, characterized in that Obtain the image to be processed, including: Obtaining the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is used to obtain the image to be processed of the object to be detected in the production process; Wherein, the method further comprises: Obtaining a current production speed of the object to be inspected and a current processing speed of the deep learning defect detection model on the image to be processed; The number of parallel threads or parallel processes used by the deep learning defect detection model to process the image to be processed is adjusted according to the current production speed and the current processing speed.

11. The method according to claim 1, characterized in that Obtain the image to be processed, including: Obtain the image to be processed from an image capturing device in the image acquisition component; If the grayscale value and grayscale distribution of the image to be processed do not meet the preset conditions, sending a control signal to adjust the exposure parameters of the image capture device; A new image to be processed is obtained from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset condition is obtained.

12. The method according to claim 1, characterized in that Also includes: If the category uncertainty and / or position uncertainty of the defect is greater than an uncertainty threshold, prompt information for the defect is sent.

13. The method according to claim 1, characterized in that Also includes: Counting the number of pixels of each defect detected in the image to be processed; Determine a defect level of each defect according to pixel quantity information of each defect; According to the defect level of each defect, corresponding processing is performed.

14. The method according to claim 13, characterized in that According to the defect level of each defect, corresponding processing is performed, including: In the image to be processed, defects at different defect levels are marked with frames with different thicknesses.

15. The method according to claim 1, wherein: Also includes: Saving the image to be processed and the category detection result, position detection result, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection sequence number of the defect detected in the image to be processed; Receive and retrieve corresponding defect information based on one or more of the input image features, detection time and detection sequence number of the defect area.

16. The method according to claim 1, wherein: Also includes: Obtaining a target training data set, wherein the target training data set includes a target sample image and annotated categories and annotated positions of target sample defects in the target sample image; Inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defects; Obtaining a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the target sample defect according to a plurality of category prediction results and a plurality of position prediction results of the target sample defect; A loss function is constructed according to the labeled category and category detection result, the labeled position and position detection result of the target sample defect, as well as the category uncertainty and position uncertainty to train the deep learning defect detection model.

17. The method according to claim 16, characterized in that The loss function includes a first loss function and a second loss function; When the difference between the labeled category and the category detection result of the target sample defect is greater, and the difference between the labeled position and the position detection result is greater, the value of the first loss function is greater; When the category uncertainty and the position uncertainty of the target sample defect are greater, the value of the second loss function is greater.

18. The method according to claim 17, characterized in that The first loss function L1 is as follows: MSE is the bounding box regression calculation, y i is the marked position of the defect of the i-th target sample, is the predicted position detection result of the i-th target sample defect, N is the number of target sample defects, N is a positive integer greater than or equal to 1, i is a positive integer greater than or equal to 1 and less than or equal to N; CE represents multivariate cross entropy calculation, Represents the category confidence of category c in the predicted category detection result; s c represents the category confidence of category c in the labeled category, C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C; BCE represents binary cross entropy calculation, o pre represents the target confidence for the target sample defect prediction, and o represents the confidence of the target marked in the target sample defect.

19. The method according to claim 17, characterized in that The second loss function L2 is as follows: μ i is the predicted position detection result of the i-th target sample defect, y i is the marked position of the defect of the i-th target sample; is the predicted position uncertainty of the i-th target sample defect; N is the number of defects in the target sample, N is a positive integer greater than or equal to 1, i is a positive integer greater than or equal to 1 and less than or equal to N; μ c Represents the category detection result of the predicted category c; s c represents the category confidence of category c in the labeled category, represents the category uncertainty of the predicted category c, C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C.

20. The method of claim 16, wherein: Obtain the target training data set, including: Acquire an initial sample image and a marked category and a marked position of an initial sample defect in the initial sample image; Performing any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an expanded sample image and a marked category and a marked position of an expanded sample defect in the expanded gain image; Wherein, the target sample image includes the initial sample image and the amplified sample image.

21. The method of claim 20, wherein: Performing defect synthesis processing on the initial sample image to generate an augmented sample image includes: Extracting initial sample defects from the initial sample image; synthesizing the initial sample defects with the augmented background image to generate the augmented sample image; and / or, The initial sample image is learned by a deep learning generation model to generate the amplified sample image.

22. The method of claim 16, wherein: Obtain the target training data set, including: Obtaining a candidate sample image and a labeled category and a labeled position of a candidate sample defect in the candidate sample image; Obtaining output features of the candidate sample image through the feature extraction module; Based on the output features of each candidate sample image, obtaining the feature distance between each candidate sample image; Select candidate sample images whose feature distance is greater than a first distance threshold among candidate sample images with the same annotated category, and / or select candidate sample images whose feature distance is less than a second distance threshold among candidate sample images with different annotated categories as target sample images in the target training data set.

23. The method of claim 16, wherein: During the training process of the deep learning defect detection model: Extracting a weight tensor for each convolutional layer in the deep learning defect detection model; Obtaining the Euclidean norm of each channel in the weight tensor; Each convolutional layer is sorted according to the Euclidean norm of its channel, and the corresponding channel is pruned according to the pruning threshold or pruning ratio.

24. The method of claim 16, wherein: The target sample images include images of masks with different materials, textures, colors, and print patterns.

25. A defect detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; An image processing module, used for processing the image to be processed by a feature extraction module in a deep learning defect detection model, wherein the feature extraction module includes an intermediate feature layer, wherein the intermediate feature layer includes a discarding layer, and the discarding layer is used for randomly discarding features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; The image processing module is also used to process the multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; The image processing module is also used to obtain a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed according to the multiple category prediction results and the multiple position prediction results of the defect.

26. A defect detection system, characterized in that: include: An image acquisition component, used for acquiring the image to be processed of the object to be detected in the production process; A defect detection device, connected to the image acquisition component, for obtaining the image to be processed from the image acquisition component; wherein the defect detection device comprises: An image processing module is used to process the image to be processed through a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, the intermediate feature layer includes a discard layer, and the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding after being processed by the feature extraction module; the multiple features after random discarding are processed by the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; according to the multiple category prediction results and multiple position prediction results of the defects, the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed are obtained.

27. The system of claim 26, wherein: The image acquisition component comprises: An image capturing device, the image capturing device comprising an area array camera; The area array camera is used to collect the original image to be processed, and perform pixel merging processing on the original image to be processed to generate the image to be processed.

28. The system of claim 27, wherein: The image acquisition component also includes: The strip light source is located at one side of the image capturing device and illuminates the object to be detected moving on the production line at a predetermined angle.

29. The system of claim 28, wherein: The predetermined angle is 45 degrees.

30. The system of claim 26, wherein: The system includes an industrial computer component, the industrial computer component includes an industrial computer, and the defect detection device is deployed on the industrial computer.

31. The system of claim 30, wherein: A data service component is also deployed on the industrial computer; The data service component is used to record and count the defect information detected during the production process.

32. The system of claim 26, wherein: Also includes: The alarm component is connected to the defect detection device and is used to receive a trigger signal sent by the defect detection device when the defect detection device detects a defect, and send alarm information according to the trigger signal.

33. The system of claim 26, wherein: Also includes: A display is connected to the defect detection device and is used to display the defect information detected by the defect detection device.

34. A defect detection system, characterized in that: include: An image capturing device, used for collecting the image to be processed of the object to be inspected in the production process; a processor and a memory for storing a computer program; When the processor executes the computer program, the steps in the method according to any one of claims 1 to 24 are implemented.

35. The system of claim 34, wherein: The memory includes a memory for storing a defect detection program and a memory for storing a service data program.

36. An electronic device, characterized in that: include: one or more processors; The memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the method according to any one of claims 1 to 24.

37. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 24.

38. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 24 is implemented.

Images