Defect detection method and related device

Abstract

The present disclosure belongs to the technical fields of computers and defect detection. Provided are a defect detection method and a related device. The method comprises: obtaining an image to be processed; processing said image by means of a feature extraction module in a deep-learning defect detection model, wherein the feature extraction module comprises an intermediate feature layer, the intermediate feature layer comprises a dropout layer, and the dropout layer is configured to randomly drop features which are extracted by means of the intermediate feature layer processing said image, such that a plurality of features subjected to random dropout are obtained after said image is processed by means of the feature extraction module; by means of a detection head in the deep-learning defect detection model, processing the plurality of features subjected to random dropout, so as to obtain a plurality of category prediction results and a plurality of position prediction results of defects in said image; and obtaining category detection results, position detection results, category uncertainties and position uncertainties of the defects in said image according to the plurality of category prediction results and the plurality of position prediction results of the defects. (FIG. 1)

Description

Defect detection methods and related equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application No. 202410709277.5, filed on June 3, 2024, entitled “Defect Detection Method and Related Equipment,” and claims priority to U.S. patent application No. 63 / 508,275, filed on June 15, 2023. The entire contents of the Chinese patent application and the U.S. patent application are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to the field of computers and defect detection technology, and in particular to a defect detection method, device and system, electronic equipment, computer-readable storage medium, and computer program product. Background Art

[0004] In real-life production and life scenarios, products or items may have various defects. These products or items need to be tested for defects before leaving the factory to prevent defective products or items from being circulated in the market.

[0005] Summary of the Invention

[0006] An embodiment of the present disclosure provides a defect detection method, comprising: obtaining an image to be processed; processing the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discarding layer, the discarding layer being used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; processing the plurality of features after random discarding processing by a detection head in the deep learning defect detection model to obtain a plurality of category prediction results and a plurality of position prediction results of defects in the image to be processed; and obtaining a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed based on the plurality of category prediction results and the plurality of position prediction results of the defect.

[0007] An embodiment of the present disclosure provides a defect detection device, comprising: an image acquisition module for obtaining an image to be processed; an image processing module for processing the image to be processed through a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discarding layer, the discarding layer being configured to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; the image processing module is further configured to process the plurality of features after random discarding processing through a detection head in the deep learning defect detection model to obtain a plurality of category prediction results and a plurality of position prediction results of defects in the image to be processed; the image processing module is further configured to obtain a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the defect in the image to be processed based on the plurality of category prediction results and the plurality of position prediction results of the defect.

[0008] An embodiment of the present disclosure provides a defect detection system, comprising: an image acquisition component for acquiring an image to be processed of an object to be inspected in a production process; a defect detection device, connected to the image acquisition component, for obtaining the image to be processed from the image acquisition component; wherein the defect detection device comprises: an image processing module for processing the image to be processed by a feature extraction module in a deep learning defect detection model, the feature extraction module comprising an intermediate feature layer, the intermediate feature layer comprising a discarding layer, the discarding layer being configured to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; the plurality of features after random discarding processing are processed by a detection head in the deep learning defect detection model to obtain a plurality of category prediction results and a plurality of position prediction results of defects in the image to be processed; and a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the defect in the image to be processed are obtained based on the plurality of category prediction results and the plurality of position prediction results of the defect.

[0009] An embodiment of the present disclosure provides a defect detection system, comprising: an image capture device for capturing the image to be processed of an object to be inspected in a production process; a processor and a memory for storing a computer program; when the processor executes the computer program, the steps in the method described in any embodiment of the present disclosure are implemented.

[0010] An embodiment of the present disclosure provides an electronic device, comprising: one or more processors; and a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the method described in any embodiment of the present disclosure.

[0011] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, the computer implements the method described in any embodiment of the present disclosure.

[0012] The embodiments of the present disclosure provide a computer program product, including a computer program. When the computer program is executed by a computer, the method described in the embodiments of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1 is a flow chart of a defect detection method according to an exemplary embodiment of the present disclosure.

[0014] FIG2 is a schematic diagram of a deep learning defect detection model according to an exemplary embodiment of the present disclosure.

[0015] FIG3 is a schematic diagram of a deep learning defect detection model according to another exemplary embodiment of the present disclosure.

[0016] FIG4 is a schematic diagram of the first cross-stage convolution block and the second cross-stage convolution block shown in FIG3 .

[0017] FIG. 5 is a schematic diagram of the first intermediate layer and the second intermediate layer shown in FIG. 4 .

[0018] FIG6 is a schematic diagram of the first depth-wise separable convolutional layer and the second depth-wise separable convolutional layer shown in FIG5 .

[0019] FIG7 is a schematic diagram of the third cross-stage convolution block shown in FIG3 .

[0020] FIG8 is a schematic diagram of a deep learning defect detection model according to another exemplary embodiment of the present disclosure.

[0021] FIG9 is a schematic diagram of a deep learning defect detection model according to yet another exemplary embodiment of the present disclosure.

[0022] FIG10 is a schematic diagram of a spatial pyramid pooling block according to an exemplary embodiment of the present disclosure.

[0023] FIG11 is a schematic diagram of an alternative embodiment of a spatial pyramid pooling block in an embodiment of the present disclosure.

[0024] FIG12 is a schematic diagram of the attention block shown in FIG9 .

[0025] FIG. 13 is a schematic diagram of a defect detection system according to an exemplary embodiment of the present disclosure.

[0026] FIG. 14 is a schematic diagram of a defect detection system according to another exemplary embodiment of the present disclosure.

[0027] FIG. 15 is a schematic diagram of a defect detection system according to yet another exemplary embodiment of the present disclosure.

[0028] FIG16 is a schematic structural diagram of a defect detection device according to an exemplary embodiment of the present disclosure.

[0029] FIG17 schematically shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions, and advantages of the present disclosure more apparent, exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the accompanying drawings, like reference numerals represent like elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of the present disclosure.

[0031] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0032] In the present disclosure, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise expressly defined. Terms such as "installed", "connected", "connected", and "fixed" should be understood in a broad sense. For example, "connected" can mean a fixed connection, a detachable connection, or an integral connection; "connected" can mean a direct connection or an indirect connection through an intermediate medium. For those skilled in the art, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances.

[0033] In the description of the present disclosure, it is to be understood that the terms "up", "down", "left", "right", "front", "back", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific direction, and therefore, cannot be understood as a limitation on the present disclosure.

[0034] Throughout this specification, terms such as "one embodiment," "some embodiments," and "specific embodiments" mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0035] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0036] The defect detection method provided in the embodiment of FIG1 can be executed by any electronic device, such as a terminal and / or a server, and the present disclosure does not limit this. As shown in FIG1 , the method provided in the embodiment of the present disclosure includes the following steps.

[0037] In S110 , an image to be processed is obtained.

[0038] The image to be processed in the embodiments of the present disclosure can be an image of any object to be inspected for defects or flaws. The object to be inspected can be, for example, a sanitary and personal care product, such as medical and sanitary products such as surgical gowns, protective clothing, disinfection wraps, masks, diapers, household rags, wipes, wet face towels, magic towels, soft towel rolls, beauty cosmetics, etc.; it can also be a home decoration product such as wall coverings, tablecloths, bed sheets, bedspreads, etc.; it can also be a clothing product such as lining, adhesive interlining, flakes, shaped cotton, various synthetic leather base fabrics, etc.; it can also be an industrial product such as filter materials, insulation materials, cement packaging bags, geotextiles, covering fabrics, etc.; it can also be an agricultural product such as crop protection fabrics, seedling cloths, irrigation fabrics, thermal insulation curtains, etc.; it can also be other products such as space cotton, thermal insulation and sound insulation materials, oil-absorbing felt, cigarette filters, tea bags, etc.

[0039] In the following embodiments, the object to be detected is a mask (including masks that have been sewn and masks in the process of cutting), and the corresponding image to be processed is a mask image. In other embodiments, the object to be detected is a mask cloth in high-speed motion on a production line (hereinafter referred to as a production line), and the corresponding image to be processed is a mask cloth image. For example, the mask cloth can be a non-woven fabric, but the present disclosure is not limited to this, and other suitable materials can also be used as the mask cloth. At this point, the present disclosure relates to the field of non-woven material detection technology, mainly for the subdivision field of mask cloth defect detection. This method can also be referred to as a non-woven mask cloth defect detection method or a mask cloth defect detection method.

[0040] For example, the process of making masks from non-woven fabric primarily involves material preparation, cutting, sewing, and finishing. After preparing the raw materials, cutting begins. The non-woven fabric is laid flat and cut with a cutting machine to create the main body of the mask. Next, sewing occurs, sewing together the non-woven fabric, nose bridge, ear straps, and other components to form a complete mask. Finally, finishing and quality inspection are required to ensure that each mask meets quality standards. Relevant technologies primarily focus on defect detection of finished masks, not defect detection of mask fabric on the mask production line, making it impossible to achieve quality management and control during the production phase.

[0041] Due to the huge threat posed to public safety by the global epidemics (such as covid-19) that have broken out in recent years, the demand for masks has surged. The quality of masks plays an important role in the prevention and control of epidemics. Therefore, mask manufacturers need to improve the production efficiency of masks while ensuring quality. In the related art, manual inspection is mainly used. The method of achieving mask quality control based on manual inspection lacks efficiency, and there are problems of inconsistent standards and low precision. The method provided in the embodiment of the present disclosure uses machine vision technology to realize automated defect detection of mask cloth on the production line, which can greatly improve production efficiency while ensuring the quality of masks, and can effectively reduce labor costs.

[0042] In S120, the image to be processed is processed by a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, the intermediate feature layer includes a discard layer, and the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module.

[0043] The deep learning defect detection model in the embodiments of the present disclosure is a model based on deep learning that can process images, detect whether there are defects in the image, and predict the defect information of the defect when the defect is detected. The defect information includes, for example, any one or more of the size, location, category, etc. of the defect. For example, the deep learning defect detection model can be based on a target detection model, such as any one or more of the YOLO (You Only Look Once) series (such as YOLOv3, YOLOv4, Yolov5, etc.), Faster R-CNN, SSD, CNN (convolutional neural network), recurrent neural network (RNN) and transformer (Transformer). In the following embodiments, the YOLO model is used as an example, but the present disclosure is not limited to this.

[0044] The deep learning defect detection model in the embodiment of the present disclosure is a real-time target detection model that divides the input image to be processed into grids, each grid is responsible for detecting a target (i.e., defect) and predicting the target's bounding box (also known as an anchor box) and category. The deep learning defect detection model includes a feature extraction module, which is the backbone network (Backbone) in the deep learning defect detection model and is mainly responsible for extracting the features of the input image to be processed. The backbone network can include convolution layers, pooling layers, and other possible network layers, which can gradually extract deep features in the image to be processed and convert the image to be processed into multi-layer feature maps. These feature maps are then used for target detection tasks such as bounding box prediction and category classification. For example, the backbone network can include a Conv module (convolution module), a C3 module (composed of three standard convolution layers, each convolution layer has a specific convolution kernel size and step size), and an SPPF module (spatial pyramid pooling layer, used to aggregate features at multiple scales). These modules extract and fuse feature information at different levels through operations such as convolution and pooling.

[0045] Backbone's intermediate feature layers refer to the hidden layers (excluding the feature pyramid and detection head) located after the input layer (or between the input and output layers if there is an output layer) in the Backbone network, which are used to generate feature maps of the image to be processed. These intermediate feature layers capture information at different levels and scales of the image to be processed. The information in the intermediate feature layers can not only be used to generate the final detection results, but can sometimes also be used for other tasks, such as feature fusion or further analysis. For example, Backbone's intermediate feature layers can be used to fuse with other network components (such as FPN, i.e., feature pyramid network / module) to improve detection performance.

[0046] In the disclosed embodiments, a dropout layer is added to the intermediate feature layer of the feature extraction module of a deep learning defect detection model. Dropout simulates ensemble learning between different sub-networks by randomly "turning off" or "erasing" a portion of neurons (or features) in the hidden layer, preventing the deleted neurons from transmitting signals. This effectively prevents model overfitting and improves its generalization ability on unseen data (i.e., data not included in the training dataset). Dropout forces the model to learn more robust feature representations because it cannot rely on any specific combination of neurons. Dropout can be viewed as a Bayesian approximation method, simulating the sparsity of model parameters by randomly "turning off" a portion of neurons. This is similar to the posterior distribution of parameters in Bayesian inference. Dropout does not directly simulate the Bayesian posterior distribution, but rather provides a regularization effect through its randomness, helping to reduce overfitting. The disclosed embodiments add dropout to the intermediate feature layer of Backbone to leverage its ability to prevent overfitting and improve the generalization performance of the model. It also reduces the complexity and number of parameters in the neural network. If dropout is only enabled during the training phase and disabled during the inference phase, it is used to prevent model overfitting. In the disclosed embodiment, dropout is enabled during both the training and inference phases to simulate Bayesian uncertainty.

[0047] In S130, the detection head in the deep learning defect detection model processes the multiple features after random discarding to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed.

[0048] The disclosed embodiment, under the effect of dropout, randomly shuts down neurons in the hidden layer (setting some hidden layer node values ​​to 0), thereby randomly characterizing the generation process of the hidden layer network structure, such as the number of layers in the neural network, the connection method of neurons in each layer, etc., and can obtain multiple different network structures of the intermediate feature layer, thereby making multiple predictions for the same image to be processed (the same target sample image during training) through the multiple different network structures of the intermediate feature layer, thereby obtaining multiple category prediction results and multiple location prediction results for each defect in the image to be processed. Uncertainty estimation is to calculate the distribution index of the results of multiple predictions.

[0049] In the disclosed embodiment, the category prediction result refers to the category result to which the defect belongs, as predicted by the detection head of the deep learning defect detection model (such as the depth-separable convolution block therein). The position prediction result refers to the position of the defect in the image (including the image to be processed and / or the target sample image) as predicted by the detection head of the deep learning defect detection model (such as the depth-separable convolution block therein), which can be an absolute position or a relative position. The category prediction result may include the predicted probability that the defect belongs to each category, and the position prediction result may include the predicted information such as the center coordinates, width and height of the bounding box of the defect in the image.

[0050] In S140 , a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the defect in the image to be processed are obtained according to the multiple category prediction results and the multiple position prediction results of the defect.

[0051] In the object detection task, the detection head of the deep learning defect detection model in the disclosed embodiments is primarily responsible for predicting the category, location, and confidence of the defect bounding box in the image to be processed. It can also predict the category uncertainty of the predicted defect category and the location uncertainty of the location. In some embodiments, the detection head can utilize multiple convolutional layers to further extract and integrate features, and then output multiple category prediction results and multiple location prediction results for the defect. Based on the multiple category prediction results and multiple location prediction results for the defect, the detection head can obtain the category detection result, location detection result, category uncertainty, and location uncertainty of the defect in the image to be processed.

[0052] In the embodiment of the present disclosure, the category detection result and the position detection result refer to the category result of the defect and the position of the defect in the image output by the detection head of the deep learning defect detection model, respectively.

[0053] In the embodiments of the present disclosure, category uncertainty refers to an indicator used to evaluate the degree of certainty of whether the predicted category result of a defect in an image (including the image to be processed and / or the target sample image) matches the true category of the defect when the deep learning defect detection model is used to predict the category prediction result of the defect. The higher the category uncertainty, the more uncertain the category prediction result of the deep learning defect detection model for the defect. The lower the category uncertainty, the more certain the category prediction result of the deep learning defect detection model for the defect.

[0054] In the embodiment of the present disclosure, position uncertainty refers to an indicator used to evaluate the degree of certainty of whether the predicted position result of a defect in an image (including the image to be processed and / or the target sample image) matches the actual position of the defect when the deep learning defect detection model is used to predict the position prediction result of the defect. The higher the position uncertainty, the more uncertain the position prediction result of the defect by the deep learning defect detection model is, and the lower the position uncertainty, the more certain the position prediction result of the defect by the deep learning defect detection model is.

[0055] In the disclosed embodiments, in addition to predicting the position and category of a bounding box, a deep learning defect detection model can also predict the confidence of the output bounding box. The predicted confidence is a value between 0 and 1 that represents the model's confidence in the detected defect. The confidence can represent the probability (also known as target confidence) of whether a defect (also known as a flaw) is present within the bounding box, i.e., whether the bounding box is just background or contains a defect. Optionally, it can also represent the degree of match between the bounding box predicted by the model and the ground-truth bounding box of the defect when the bounding box contains a defect; or it can represent the degree of match between the category predicted by the model and the ground-truth category of the defect when the bounding box contains a defect. For example, the confidence can be calculated by calculating the Intersection over Union (IoU) between the predicted bounding box and the actual defect. If the IoU is greater than a set threshold, the defect is considered to be detected, and the confidence is the IoU score of the bounding box; if the IoU is less than the set threshold, the defect is considered not to be detected, and the confidence is 0.

[0056] Uncertainty estimation is an effective method to measure the credibility of detection results. The uncertainty in target detection tasks can be divided into data uncertainty and model uncertainty. Data uncertainty mainly comes from the inherent noise in real data (such as the input image to be processed or the target sample image), and model uncertainty comes from the uncertainty of model parameters. The category uncertainty in the embodiment of the present disclosure can include both data uncertainty and model uncertainty, and the position uncertainty can also include both data uncertainty and model uncertainty. Among them, the proportion of data uncertainty is relatively small, which is related to the noise and the annotation quality of the sample.

[0057] When prediction uncertainty is insufficient, the model may overconfidently produce incorrect category and location detection results. For example, during quality inspection on a production line, the model may mistakenly classify a defective / blemished product as non-defective, or vice versa. This misclassification can lead to inefficient production lines and even affect overall product quality. For production processes that rely on model predictions for quality control, if the model cannot provide reliable and accurate predictions, these predictions may not be trusted by end users.

[0058] In an exemplary embodiment, the mean of multiple defect category prediction results can be calculated as the defect category detection result. The category prediction result indicates the probability of a detected defect belonging to a certain type of defect. The category prediction result can also be called the category confidence, and therefore the category detection result can also be called the category confidence mean. The mean of multiple defect location prediction results can be used as the defect location detection result. When coordinates are used to represent the location, the location detection result can also be called the coordinate mean.

[0059] In the disclosed embodiments, category uncertainty is the uncertainty of category confidence and is a probability distribution indicator of category confidence. That is, the probability distribution of category confidence represents category uncertainty, and therefore it can also be referred to as category confidence uncertainty. For example, it can be calculated using the variance of category confidence, in which case it can be referred to as category confidence variance. Position uncertainty is a probability distribution indicator of the bounding box. The probability distribution of the bounding box represents position uncertainty. For example, the variance of the position prediction result can be calculated to obtain position uncertainty. When coordinates are used to represent position and the variance of the coordinates is calculated as position uncertainty, position uncertainty can also be referred to as coordinate variance.

[0060] For example, the model output includes [coordinate mean, coordinate variance (position uncertainty), category confidence mean, category confidence variance (category uncertainty), target confidence]. The target confidence indicates the probability of a defect, while the category confidence mean indicates the probability of a defect type.

[0061] In the embodiments of the present disclosure, a confidence threshold can be set. If the target confidence level output by the model is less than the confidence threshold, the defect with the lower target confidence level is detected, for example, prompting manual judgment. The confidence threshold is a real number greater than 0 and less than 1, for example, it can be set to 0.3, but the present disclosure is not limited to this.

[0062] Assuming that the position or coordinates of each bounding box are represented by (x, y, w, h), where x, y, w, h represent the center horizontal and vertical coordinates, width, and height of the bounding box respectively, the coordinate mean is expressed as: The coordinate variance is expressed as: For example, the category confidence mean and category confidence variance are: Then the category confidence p(c|Object) can be expressed as (c represents the current category, k represents traversing all categories):

[0063] However, the present disclosure is not limited to this. For example, the category prediction result that appears most frequently among the multiple category prediction results of the defect may be selected as the category detection result of the defect, and the position prediction result that appears most frequently among the multiple position prediction results of the defect may be selected as the position detection result of the defect. Because the category detection result and position detection result of the defect in the embodiment of the present disclosure comprehensively consider the multiple category prediction results and multiple position prediction results of the defect, the detection results (including the category detection result and the position detection result) output by it are more accurate.

[0064] It should be noted that in the disclosed embodiments, the category prediction result output by the deep learning defect model is the mean category confidence. The mean category confidence output by the model is subjected to a system post-processing process using the above formula (1). Post-processing is actually adding a softmax process after the model output to map the category confidence to a real number between [0, 1]. Optionally, the post-processing process can also protect the bounding box NMS (non-maximum suppression) and other methods.

[0065] In an exemplary embodiment, the variance of multiple category prediction results of a defect may be calculated as the category uncertainty of the defect, and the variance of multiple position prediction results of the defect may be calculated as the position uncertainty of the defect.

[0066] The present disclosure is not limited to directly using variance as uncertainty. For example, uncertainty can also be calculated using the following formula: Uncertainty = exp(-λ·variance) (2)

[0067] Where λ represents the adjustment term, which is a real number greater than 0 and less than 1. Variance represents the variance of multiple category prediction results or the variance of multiple location prediction results. Uncertainty represents category uncertainty or location uncertainty.

[0068] In some embodiments, an uncertainty threshold may be set to filter prediction results with higher uncertainty.

[0069] Sensitivity to environmental factors, such as lighting, shadows, and background clutter, can affect a model's detection performance. The disclosed embodiments optimize the model to improve its stability and accuracy in real-world environments. For example, Bayesian estimation and prediction are introduced to enhance the model's generalization and robustness to natural environments. Bayesian estimation simulates randomness by adding a dropout layer, randomly erasing a portion of the feature output. The model's detection head output is also modified, interpreting the output as a probability distribution rather than simply a regression problem. This interpretation of the output as a probability distribution better captures the uncertainty of defect location and classification. This allows the model to not only provide deterministic estimates of the predictions but also provide a measure of the uncertainty surrounding these estimates. This is particularly important for dealing with complex natural environments, where the location, size, and appearance of objects can vary due to a variety of factors. By combining Bayesian estimation with a dropout layer and interpreting the output as a probability distribution, a more robust and generalizable deep learning defect detection model can be constructed. Such a model can better adapt to changes in the natural environment and improve detection accuracy and reliability.

[0070] The defect detection method provided by the embodiment of the present disclosure simplifies the approximation of the posterior probability distribution by adding a new discard layer in the intermediate feature layer in the feature extraction module of the deep learning defect detection model, thereby not only being able to quickly and effectively predict the category and position of the defects in the processed image, but also being able to evaluate the category uncertainty and position uncertainty of the prediction results. The embodiment of the present disclosure has developed a new deep learning defect detection model that is more accurate, efficient and has good generalization ability. It can improve the accuracy of defect recognition without sacrificing detection speed, and can adapt to the needs of various production environments and types of objects to be detected. It has broad application prospects. For example, it can be applied to a mask fabric defect detection system to identify defects in different types of masks.

[0071] As shown in Figure 2, the deep learning defect detection module 1 provided in an embodiment of the present disclosure includes a feature extraction module 11 and a detection head 12. The feature extraction module 11 includes an input block 111 and an intermediate feature layer 112. The input block 111 is used to receive an image to be processed and send it to the intermediate feature layer 112. The intermediate feature layer 112 includes a dropout layer 1121. The input block 111 can also be used to preprocess the received input image to be processed, such as resizing and normalizing it, to meet the input requirements of the model.

[0072] The image to be processed is processed by the feature extraction module 11 in the deep learning defect detection model 1. The discarding layer 1121 in the intermediate feature layer 112 of the feature extraction module 11 is used to randomly discard the features extracted after the intermediate feature layer 112 processes the image to be processed, so that the image to be processed obtains multiple features (the specific number of samples can be set according to actual needs) after being processed by the feature extraction module 11. Then, the multiple features after random discarding are processed separately by the detection head 12 in the deep learning defect detection model 1 to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed. The detection head 12 also obtains the category detection result, position detection result, category uncertainty and position uncertainty of the defects in the image to be processed based on the multiple category prediction results and multiple position prediction results of the defects.

[0073] In an exemplary embodiment, the intermediate feature layer includes a first cross-stage convolution block, and the dropout layer includes a first dropout layer. The first cross-stage convolution block includes a first cross-stage input layer, a first convolution layer, a first intermediate layer, a second convolution layer, a first fusion layer, a third convolution layer, and the first dropout layer. The first convolution layer is configured to convolve a first cross-stage input feature map input by the first cross-stage input layer and input the first convolution feature map to the first intermediate layer. The first intermediate layer is configured to process the first convolution feature map and input a first feature map to be fused to the first fusion layer. The second convolution layer is configured to convolve the first cross-stage input feature map and input a second feature map to be fused to the first fusion layer. The first fusion layer is configured to fuse the first feature map to be fused with the second feature map to be fused and input the first fused feature map to the third convolution layer. The third convolution layer is configured to convolve the first fused feature map and input the second convolution feature map to the first dropout layer. The first discard layer is used to randomly discard features in the second convolutional feature map to obtain multiple first discarded feature maps. After the image to be processed is processed by the feature extraction module, the multiple features obtained after random discarding include multiple first discarded feature maps.

[0074] The cross-stage input layers (including the first to third cross-stage input layers), intermediate input layers (including the first to third intermediate input layers), and depth separation input layers (including the first to third depth separation input layers) in the embodiments of the present disclosure are all input layers of the corresponding modules in the model. Their names are only used to distinguish which module the input layer belongs to. For example, the cross-stage input layer indicates that it is the input layer of the cross-stage convolution block. The input layer is used to preprocess the input feature map or image data, such as adjusting the size, to adapt it to the input requirements of the corresponding module in the model.

[0075] The convolution feature map in the embodiment of the present disclosure (for example, including the first to sixth convolution feature maps) refers to the feature map obtained after the convolution process; the feature map to be fused (for example, including the first to sixth feature maps to be fused) refers to the feature map to be fused, and the fusion process here can be, for example, splicing or cascading, for example, the first feature map to be fused and the second feature map to be fused are cascaded to obtain the first fused feature map. The discarded feature map (for example, including the first to third discarded feature maps) refers to the feature map generated by randomly discarding features through the corresponding dropout layer. The spatial pooling feature map (for example, including the first to third spatial pooling feature maps) refers to the feature map generated after processing by the spatial pyramid pooling block (for example, including the first to third spatial pyramid pooling blocks).

[0076] In an exemplary embodiment, the intermediate feature layer further includes a second cross-stage convolution block and a third cross-stage convolution block, and the dropout layer further includes a second dropout layer and a third dropout layer.

[0077] The input end of the second cross-stage convolution block is connected to the output end of the first cross-stage convolution block to receive the plurality of first discarded feature maps respectively, process the plurality of first discarded feature maps, and input the plurality of second discarded feature maps to the third cross-stage convolution block through the second discard layer. The input end of the third cross-stage convolution block is connected to the output end of the second cross-stage convolution block to receive the plurality of second discarded feature maps respectively, process the plurality of second discarded feature maps, and output the plurality of third discarded feature maps through the third discard layer.

[0078] The second cross-stage convolution block includes a second cross-stage input layer, a fourth convolutional layer, a second intermediate layer, a fifth convolutional layer, a second fusion layer, a sixth convolutional layer, and a second dropout layer. The second cross-stage input layer is connected to the first dropout layer and is configured to use the plurality of first dropped feature maps as second cross-stage input feature maps. The fourth convolutional layer is configured to convolve the second cross-stage input feature map input from the second cross-stage input layer and input a third convolutional feature map to the second intermediate layer. The second intermediate layer is configured to process the third convolutional feature map and input a third feature map to be fused to the second fusion layer. The fifth convolutional layer is configured to convolve the second cross-stage input feature map and input a fourth feature map to be fused to the second fusion layer. The second fusion layer is configured to fuse the third feature map to be fused with the fourth feature map to be fused and input the second fused feature map to the sixth convolutional layer. The sixth convolutional layer is configured to convolve the second fused feature map and input the fourth convolutional feature map to the second dropout layer. The second discard layer is used to randomly discard the features in the fourth convolutional feature map to obtain multiple second discarded feature maps. The multiple features obtained after the image to be processed is processed by the feature extraction module and subjected to random discard processing also include multiple second discarded feature maps.

[0079] The cross-stage convolution blocks in the embodiments of the present disclosure (for example, including the first to third cross-stage convolution blocks) adopt the idea of ​​cross-stage partial connection, and enhance the diversity and robustness of features by fusing feature maps of different stages. This design helps to improve the performance of the model, especially when dealing with complex target detection tasks. Specifically, the cross-stage convolution block adopts a staged processing approach, dividing the input feature map into multiple stages, each of which contains a series of convolution operations. At the end of each stage, the cross-stage convolution block fuses the feature map of the current stage with the feature map of the previous stage to obtain a richer feature representation. This cross-stage connection method helps the model capture feature information of different scales and levels, and improves the model's ability to perceive the target.

[0080] In some other embodiments, the cross-stage convolutional blocks can also adopt a partial connection strategy, that is, instead of fusing the feature maps of all stages, they selectively fuse some feature maps. This design can reduce the computational complexity of the model while maintaining high performance.

[0081] In an exemplary embodiment, the deep learning defect detection model further includes a feature pyramid module, which includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block. The first spatial pyramid pooling block is connected to the first cross-stage convolution block and is configured to receive a plurality of first discarded feature maps. The second spatial pyramid pooling block is connected to the second cross-stage convolution block and is configured to receive a plurality of second discarded feature maps and process each of the plurality of second discarded feature maps to obtain a plurality of second spatial pooled feature maps. The third spatial pyramid pooling block is connected to the third cross-stage convolution block and is configured to receive a plurality of third discarded feature maps and process each of the plurality of third discarded feature maps to obtain a plurality of third spatial pooled feature maps. The first spatial pyramid pooling block is also connected to the second spatial pyramid pooling block and is configured to receive a plurality of second spatial pooled feature maps and obtain a plurality of first spatial pooled feature maps based on the plurality of first discarded feature maps and the plurality of second spatial pooled feature maps. The second spatial pyramid pooling block is further connected to the third spatial pyramid pooling block, and is configured to receive a plurality of the third spatial pooling feature maps, and obtain a plurality of second spatial pooling feature maps according to the plurality of the second discarded feature maps and the plurality of the third spatial pooling feature maps.

[0082] In the embodiment of the present disclosure, a feature pyramid module can be added between the backbone network (such as the feature extraction module 11) and the detection head 12 to further process and enhance the features extracted by the backbone network. In an exemplary embodiment, the feature pyramid module can include one or more of FPN (Feature Pyramid Networks) and PANet (Path Aggregation Network), which combine feature information at different levels through multi-scale feature fusion to improve the detection performance of the model. In the following embodiments, the feature pyramid module includes a spatial pyramid pooling block for illustration, but the present disclosure is not limited to this.

[0083] The Spatial Pyramid Pooling block (SSP block) uses spatial pyramid pooling technology. The idea of ​​spatial pyramid pooling comes from the use of pyramid-shaped grid cells that cover receptive fields of different sizes. Each grid cell performs the same pooling operation, and ultimately all feature maps of different sizes are concatenated together to form a feature vector of fixed length. Specifically, the spatial pyramid pooling block scales the input feature map and performs pooling operations at each scale. This multi-scale pooling method enables the network to process inputs of arbitrary sizes and generate outputs of fixed sizes. It does this by splicing feature maps of different scales together to obtain a multi-scale feature representation, thereby improving the flexibility and adaptability of the model. Because the spatial pyramid pooling block uses this multi-scale pooling technology, it can flexibly process inputs of different sizes and generate fixed-length feature representations, so it is called the spatial pyramid pooling block.

[0084] In an exemplary embodiment, the detection head includes a first depthwise separable convolution block, a second depthwise separable convolution block, and a third depthwise separable convolution block. The defects include a first defect, a second defect, and a third defect. The first depthwise separable convolution block is connected to the first spatial pyramid pooling block, configured to receive a plurality of first spatial pooling feature maps and process each of the plurality of first spatial pooling feature maps to obtain a plurality of first category prediction results and a plurality of first position prediction results for the first defect. The second depthwise separable convolution block is connected to the second spatial pyramid pooling block, configured to receive a plurality of second spatial pooling feature maps and process each of the plurality of second spatial pooling feature maps to obtain a plurality of second category prediction results and a plurality of second position prediction results for the second defect. The third depthwise separable convolution block is connected to the third spatial pyramid pooling block, configured to receive a plurality of third spatial pooling feature maps and process each of the plurality of third spatial pooling feature maps to obtain a plurality of third category prediction results and a plurality of third position prediction results for the third defect.

[0085] The detection head in the embodiment of the present disclosure may include two or more depth-separable convolution blocks (for example, including first to third depth-separable convolution blocks). The number of depth-separable convolution blocks in the detection head corresponds to the number of cross-stage convolution blocks in the intermediate feature layer. When the model also includes a feature pyramid module, the number of depth-separable convolution blocks also corresponds to the number of spatial pyramid pooling blocks in the feature pyramid module. In the following embodiments, the detection head is taken as an example to include 2 or 3 depth-separable convolution blocks, but the present disclosure is not limited to this. For example, more depth-separable convolution blocks may be included to extract defects of more scales.

[0086] In the disclosed embodiment, each depthwise separable convolution block decomposes a complete convolution operation into two steps, namely Depthwise Convolution and Pointwise Convolution. In the depthwise convolution step, for example, a three-channel color feature map with a size of 64×64 pixels (for example, it can come from the output of the corresponding cross-stage convolution block or the output of the spatial pyramid pooling block) first undergoes the first convolution operation. This convolution is performed entirely in a two-dimensional plane, and the number of filters is the same as the Depth (depth, here refers to the number of channels rather than the network level) of the previous layer. Therefore, a three-channel feature map generates 3 feature maps after the operation. One of the filters contains a kernel of size 3×3. The number of feature maps after Depthwise Convolution is the same as the Depth of the input layer. Then, through Pointwise Convolution, the information of different feature maps at the same spatial position is used to combine these feature maps to generate a new feature map. The size of the convolution kernel of Pointwise Convolution is 1×1×M, where M is the Depth of the previous layer. The convolution operation here performs a weighted combination of the feature maps from the previous step in the depthwise direction to generate a new feature map. After pointwise convolution, four feature maps are output, the same output dimensions as a regular convolution. Given the same input and four feature maps, depthwise separable convolution requires approximately one-third the number of parameters of a regular convolution, reducing the computational overhead by 8-9 times. Therefore, while maintaining the same number of parameters, neural networks using depthwise separable convolution can be made deeper.

[0087] In the disclosed embodiment of the present invention, the structure of the depth-wise separable convolution block in the detection head may refer to the depth-wise separable convolution layer in the intermediate feature layer (for example, the first to third depth-wise separable convolution layers), and the convolution kernel size of the depth-wise separable convolution block may be different from that of the depth-wise separable convolution layer, but the convolution method may be the same. For example, the convolution kernel sizes in the detection head may be 1×1 and 3×3, respectively. The convolution kernel sizes of the depth-wise separable convolution layer in the intermediate feature layer may be 1×1, 3×3, 5×5, and 7×7, respectively. The role of different convolution kernels is to have different feature receptive fields, and depth-wise separable convolution can effectively reduce the amount of convolution parameters and thus enhance processing speed. For example, for normal convolution, such as 128x3x3x256 (c1 wh c2), 128 represents the number of input channels, 3x3 represents the convolution kernel size, 256 represents the number of output channels, and the number of parameters is 294912. Depthwise separable convolution first performs a 3x3 depthwise convolution on each input channel, requiring 128x3x3 = 1152 parameters. It then performs a 1x1 pointwise convolution on each subregion to expand the channel, requiring 128x1x1x256 = 32768 parameters. The total number of parameters is 1152 + 32768 = 33930. This reduces the number of parameters by 88% compared to conventional convolution, with only a slight loss in accuracy. Depthwise separable convolution can completely replace conventional convolution, using the same activation and connection methods.

[0088] In some other embodiments, the depthwise separable convolution block may not be used in the detection head, that is, the detection head may include using normal convolution operations to achieve prediction.

[0089] Wherein, based on multiple category prediction results and multiple position prediction results of the defect, the category detection result, position detection result, category uncertainty and position uncertainty of the defect in the image to be processed are obtained, including: based on multiple first category prediction results and multiple first position prediction results of the first defect, the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect are obtained; based on multiple second category prediction results and multiple second position prediction results of the second defect, the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect are obtained; based on multiple third category prediction results and multiple third position prediction results of the third defect, the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect are obtained.

[0090] In the embodiment of the present disclosure, the first defect, the second defect, the third defect, etc. refer to defects of different scales in the image to be processed, and are not used to limit the number of defects.

[0091] In an exemplary embodiment, the first intermediate layer includes a first intermediate input layer, a first depthwise separable convolution layer, and a first accumulation layer. The first intermediate input layer is used to receive the first convolution feature map output by the first convolution layer, and input the first convolution feature map into the first depthwise separable convolution layer and the first accumulation layer, respectively. The first depthwise separable convolution layer is used to perform depthwise separable convolution processing on the first convolution feature map, and input the first depthwise separated convolution feature map into the first accumulation layer. The first accumulation layer accumulates the first convolution feature map and the first depthwise separated convolution feature map to obtain the first feature map to be fused.

[0092] In an exemplary embodiment, the first depthwise separable convolution layer includes a first depthwise separation input layer, a first depthwise convolution layer, a first pointwise convolution layer, and a first activation function layer. The first depthwise separation input layer is used to receive the first convolution feature map output by the first intermediate input layer, and input the first convolution feature map into the first depthwise convolution layer and the first activation function layer, respectively. The first depthwise convolution layer is used to perform depthwise convolution processing on the first convolution feature map, and input the first depthwise convolution feature map into the first pointwise convolution layer. The first pointwise convolution layer is used to perform pointwise convolution processing on the first depthwise convolution feature map, and input the first pointwise convolution feature map into the first activation function layer. The first activation function layer is used to process the first pointwise convolution feature map and the first convolution feature map to obtain the first depthwise separation convolution feature map.

[0093] In an exemplary embodiment, the feature extraction module further includes an attention block connected to the intermediate feature layer. The attention block is configured to process the image to be processed, obtain a downsampled feature map of the image to be processed, and input the downsampled feature map to the intermediate feature layer.

[0094] In an exemplary embodiment, the deep learning defect detection model further includes a background removal module configured to remove background information from the image to be processed and input the image to be processed after the background information is removed into the feature extraction module.

[0095] In the embodiment of the present disclosure, background removal can be used to remove or separate parts that are not related to the object to be detected from the image to be processed, so as to more accurately focus on the object to be detected itself and its possible defects. These backgrounds may include factors such as factory production lines, conveyor belts, support structures, ambient lighting, etc. During the image processing and analysis process, they may introduce noise, interfere with edges, or affect the extraction of features such as color and texture. By removing the background, you can focus more on the object to be detected itself, so as to more accurately identify and locate defects. Background removal can involve background modeling, threshold segmentation, morphological operations, etc. to separate the foreground (i.e., the object to be detected) and the background from the original image to be processed. It should be noted that background removal does not mean completely eliminating all traces of the background in the image. On the contrary, its purpose is to reduce the interference of the background on the defect detection process while retaining sufficient contextual information so that the state and quality of the object to be detected can be accurately judged.

[0096] In the disclosed embodiment, the intermediate feature layer may include three or more cross-stage convolution blocks, and the number of depthwise separable convolution blocks in the detection head is the same as the number of cross-stage convolution blocks in the intermediate feature layer. When the model includes a feature pyramid module, the number of spatial pyramid pooling blocks included in the feature pyramid module is the same as the number of depthwise separable convolution blocks in the detection head.

[0097] The deep learning defect detection model provided by the embodiments of the present disclosure is illustrated below with reference to Figures 3 to 11 , but the present disclosure is not limited thereto.

[0098] FIG3 is a schematic diagram of a deep learning defect detection model of another exemplary embodiment of the present disclosure. As shown in FIG3 , the intermediate feature layer 112 in FIG2 includes a first cross-stage convolution block, a second cross-stage convolution block, and a third cross-stage convolution block. Optionally, the detection head 12 includes a first depthwise separable convolution block, a second depthwise separable convolution block, and a third depthwise separable convolution block. The input block 111 inputs the received image to be processed into the first cross-stage convolution block. After the first cross-stage convolution block processes the first image to be processed, it obtains a plurality of first discarded feature maps of the image to be processed, and inputs the plurality of first discarded feature maps into the first depthwise separable convolution block for processing. The first depthwise separable convolution block processes the plurality of first discarded feature maps respectively, and predicts a plurality of first category prediction results and a plurality of first position prediction results of the first defect in the image to be processed based on each first discarded feature map.

[0099] It should be noted that the backbone network structure in the embodiments of the present disclosure is not limited to the cross-stage convolution block exemplified, and can also be replaced by at least one of RepVGG, EfficientNet, ConvNeXt, etc.

[0100] In the embodiment of Figure 3, the first cross-stage convolution block further inputs the plurality of first discarded feature maps to the second cross-stage convolution block. The second cross-stage convolution block processes each first discarded feature map and outputs a plurality of second discarded feature maps.

[0101] The second cross-stage convolution block inputs each second discarded feature map into a second depthwise separable convolution block. The second depthwise separable convolution block receives the plurality of second discarded feature maps, processes each second discarded feature map, and predicts a plurality of second category prediction results and a plurality of second position prediction results for a second defect in the processed image.

[0102] The second cross-stage convolution block further inputs the plurality of second discarded feature maps to a third cross-stage convolution block. The third cross-stage convolution block processes each second discarded feature map and outputs a plurality of third discarded feature maps.

[0103] The third cross-stage convolution block inputs each third discarded feature map into a third depthwise separable convolution block. The third depthwise separable convolution block receives the plurality of third discarded feature maps, processes each of the third discarded feature maps, and predicts a plurality of third category prediction results and a plurality of third position prediction results for a third defect in the processed image.

[0104] Continuing to refer to FIG3 , the detection head 12 is further used to perform Bayesian prediction / analysis based on multiple category prediction results and multiple position prediction results predicted by each depthwise separable convolution block to obtain a category detection result, a position detection result, a category uncertainty, and a position uncertainty of each defect.

[0105] For example, Bayesian prediction processing (such as averaging and variance calculation) is performed on multiple first category prediction results and multiple first position prediction results of the first defect predicted by the first depthwise separable convolution block to obtain the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect.

[0106] For another example, Bayesian prediction processing is performed on multiple second category prediction results and multiple second position prediction results of the second defect predicted output by the second depth-separable convolution block to obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect.

[0107] For another example, Bayesian prediction processing is performed on multiple third category prediction results and multiple third position prediction results of the third defect predicted by the third depth-separable convolution block to obtain the third category detection result, third position detection result, third category uncertainty and third position uncertainty of the third defect.

[0108] In an embodiment of the present disclosure, the multiple category prediction results of defects in the image to be processed obtained include multiple first category prediction results for the first defect, multiple second category prediction results for the second defect, and multiple third category prediction results for the third defect. The multiple position prediction results of defects in the image to be processed obtained include multiple first position prediction results for the first defect, multiple second position prediction results for the second defect, and multiple third position prediction results for the third defect. The category detection result of defects in the image to be processed obtained includes the first category detection result of the first defect, the second category detection result of the second defect, and the third category detection result. The position detection result of defects in the image to be processed obtained includes the first position detection result of the first defect, the second position detection result of the second defect, and the third position detection result of the third defect. The category uncertainty of defects in the image to be processed obtained includes the first category uncertainty of the first defect, the second category uncertainty of the second defect, and the third category uncertainty of the third defect. The position uncertainty of defects in the image to be processed obtained includes the first position uncertainty of the first defect, the second position uncertainty of the second defect, and the third position uncertainty of the third defect.

[0109] In Figure 3, taking the image to be processed as an image of a mask fabric on a production line, the category detection result, i.e., the classification result, can be any of adhesive tape / stickers, stains, printing errors, creases (wrinkles in the mask fabric), etc. It is understood that these four classification results (e.g., C=4) are only for illustration and are not limited to one, and can be set according to the actual scenario. The position detection result is the location / position information of the defect in the image to be processed.

[0110] As shown in Figure 4, the dropout layer 1121 includes a first dropout layer and a second dropout layer. The first dropout layer is located in the first cross-stage convolution block, and the second dropout layer is located in the second cross-stage convolution block.

[0111] The input of the first cross-stage input layer in the first cross-stage convolution block is connected to the output of input block 111 in the embodiment of FIG3 . The output of the first cross-stage input layer is connected to the input of the first convolution layer and the second convolution layer, respectively. The output of the first convolution layer is connected to the input of the first intermediate layer. The output of the first intermediate layer is connected to the first input of the first fusion layer. The output of the second convolution layer is connected to the second input of the first fusion layer. The output of the first fusion layer is connected to the input of the third convolution layer. The input of the third convolution layer is connected to the input of the first dropout layer. The output of the first dropout layer is connected to the input of the second cross-stage input layer in the second cross-stage convolution block and the input of the first depthwise separable convolution block, respectively.

[0112] The output of the second cross-stage input layer in the second cross-stage convolutional block is connected to the inputs of the fourth and fifth convolutional layers, respectively. The output of the fourth convolutional layer is connected to the input of the second intermediate layer. The output of the second intermediate layer is connected to the first input of the second fusion layer. The output of the fifth convolutional layer is connected to the second input of the second fusion layer. The output of the second fusion layer is connected to the input of the sixth convolutional layer. The input of the sixth convolutional layer is connected to the input of the second dropout layer. The output of the second dropout layer is connected to the input of the second depthwise separable convolutional block.

[0113] In the embodiment of FIG4 , the convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, and the sixth convolution layer may all be 1×1.

[0114] As shown in FIG5 , the first intermediate layer includes a first depthwise separable convolutional layer, and the second intermediate layer includes a second depthwise separable convolutional layer.

[0115] The input of the first intermediate input layer of the first intermediate layer is connected to the output of the first convolutional layer in the embodiment of FIG4 . The output of the first intermediate input layer is respectively connected to the input of the first depthwise separable convolutional layer and the input of the first accumulation layer. The output of the first accumulation layer is connected to the input of the first fusion layer in the embodiment of FIG4 .

[0116] The input of the second intermediate input layer of the second intermediate layer is connected to the output of the fourth convolutional layer in the embodiment of Figure 4. The output of the second intermediate input layer is respectively connected to the input of the second depthwise separable convolutional layer and the input of the second accumulation layer. The output of the second accumulation layer is connected to the input of the second fusion layer in the embodiment of Figure 4.

[0117] As shown in Figure 6, the first depthwise separable convolutional layer includes a first depthwise separation input layer, a first depthwise convolutional layer, a first pointwise convolutional layer, and a first activation function layer. The second depthwise separable convolutional layer includes a second depthwise separation input layer, a second depthwise convolutional layer, a second pointwise convolutional layer, and a second activation function layer.

[0118] The input of the first depthwise separation input layer is connected to the output of the first intermediate input layer in the embodiment of FIG5 . The output of the first depthwise separation input layer is connected to the input of the first depthwise convolutional layer and the input of the first activation function layer, respectively. The output of the first depthwise convolutional layer is connected to the input of the first pointwise convolutional layer. The output of the first pointwise convolutional layer is connected to the first activation function layer.

[0119] The input of the second depthwise separation input layer is connected to the output of the second intermediate input layer in the embodiment of FIG5 . The output of the second depthwise separation input layer is respectively connected to the input of the second depthwise convolutional layer and the input of the second activation function layer. The output of the second depthwise convolutional layer is connected to the input of the second pointwise convolutional layer. The output of the second pointwise convolutional layer is connected to the second activation function layer.

[0120] In the embodiment of FIG7 , the drop layer further includes a third drop layer. As shown in FIG7 , the third cross-stage convolution block includes a third cross-stage input layer, a seventh convolution layer, a third intermediate layer, an eighth convolution layer, a third fusion layer, a ninth convolution layer, and the third drop layer. The input end of the third cross-stage input layer is connected to the output end of the second cross-stage convolution block, for example, to the output end of the second drop layer, and the multiple second drop feature maps output by the second drop layer are respectively used as the third cross-stage input feature maps.

[0121] The seventh convolutional layer is configured to perform convolution processing on the third cross-stage input feature map input by the third cross-stage input layer, and input a fifth convolutional feature map to the third intermediate layer. The third intermediate layer is configured to process the fifth convolutional feature map and input a fifth feature map to be fused to the third fusion layer. The eighth convolutional layer is configured to perform convolution processing on the third cross-stage input feature map and input a sixth feature map to be fused to the third fusion layer. The third fusion layer is configured to fuse the fifth feature map to be fused with the sixth feature map to be fused and input the third fused feature map to the ninth convolutional layer. The ninth convolutional layer is configured to perform convolution processing on the third fused feature map and input the sixth convolutional feature map to the third dropout layer. The third dropout layer is configured to randomly drop features in the sixth convolutional feature map to obtain multiple third dropped feature maps. The multiple features obtained after the image to be processed by the feature extraction module and subjected to random dropout processing include the multiple third dropped feature maps.

[0122] Continuing to refer to Figure 7, the third intermediate layer includes a third intermediate input layer, a third depth-wise separable convolution layer, and a third accumulation layer. The third intermediate input layer is used to receive the fifth convolution feature map output by the seventh convolution layer, and input the fifth convolution feature map into the third depth-wise separable convolution layer and the third accumulation layer, respectively. The third depth-wise separable convolution layer is used to perform depth-wise separable convolution processing on the fifth convolution feature map, and input the third depth-wise separable convolution feature map into the third accumulation layer. The third accumulation layer accumulates the fifth convolution feature map and the third depth-wise separable convolution feature map to obtain the fifth feature map to be fused.

[0123] Continuing to refer to Figure 7, the third depthwise separable convolution layer includes a third depthwise separation input layer, a third depthwise convolution layer, a third pointwise convolution layer and a third activation function layer. The third depthwise separation input layer is used to receive the fifth convolution feature map output by the third intermediate input layer, and input the fifth convolution feature map into the third depthwise convolution layer and the third activation function layer, respectively. The third depthwise convolution layer is used to perform depthwise convolution processing on the fifth convolution feature map, and input the third depthwise convolution feature map into the third pointwise convolution layer. The third pointwise convolution layer is used to perform pointwise convolution processing on the third depthwise convolution feature map, and input the third pointwise convolution feature map into the third activation function layer. The third activation function layer is used to process the third pointwise convolution feature map and the fifth convolution feature map to obtain the third depthwise separation convolution feature map.

[0124] In the embodiments of the present disclosure, the activation function used by the activation function layer (for example, including the first to third activation function layers) can be any appropriate activation function. In the following embodiments, SiLU (Swish activation function) is used as an example, but the present disclosure is not limited to this. The SiLU function is differentiable in the entire domain, so the gradient calculation during the back propagation process is more stable. The SiLU function has the characteristics of no upper bound, lower bound, smoothness, non-monotonicity in deep models, and a larger and more stable gradient.

[0125] In the embodiment of FIG7 , the convolution kernel sizes of the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer may all be 1×1.

[0126] A depthwise separable convolution layer is introduced in the cross-stage convolution block of the embodiment of the present disclosure. The depthwise separable convolution layer reduces the number of parameters required for convolution calculation and improves the efficiency of the use of convolution kernel parameters by splitting the correlation between the spatial dimension and the channel (depth) dimension. The convolution calculation process of the depthwise separable convolution layer is divided into two parts: depthwise convolution, which performs spatial convolution on each channel separately, that is, a single convolution kernel is used to perform convolution operation on each input channel, and then the outputs are spliced; pointwise convolution, which uses a unit convolution kernel (that is, a 1x1 convolution kernel) to perform channel convolution to integrate the information of each channel and obtain the final feature map. The advantage of depthwise separable convolution is that it reduces the computational complexity and the number of model parameters while maintaining the expressiveness of the model. Depthwise separable convolution is a lightweight convolution method that improves the efficiency and performance of the model by reducing parameters and computational complexity.

[0127] The embodiment of FIG8 differs from the embodiment of FIG3 in that the feature extraction module 11 may further include a fourth cross-stage convolution block, and the detection head 12 may further include a fourth depthwise separable convolution block. The structure of the fourth cross-stage convolution block can be seen in FIG4 to FIG7 . The fourth cross-stage convolution block is connected to the third cross-stage convolution block and the fourth depthwise separable convolution block in the detection head 12, and is configured to receive multiple third discarded feature maps, respectively, and process the multiple third discarded feature maps to generate multiple fourth discarded feature maps. The fourth cross-stage convolution block may include a fourth dropout layer to output the multiple fourth discarded feature maps to the fourth depthwise separable convolution block. The fourth depthwise separable convolution block may predict and output multiple fourth category prediction results and multiple fourth position prediction results for a fourth defect. The detection head 12 may obtain a fourth category detection result and fourth category uncertainty for the fourth defect based on the multiple fourth category prediction results, and obtain a fourth position detection result and fourth position uncertainty for the fourth defect based on the multiple fourth position prediction results.

[0128] One of the differences between the embodiment of FIG. 9 and the embodiment of FIG. 3 is that the deep learning defect detection model 1 further includes a feature pyramid module 13 located between the feature extraction module 11 and the detection head 12 .

[0129] The embodiment shown in Figure 9 addresses the multi-scale issue in defect detection by introducing a feature pyramid module 13 between the feature extraction module 11 and the detection head 12. Defects of different sizes exhibit distinct features in an image, and the feature pyramid can extract and fuse these features at different scales. By modifying the network connection, the feature pyramid significantly improves the performance of small defect detection without significantly increasing the computational complexity of the original model. Specifically, the feature pyramid fuses feature layers of different sizes through upsampling and downsampling operations. This allows the model to simultaneously capture high-level semantic information and low-level detail information, thereby extracting better features. This fused feature not only helps improve the model's detection performance but also makes it more adaptable to defect detection tasks of varying sizes. First, the intermediate feature layers in the feature extraction module 11 generate feature maps of different scales. These feature maps, from the bottom layer to the top layer in the pyramid structure, contain increasingly abstract information, corresponding to defects of varying sizes. Low-level feature maps typically have higher resolution and more detailed information, making them suitable for detecting small defects; high-level feature maps, on the other hand, have lower resolution and more abstract semantic information, making them suitable for detecting large defects. Next, the feature pyramid fuses these feature maps of different scales through a series of upsampling and downsampling operations. Specifically, the feature maps of the lower layers are upsampled to match the resolution of the feature maps of the higher layers, and then fused with the feature maps of the higher layers through concatenation or addition. This fused feature map contains both low-level detail information and high-level semantic information, helping to improve the model's detection performance for defects of varying sizes. By concatenating feature maps of different scales, multiple predictions of different scales can be generated. These predictions correspond to defects of different sizes, thus enabling multi-scale defect detection. This mechanism enables the model to simultaneously focus on global information and local details in the image, thereby improving the accuracy and robustness of defect detection.

[0130] For example, as shown in Figure 9, the feature pyramid module 13 includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block. In the embodiment of the present disclosure, the cross-stage convolution block can be set to implement downsampling of the feature map. In the embodiment of Figure 9, feature maps of three different scales are input to the feature pyramid module 13 as input to the YOLO multi-scale detection head 12. Because the cross-stage convolution block can enable the downsampling function by defining parameters, more than three cross-stage convolutions can be used to implement three feature maps of different scales input to the feature pyramid module 13.

[0131] The first input of the first spatial pyramid pooling block is connected to the output of the first cross-stage convolution block, for example, to the output of the first dropout layer, to receive the multiple first dropout feature maps output by the first dropout layer. The second input of the first spatial pyramid pooling block is connected to the output of the second spatial pyramid pooling block to receive the multiple second spatial pooling feature maps output by the second spatial pyramid pooling block. The first spatial pyramid pooling block processes each of the multiple first dropout feature maps and a corresponding second spatial pooling feature map in the multiple second spatial pooling feature maps, and outputs the multiple first spatial pooling feature maps to the first depthwise separable convolution block in the detection head 12 via its output. The first depthwise separable convolution block in the detection head 12 processes each of the first spatial pooling feature maps to predict multiple first category prediction results and multiple first position prediction results for the first defect. The detection head 12 performs Bayesian prediction on the multiple first category prediction results and the multiple first position prediction results for the first defect to obtain a first category detection result, a first position detection result, a first category uncertainty, and a first position uncertainty for the first defect.

[0132] The first input of the second spatial pyramid pooling block is connected to the output of the second cross-stage convolution block, for example, to the output of the second dropout layer, to receive the multiple second dropout feature maps output by the second dropout layer. The second input of the second spatial pyramid pooling block is connected to the output of the third spatial pyramid pooling block to receive the multiple third spatial pooling feature maps output by the third spatial pyramid pooling block. The second spatial pyramid pooling block processes each of the multiple second dropout feature maps and a corresponding third spatial pooling feature map in the multiple third spatial pooling feature maps, and outputs the multiple second spatial pooling feature maps to the second depthwise separable convolution block in the detection head 12 via its output. The second depthwise separable convolution block in the detection head 12 processes each of the second spatial pooling feature maps to predict multiple second category prediction results and multiple second position prediction results for the second defect. The detection head 12 performs Bayesian prediction on the multiple second category prediction results and the multiple second position prediction results for the second defect to obtain a second category detection result, a second position detection result, a second category uncertainty, and a second position uncertainty for the second defect.

[0133] The input of the third spatial pyramid pooling block is connected to the output of the third cross-stage convolution block, for example, to the output of the third dropout layer, to receive the multiple third dropout feature maps output by the third dropout layer. The third spatial pyramid pooling block processes each of the multiple third dropout feature maps and outputs the multiple third spatial pooling feature maps to the third depthwise separable convolution block in the detection head 12 via its output. The third depthwise separable convolution block in the detection head 12 processes each of the third spatial pooling feature maps to predict multiple third category prediction results and multiple third position prediction results for the third defect. The detection head 12 performs Bayesian prediction on the multiple third category prediction results and multiple third position prediction results for the third defect to obtain a third category detection result, a third position detection result, a third category uncertainty, and a third position uncertainty for the third defect. In the embodiment of FIG9 , the defects in the image to be processed also include a third defect, the multiple category prediction results for the defects in the image to be processed also include a third category prediction result for the third defect, and the multiple position prediction results also include a third position prediction result for the third defect. The obtained category detection result, position detection result, category uncertainty and position uncertainty of the defect in the image to be processed also include a third category detection result, a third position detection result, a third category uncertainty and a third position uncertainty of the third defect, respectively.

[0134] In the disclosed embodiment, the feature pyramid module 13 can fuse convolutional features of receptive fields of different sizes under the original three-scale feature maps to achieve multi-scale feature fusion and enhance the robustness of the features.

[0135] A second difference between the embodiment of FIG. 9 and the embodiment of FIG. 3 is that the deep learning defect detection model 1 may further include a background removal module located before the feature extraction module 11. The background removal module is used to remove background information from the image to be processed and input the image to be processed after the background information is removed into the feature extraction module 11. Here, the image to be processed without background information removed is used as an example of a mask cloth image, and the mask cloth image with background information removed is referred to as the image to be processed.

[0136] FIG10 is a schematic diagram of a spatial pyramid pooling block according to an exemplary embodiment of the present disclosure. As shown in FIG10 , each spatial pyramid pooling block in the feature pyramid module 13 (e.g., the first through third spatial pyramid pooling blocks in FIG9 ) may have the same structure. The spatial pyramid pooling block may include an input layer, a first convolutional layer connected to the input layer, three parallel max pooling layers connected to the first convolutional layer, a fusion layer connected to each of the three max pooling layers, and a second convolutional layer connected to the fusion layer.

[0137] The maximum pooling layers in the spatial pyramid pooling block can be used to reduce the number of feature map parameters, improve computation speed, and increase the receptive field, representing a downsampling operation. Through pooling, the model can focus on global features rather than local locations. This dimensionality reduction process can preserve important feature information, improve fault tolerance, and also help prevent overfitting to a certain extent.

[0138] In the embodiment of FIG10 , the convolution kernel sizes of the first convolution layer and the second convolution layer may both be 1×1.

[0139] Continuing to refer to Figure 10, the convolution kernel sizes of the three parallel maximum pooling layers can be 3×3, 5×5, and 7×7 respectively.

[0140] It should be noted that the feature pyramid module in the embodiment of the present disclosure is not limited to the spatial pyramid pooling blocks exemplified in Figures 9 and 10, and other structures may be used instead. In the embodiment of Figure 10, for the same feature map, convolution kernels of different sizes are used for parallel processing, and then the feature map after direct fusion processing is obtained. Other structures, such as those shown in Figure 11, can be changed to serial processing of the feature map of the input value feature pyramid module using convolution kernels of different sizes. For example, three convolution layers with convolution kernel sizes of 3×3, 5×5, and 7×7 are used in series, and then a shortcut (direct connection or shortcut) is used to jump to the feature maps output by different convolution kernels to achieve fusion.

[0141] The third difference between the embodiment of FIG. 9 and the embodiment of FIG. 3 is that the input block 111 may be an attention block 111 ′. The attention block 111 ′ performs a slicing operation on the image to be processed (b1, c1, w1, and h1 represent the batch size, number of channels, width, and height of the image to be processed, respectively) before it enters the feature extraction module 11. Specifically, the attention block 111 ′ obtains a value for every other pixel in the image to be processed, similar to neighboring subsampling. This results in four complementary images with no information loss. This concentrates the w1 / h1 information in the channel space, expanding the input channel by a factor of four (obtaining an input of (b1, 4*c1, w1, h1)). This means that the spliced ​​image becomes 4*c1 channels instead of the original c channels. Finally, the resulting new image is subjected to a convolution operation (e.g., with a convolution kernel size of 1x1), ultimately obtaining a doubly downsampled feature map with no information loss, which is then input to the first cross-stage convolution block. After such processing, the attention block 111' obtains a two-fold downsampled feature map without information loss, which can reduce the number of parameters, the number of network layers, the gradient, and the number of calculations, while improving the precision and recall rate of the model. This enables the model to more accurately detect and identify defects in the image while maintaining high efficiency, especially those with small size and unclear features.

[0142] In the embodiment of Figure 12, a spatial attention mechanism is used to design the attention block. It is understandable that the attention block in the embodiment of the present disclosure is not limited to the structure shown in Figure 12. Channel attention blocks can also be implemented through Squeeze-Excitation (compression-excitation, explicitly modeling the interdependence between feature channels, obtaining the importance of each channel through learning, and then weighting the features on each channel according to this importance, highlighting important features and suppressing unimportant features) operations. In addition, spatial attention blocks can also be implemented through dynamic convolution.

[0143] It is understandable that the model provided by the embodiments of the present disclosure is not limited to the structures exemplified above. The embodiments of the present disclosure provide a method that adds a dropout layer to the intermediate feature layer and adds a Bayesian loss function (such as the second loss function described below) during the training process of the model. The methods provided by the embodiments of the present disclosure can adapt to various network structures, achieve the integration of uncertainty estimation while maintaining compatibility with various network architectures, provide flexibility in application, and further improve performance.

[0144] In the disclosed embodiments, the use of convolutional layers enables the model to adapt to different image resolutions and blemish / defect detection scenarios. The convolution operation allows the network to focus on local features in the image rather than the entire image. Due to this property of the convolutional layer, the model is able to process input images of different sizes without having to make major modifications to the model structure. This makes the model highly flexible and adaptable to various image resolutions. In the field of defect detection, this means that regardless of whether the input image is high-definition or low-resolution, or how the size and position of the defects in the image vary, the model using convolutional layers can effectively perform defect detection. Therefore, this model can easily adapt to different defect detection scenarios, improving the versatility and practicality of the model.

[0145] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: obtaining a target training data set, which includes a target sample image and the labeled category and labeled position of the target sample defect in the target sample image; inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defect; obtaining the category detection result, position detection result, category uncertainty and position uncertainty of the target sample defect based on the multiple category prediction results and multiple position prediction results of the target sample defect; constructing a loss function based on the labeled category and category detection result, labeled position and position detection result, and category uncertainty and position uncertainty of the target sample defect to train the deep learning defect detection model.

[0146] In an embodiment of the present disclosure, a data set for defect detection can be obtained and divided into a target training data set and a target test data set according to a predetermined ratio, for example, a ratio of 9:1. During model training, a dropout operation is performed during the forward propagation process. Under the effect of dropout, multiple different network structures can be obtained by randomly closing neurons, thereby making multiple predictions for the same target sample image, and obtaining multiple category prediction results and multiple position prediction results. The mean of the multiple category prediction results and the mean of the multiple position prediction results are calculated to represent the category detection result and the position detection result of the target sample defect, respectively. The mean of the multiple category prediction results and the variance of the multiple position prediction results are calculated to represent the category uncertainty and the position uncertainty of the target sample defect, respectively.

[0147] For example, the training parameters of the model can be set as follows: batch size is set to 64, number of epochs is set to 100, initial learning rate is set to 0.001, decay coefficient is set to 0.005, and dropout probability is set to 0.3. After dropout, the number of neurons in each layer is reduced to about half of the original number.

[0148] In an exemplary embodiment, the loss function includes a first loss function and a second loss function; when the difference between the labeled category and the category detection result of the target sample defect is greater, and the difference between the labeled position and the position detection result is greater, the value of the first loss function is greater; when the category uncertainty and the position uncertainty of the target sample defect are greater, the value of the second loss function is greater.

[0149] The disclosed embodiment introduces a Bayesian approach, adding dropout to the model to create random uncertainty to implement the Bayesian posterior distribution. At the same time, the detection head of the model is optimized, and the mean and variance of the anchor boxes (i.e., position prediction results) and categories (i.e., category prediction results) are added to represent their distribution. For example, the second loss function can be a negative log-likelihood loss function to optimize uncertainty estimation, which not only drives the model to evolve towards accurate predictions, but also encourages the model to accurately estimate the uncertainty of its predictions, thereby improving the detection accuracy and robustness of the model.

[0150] For example, the first loss function L1 is as follows:

[0151] MSE is the bounding box regression calculation, y i is the coordinate of the i-th bounding box in the real data (i.e., the marked position of the i-th target sample defect), is the i-th bounding box coordinate predicted by the model (i.e., the position detection result of the i-th target sample defect), N is the number of bounding boxes (i.e., the number of candidate defects in the target sample), N is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N. CE represents the multivariate cross entropy calculation, Represents the category confidence of category c in the predicted category detection result (i.e., the probability of belonging to category c). c The confidence of category c in the real data (i.e., the category confidence of category c in the labeled category). C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C. BCE represents the binary cross entropy calculation, pre represents the predicted target confidence, and o represents the confidence of the target in the real data, that is, the confidence of the target marked in the target sample defect.

[0152] For another example, the second loss function L2 is as follows:

[0153] μ i y is the predicted position detection result of the i-th target sample defect, for example, the mean of the i-th bounding box coordinates predicted by the model. i is the labeled location of the defect in the i-th target sample, for example, the i-th bounding box coordinate in the real data. is the position uncertainty of the predicted defect of the i-th target sample, for example, the variance of the i-th bounding box coordinates predicted by the model. N is the number of defects in the target sample, N is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N. μ c Represents the category confidence mean of the predicted category c, that is, the category detection result. c is the category confidence of category c in the real data. is the class confidence variance of the predicted class c, that is, the class uncertainty. C represents the number of classes, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C.

[0154] In an exemplary embodiment, obtaining a target training dataset includes: obtaining an initial sample image and annotated categories and locations of initial sample defects in the initial sample image; and performing any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an augmented sample image and annotated categories and locations of extended sample defects in the extended gain image. The target sample image includes the initial sample image and the augmented sample image.

[0155] In an exemplary embodiment, the initial sample image is subjected to defect synthesis processing to generate an augmented sample image, including: extracting initial sample defects from the initial sample image; synthesizing the initial sample defects with an augmented background image to generate the augmented sample image; and / or, learning the initial sample image through a deep learning generative model to generate the augmented sample image.

[0156] Dataset quality can impact model accuracy and robustness. For example, to train an efficient and accurate model for defect detection in mask fabric images, a large number of high-quality, defective mask fabric images are required to form a dataset. If the image quality in the dataset is low or the sample size is insufficient, the model's accuracy and robustness may be affected.

[0157] In reality, the number of mask cloth images with defects is less than the number of mask cloth images without defects, that is, the number of positive and negative samples in the training dataset is unbalanced. This disclosure solves the dataset quality problem by implementing a multi-stage data augmentation strategy. First, a large number of original mask cloth images are collected and initial training data (including initial sample images) are generated through expert annotation. Then, the dataset is expanded using one or more image processing techniques such as random cropping, rotation, scaling, and illumination change simulation, as well as synthetic defect generation methods, to obtain amplified sample images.

[0158] In the disclosed embodiments, the defective mask cloth image can be enhanced by using any one or more of the following data augmentation methods: random cropping, rotation, scaling, illumination change simulation (e.g., brightness adjustment), defect synthesis, adding Gaussian noise, random occlusion, etc. Illumination change simulation is to randomly adjust one or more of the saturation, brightness, contrast, etc. of the original defective mask cloth image, which can simulate the vulnerability of the captured image to changes in illumination and weather; random occlusion is to block random areas of the defective mask cloth image with black, which can simulate the situation where it is easily blocked by other objects during shooting; Gaussian noise is to add noise whose probability density function follows a Gaussian distribution to the original defective mask cloth image, which can better simulate unknown real noise.

[0159] Models in related technologies struggle to strike a good balance between detection speed and accuracy. This can lead to unsatisfactory detection performance for small defects in practical applications, as these require processing high-resolution images, which increases computing power and reduces detection speed. The disclosed embodiments utilize data augmentation to randomly scale and crop multiple initial sample images for splicing, significantly increasing the number of small defects in the training dataset and improving detection performance.

[0160] The disclosed embodiment can also generate new defect samples based on real defect samples by introducing AIGC (stable diffusion) to increase data diversity. Any one or two of the following two methods of synthesizing defects can be used: the first is to extract the defect foreground and replace the background of other fabrics. The second is to learn the real defect image through stable diffusion and then generate a fake defect image. By introducing AIGC into the field of defect sample generation, new defect samples can be generated based on real defect samples, which can enrich the data set and help train a more accurate model. The generated defect samples can be used to simulate various possible defect situations, so as to more comprehensively test the performance of the defect detection system in practical applications.

[0161] In an exemplary embodiment, obtaining a target training data set includes: obtaining candidate sample images and the labeled categories and labeled positions of candidate sample defects in the candidate sample images; obtaining output features of the candidate sample images through the feature extraction module; obtaining feature distances between the candidate sample images based on the output features of each candidate sample image; and selecting candidate sample images with feature distances greater than a first distance threshold among candidate sample images with the same labeled category, and / or candidate sample images with feature distances less than a second distance threshold among candidate sample images with different labeled categories, as target sample images in the target training data set.

[0162] In the embodiment of the present disclosure, Euclidean distance may be used as the feature distance, but the present disclosure is not limited thereto.

[0163] The disclosed embodiments can also employ an adaptive sample selection mechanism to prioritize samples for training that are most effective in improving model performance. For example, during training, features of the backbone network outputs of multiple samples are extracted to capture similarities and differences between different samples. Feature distances are then calculated for each sample.

[0164] For example, the i-th sample (x i ) and the jth sample (x j ) between the characteristic distance d ij :

[0165] F k (x i ) is the sample x i The eigenvalue in the kth dimension, F k (x j ) is the sample x jThe eigenvalue of the kth dimension, where D is the dimension of the sample feature, D is a positive integer greater than or equal to 1, and k is a positive integer greater than or equal to 1 and less than or equal to D. i and j are both positive integers greater than or equal to 1, and i is not equal to j.

[0166] Calculating the characteristic distance of samples helps us understand their distribution in feature space. For example, we can select the farthest sample from the same type of sample (images corresponding to the same type of defect, that is, different samples with the same label, such as samples all labeled "tape") and the closest sample from different types of samples (images corresponding to different types of defects, such as one sample labeled "stain" and another labeled "crease") to participate in backpropagation. This is equivalent to using the most error-prone (most difficult to judge) samples to optimize the model weights, thereby improving the model's accuracy and robustness.

[0167] In the disclosed embodiment, the sample selection criteria of the adaptive sample selection mechanism is to first calculate the feature distance, and then divide the first distance threshold and the second distance threshold according to the statistical value of the feature distance. For example, the first distance threshold (also called intra-class distance) is selected to be greater than 0.3, and the second distance threshold (also called inter-class distance) is selected to be less than 0.2. This training strategy of the adaptive sample selection mechanism is newly added after conventional training and is used for secondary training to solve the hard sample problem. After secondary training, the feature distribution of intra-class samples is more concentrated, and the feature distribution of inter-class samples is more discrete, thereby enhancing classification accuracy.

[0168] Based on the calculated feature distance, selecting the farthest samples from the same type means these samples have the greatest dissimilarity in feature space, making it difficult for the model to distinguish them correctly, even if they belong to the same category. By having the model learn these difficult-to-distinguish samples, it can focus more on detailed differences within categories, thereby improving model accuracy. At the same time, selecting the closest samples from different types means these samples are highly similar in feature space, but actually belong to different categories. These samples are the most prone to errors because they have similar features but different labels. Learning from these samples strengthens the model's ability to distinguish between different categories, thereby improving its robustness. Using this strategy for backpropagation is equivalent to using the most error-prone (most difficult to distinguish) samples to optimize the model's weights. This allows the model to focus more on these challenging samples during training, thereby improving its generalization and robustness in complex and changing environments.

[0169] In an exemplary embodiment, during the training process of the deep learning defect detection model: the weight tensor of each convolutional layer in the deep learning defect detection model is extracted; the Euclidean norm of each channel in the weight tensor is obtained; each convolutional layer is sorted according to the Euclidean norm of its channel, and the corresponding channel is pruned according to a pruning threshold or a pruning ratio.

[0170] Channel pruning is an effective model compression and acceleration technique, particularly suitable for lightweight model design. Performing channel pruning in a model can significantly reduce the number of parameters and computational complexity while maintaining model performance.

[0171] In the disclosed embodiment, the contribution of the convolution channel to the network output is measured based on the L2-norm (Euclidean norm), and channel pruning is performed according to the degree of contribution. First, the contribution of the convolution channel is measured based on the L2-norm: the L2-norm (Euclidean norm) can be used to evaluate the importance or contribution of the convolution channel. For each convolution layer in the model, the L2-norm value of each convolution channel weight can be calculated. This value reflects the influence and contribution of the channel in the network. Specifically, for each convolution layer in the model, its weight tensor is extracted. The L2-norm value of each channel in the weight tensor is calculated. For example, operations such as squaring, summing, and square root are performed on the weight tensor. The calculated L2-norm value is saved for subsequent channel pruning decisions. Then, channel pruning is performed according to the degree of contribution: based on the L2-norm value calculated in the previous step, the importance of each convolution channel can be evaluated, and channel pruning is performed according to a certain strategy. The goal of channel pruning is to remove channels that contribute less to the network output, thereby reducing the number of model parameters and computational complexity. Specifically, a pruning threshold or pruning ratio is set. The pruning threshold or pruning ratio can be determined based on specific needs and experiments. For each convolutional layer, the channels are sorted according to their L2-norm values. According to the set pruning threshold or pruning ratio, channels with smaller contributions are selected for pruning. For example, the weights of the corresponding channels are set to zero or these channels are directly removed from the model. After pruning, the model is retrained and fine-tuned to restore or maintain its performance. The Euclidean norm reflects the response weight of the channel to the output, and channels with lower response weights are pruned first.

[0172] In some embodiments, in order to strike a balance between detection speed and accuracy, the deep learning defect detection model is lightweight designed, the number of feature channels is reduced through channel pruning, and the original convolution layer is replaced with depthwise separable convolution to reduce the model parameters and computational complexity, thereby achieving faster detection while maintaining high accuracy.

[0173] In an exemplary embodiment, the target sample images include mask images (or mask cloth images) with different materials, textures, colors, and prints.

[0174] In order to increase the generalization ability of the model, the embodiment of the present disclosure designs a mixed dataset. Taking mask cloth images as an example, the mixed dataset contains multiple types of mask samples, such as mask images with different mask materials, colors, textures, and print patterns.

[0175] For certain types of mask materials, such as printed mask cloth, the patterns can vary greatly, and the training samples cannot include sample images of masks with all patterns, resulting in the model not being able to achieve high detection accuracy. In the disclosed embodiments, transfer learning and domain adaptation techniques, including one or more of feature space alignment, fine-tuning, and metric learning, can be utilized to enable the model to adapt to new materials, colors, textures, printed patterns, and other types.

[0176] In some embodiments, the model is periodically retrained on newly collected data to maintain its generalization performance.

[0177] In an exemplary embodiment, obtaining an image to be processed includes obtaining the image to be processed from an image capture device in an image acquisition component, the image acquisition component being configured to capture the image to be processed of an object to be inspected during a production process. The method further includes obtaining a current production speed of the object to be inspected and a current processing speed of the image to be processed by the deep learning defect detection model; and adjusting the number of parallel threads or processes used by the deep learning defect detection model to process the image to be processed based on the current production speed and the current processing speed.

[0178] In the embodiment of the present disclosure, a real-time detection feedback mechanism is introduced to dynamically adjust the model inference time (i.e., the current processing speed of the model) by changing the number of parallel threads of model inference according to the current production line speed (or current production speed), ensuring that the required detection speed is achieved while maintaining high accuracy.

[0179] For example, after pruning, the model is deployed using TensorRT. The encapsulated TensorRT architecture supports multi-process and multi-thread concurrency. At the same time, considering that long-term high load on the hardware will affect its lifespan, a dynamic adjustment mechanism is designed. In the disclosed embodiment, a thread pool can be used to manage the concurrent execution of inference tasks. The thread pool can ensure the effective use of thread resources and avoid the overhead caused by frequent creation and destruction of threads. Design a task scheduler to be responsible for assigning inference tasks to idle threads. A load balancing strategy can be used to ensure that the loads of each thread are relatively balanced to avoid situations where some threads are overloaded while other threads are idle.

[0180] TensorRT natively supports multi-threaded reasoning, which can be achieved through execute_async_v2 of IExecutionContext. In addition, the disclosed embodiment also takes advantage of the multi-core CPU (Central Processing Unit) to serialize the TensorRT engine to disk, and then load this serialized engine into multiple processes, allowing each process to independently execute reasoning tasks, maximizing the resources of the CPU and GPU (Graphics Processing Unit).

[0181] The disclosed embodiment can use sensors or other monitoring devices to obtain the current production speed in real time; the current inference speed of the monitoring model, that is, the time required to process each image, is used as its current processing speed. According to the comparison between the current production speed and the current processing speed of the model, the number of parallel inference threads of the model is adjusted. If the current production speed is accelerated, the number of parallel threads needs to be increased to speed up the inference speed; conversely, if the current production speed is slowed down, the number of parallel threads can be reduced to save resources. By combining a real-time detection feedback mechanism, dynamically adjusting the number of parallel threads for model inference, and the efficient deployment of TensorRT, it is possible to ensure that the defect detection tasks on the production line are both accurate and efficient.

[0182] In an exemplary embodiment, obtaining an image to be processed includes: obtaining the image to be processed from an image capturing device in an image acquisition component; if the grayscale value and grayscale distribution of the image to be processed do not meet preset conditions, sending a control signal to adjust the exposure parameters of the image capturing device; obtaining a new image to be processed from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset conditions is obtained.

[0183] To improve the model's robustness to environmental factors, an automatic exposure module is designed in the disclosed embodiment. Each time the system is turned on, an image capturing device, such as a camera, captures an image of the current environment and analyzes its grayscale distribution. If the grayscale value and grayscale distribution of the image do not meet the preset conditions, the exposure parameters are adjusted and the image is captured again for analysis. This cycle continues until the grayscale value and grayscale distribution of the captured image meet the requirements, and then it is used as the image to be processed by the input model. The automatic exposure module can monitor the lighting conditions in real time, calculate the latest exposure parameters of the camera in real time, and suppress the influence of external light.

[0184] For example, the desired preset condition is histogram equalization, with the grayscale mean between 130 and 140 and the grayscale variance as small as possible. The grayscale histogram, grayscale mean, and grayscale variance of the image captured by the camera are used to determine whether the preset conditions are met. If not, the target difference is used to determine whether to increase or decrease the exposure.

[0185] In an exemplary embodiment, the method provided by the embodiment of the present disclosure further includes: if the category uncertainty and / or location uncertainty of the defect is greater than an uncertainty threshold, sending prompt information for the defect.

[0186] For example, during online operation, the entire system is end-to-end. It takes the acquired mask cloth image as input, and the model first deletes irrelevant background to obtain the image to be processed. Then, it performs feature extraction and finally outputs the category detection result, position detection result, category uncertainty and position uncertainty of the defect. The system can determine that the defect with category uncertainty and / or position uncertainty greater than the uncertainty threshold (its value can be set according to actual needs) has a high uncertainty defect. At this time, a prompt message can be triggered to prompt the user that the prediction of the defect has a high uncertainty. Further, the user can determine how to deal with the defect, such as whether to manually perform the location and category annotation of the defect. That is, the embodiment of the present disclosure can also use the category uncertainty and / or position uncertainty predicted by the model to assist the user in identifying the most informative data for marking, thereby improving the efficiency of the annotation process.

[0187] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: counting the pixel number information of each defect detected in the image to be processed; determining the defect level of each defect based on the pixel number information of each defect; and performing corresponding processing based on the defect level of each defect.

[0188] The defect detection system provided by the embodiments of the present disclosure includes an algorithm layer and a data service layer / data service component. The algorithm layer deploys the above-mentioned deep learning defect detection model to process the image to be processed and obtain defect data, such as whether the image to be processed has defects. If defects are present, the defect information includes the defect category detection result, location detection result, category uncertainty, and location uncertainty. The data service layer is responsible for analyzing and processing the defect data provided by the algorithm layer to determine the severity of the defect. For example, by counting the number of pixels of the defect (such as the number of pixels contained in the long side of the defect), the defects are divided into four levels: the first level is a minor defect, such as a defect with less than 20 pixels on the long side; the second level is a general defect, such as a defect with greater than or equal to 20 and less than 60 pixels on the long side; the third level is a more serious defect, such as a defect with greater than or equal to 60 and less than 100 pixels on the long side; and the fourth level is a severe defect, such as a defect with more than 100 pixels on the long side. Based on this grading system, the data service layer can develop and implement corresponding handling strategies for different levels of defects. For example, when a severe defect is detected, an alarm is triggered, prompting the user (e.g., a worker) to immediately address it. Minor defects may not trigger an alarm, but only provide a prompt on the interactive interface, leaving it to the user to decide whether to address the defect. This ensures efficient and reliable defect detection and handling.

[0189] In an exemplary embodiment, corresponding processing is performed according to the defect level of each defect, including: in the image to be processed, defects at different defect levels are marked with frames of different thicknesses.

[0190] In the disclosed embodiment, the data service layer uses the thickness of the border of the defect area to assist in determining the classification level of the defect. The thicker the border, the higher the defect level, and the thinner the border, the lower the defect level. For example, the data service layer first counts the number of pixels of each defect, and then divides the defects into four levels based on the number of defect pixels. The first level of defects are then marked with the thinnest border in the image to be processed, and the fourth level of defects are marked with the thickest border in the image to be processed. The marked defect image is then provided to the user and displayed on the interactive interface, so that the user can immediately identify the severity of the defect and quickly decide whether to stop the machine for processing. This visual identification allows the operator to quickly and intuitively understand the severity of the defect during the subsequent manual inspection process.

[0191] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: saving the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed; receiving and retrieving corresponding defect information based on one or more of the input image features, detection time and detection serial number of the defect area.

[0192] The defect detection system provided by the embodiment of the present disclosure may also include a defect recording module. When a defect is detected, the defect recording module will automatically record the level of the defect (i.e., the defect level), the exact location (i.e., the location detection result), and the detailed area image of the defect (i.e., the defect or defect point). In order to further improve efficiency, the data service layer can also provide a defect information retrieval function. The image features of the defect / defect area (for example, the features extracted by the feature extraction module), the detection time, and the detection serial number will be saved in the database and can be retrieved through the above three data. Through this function, historical defect records can be quickly retrieved and reviewed to support the continuous improvement of quality according to the production process.

[0193] When the defect detection method and system are applied to non-woven mask fabric defect detection, they can also be referred to as non-woven mask fabric defect detection method and system. When the defect detection method and system are applied to the defect detection of hygiene and personal care product materials, they can also be referred to as the intelligent defect detection method and system for hygiene and personal care product materials.

[0194] An embodiment of the present disclosure also provides a defect detection system, including: an image acquisition component for acquiring the image to be processed of the object to be detected in the production process; a defect detection device, connected to the image acquisition component, for obtaining the image to be processed from the image acquisition component.

[0195] In which, the defect detection device includes: an image processing module, used to process the image to be processed through a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, and the intermediate feature layer includes a discarding layer, and the discarding layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module; the multiple features after random discarding processing are processed by the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; based on the multiple category prediction results and multiple position prediction results of the defects, the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed are obtained.

[0196] In an exemplary embodiment, the image acquisition component includes an image capture device, wherein the image capture device includes an area array camera, wherein the area array camera is used to capture an original image to be processed and perform pixel merging processing on the original image to be processed to generate the image to be processed.

[0197] An area scan camera is a camera used to capture images of the surface of an object to be inspected. It can use different types of sensors, such as CCD (Charge-Coupled Device), array cameras, or CMOS area scan cameras. These cameras consist of a fixed array of pixels, each of which captures an image simultaneously, allowing the entire image to be captured in one go. Area scan cameras offer higher resolution and sensitivity, especially in low-light conditions.

[0198] In some embodiments, binning (pixel merging) can be used to optimize basic camera algorithms to fuse multiple real pixels from area array cameras, such as CCD cameras, to achieve clear images with very short exposure times. This is an effective high-speed image capture method. Pixel merging combines pixels in an image into "super pixels" or "virtual pixels." For example, 2×2 or 3×3 pixels can be combined into a single virtual pixel to enhance the image. This allows the camera to capture images at very low exposure times without underexposure, thus avoiding the blur and smearing caused by long exposure times.

[0199] In CCD cameras, the pixel binning process is performed in hardware, or "on-chip." The resulting superpixels have higher signal intensity, which helps produce sharper images with very short exposure times. Because of the shortened exposure time, the camera can capture images more quickly and continuously, enabling high-speed image capture.

[0200] For example, the area array camera uses binning technology to achieve pixel fusion, thereby improving the initial brightness of the image at the hardware level, ensuring that the camera can capture high-speed moving mask fabric within a shorter exposure time and avoiding motion blurred images.

[0201] In an exemplary embodiment, the image acquisition component further includes: a bar-shaped light source, located on one side of the image capture device, and irradiating the object to be inspected moving on the production line at an oblique angle at a predetermined angle.

[0202] A bar light source is a special light source with a narrow and long shape, typically using LEDs or CCFLs as its light source. The brightness of a bar light source is more uniform than that of ordinary light sources, making it ideal for lighting or display applications, capable of providing more uniform illumination within a certain space. Bar light sources can be customized according to specific usage scenarios, including adjusting parameters such as length, brightness, and color to meet different lighting needs and achieve the optimal lighting effect. Bar light sources have high durability and a long lifespan, maintaining stable light output for an extended period of time. Bar light sources can not only adjust brightness and color, but also change the shape and direction of the light beam to meet the needs of various application scenarios. Bar light sources use LED technology, which is highly efficient and energy-saving, making them more energy-efficient than traditional fluorescent or incandescent lamps. They also have better environmental performance, helping to reduce energy consumption and environmental pollution.

[0203] It should be noted that in the defect detection system of the embodiment of the present disclosure, the camera may also have its own light source. In other embodiments, other light sources may also be used, such as a surface light source, an annular light source, etc.

[0204] In an exemplary embodiment, the predetermined angle is 45 degrees.

[0205] For example, a bar light source is positioned to one side of the camera, illuminating the mask fabric moving along the production line at a 45-degree angle. This specific lighting angle maximizes the visual contrast of defects, making them more prominent in the image captured by the camera. First, the oblique illumination creates shadows and reflections, which help highlight surface irregularities or imperfections on the mask fabric. When light strikes the fabric at an angle, imperfections may reflect or absorb differently due to their shape, texture, or depth. This makes it easier for the camera to distinguish defects from normal areas when capturing the image. The 45-degree angle ensures that the light fully covers the fabric surface while avoiding excessively direct or oblique light, which would result in heavy or light shadows. Appropriate shadows and reflections enhance the contrast of defects and make them stand out more clearly in the image. The narrow shape of the bar light source allows the light to be concentrated in a narrow area, improving the uniformity and consistency of the illumination. This is particularly important when inspecting mask fabric moving along the production line, as the movement of the fabric can cause varying lighting conditions. The stability and consistency of the bar light helps ensure consistent and reliable image quality captured by the camera. By properly setting the position and angle of the bar light, the visual contrast of defects can be maximized, improving the quality and accuracy of the image captured by the camera, thereby achieving effective detection of mask fabric defects.

[0206] It should be noted that the above 45 degrees does not require strict accuracy and can have a certain error range, for example, greater than ±3 degrees.

[0207] In an exemplary embodiment, the system includes an industrial computer component, the industrial computer component includes an industrial computer, and the defect detection device is deployed on the industrial computer.

[0208] In an exemplary embodiment, a data service component is further deployed on the industrial computer, and the data service component is used to record and collect statistics on defect information detected during the production process.

[0209] In some embodiments, the data service component manages all defect data (defect information, such as whether there is a defect in the image, the defect category, location, category uncertainty and location uncertainty, defect level, and other any data related to the defect) through MySQL.

[0210] In other embodiments, the data service component can upload detected defect information to a cloud server in real time, which enables centralized data management and facilitates remote monitoring, analysis, and optimization of production processes.

[0211] In an exemplary embodiment, the system provided by the present disclosure further includes an alarm component connected to the defect detection device, configured to receive a trigger signal sent by the defect detection device when the defect detection device detects a defect, and to transmit an alarm message based on the trigger signal. For example, the alarm component may include an alarm, but the present disclosure is not limited thereto.

[0212] In an exemplary embodiment, the system provided by the embodiment of the present disclosure further includes: a display connected to the defect detection device, for displaying defect information detected by the defect detection device.

[0213] The system provided by the embodiment of the present disclosure is described below with reference to FIG. 13 to FIG. 15 , but the present disclosure is not limited thereto.

[0214] As shown in FIG13 , defect detection system 120 may include an image acquisition component 121 and a defect detection device 122. For example, defect detection device 122 may be housed in a high-performance industrial computer component. Defect detection device 122 may include an image processing module 1221. Image acquisition component 121 includes an image capture device 1211 (e.g., an area array camera) and a light source 1212 (e.g., a bar light source). Image capture device 1211 is connected to the industrial computer component, which includes an industrial computer.

[0215] Optionally, the defect detection system 120 may further include an alarm component 123 , and the alarm component 123 may be connected to the industrial computer component.

[0216] For example, a high-speed area scan camera is connected to a high-performance industrial computer, transmitting captured images to the computer in real time. Alarm component 123 is also connected to the high-performance industrial computer. When the computer detects a flaw or defect in the fabric image or mask fabric image, it sends a trigger signal to the alarm, which then emits an audible and visual alarm signal, prompting workers to address the defective masks. The algorithm recognition component / algorithm layer and data service component both run on the high-performance industrial computer.

[0217] Figure 14 shows an exemplary scenario of a defect detection system for mask production. As shown in Figure 14, the hardware of the defect detection system 120 may include an image acquisition component / system (including a high-speed camera 1211' and a bar light source 1212', responsible for acquiring image information, such as images to be processed), a high-performance industrial control computer (IPC, short for industrial control computer, responsible for processing the acquired image information and generating detection results, such as category detection results and position detection results, as well as category uncertainty and position uncertainty) 122', and optionally, an alarm. A camera link (Camera Link interface) 124 is provided between the high-performance IPC 122' and the high-speed camera 1211'. The high-performance IPC 122' can be connected to the bar light source 1212' via RS232. The high-speed camera 1211' and the bar light source 1212' are located before the mask material cutting machine 125. Here, it is assumed that the mask material is non-woven fabric. The mask material or mask cloth moves along the direction of movement of the mask nonwoven fabric on the cloth guide roller 126 toward the cutting machine 125. The angle between the light emitted by the strip light source 1212' toward the mask cloth on the cloth guide roller 126 and the mask cloth can be, for example, 45 degrees.

[0218] Referring to Figure 14, the mask fabric / material image captured by camera 1211' (e.g., a camera) is processed by the IPC to identify defects / blemishes. If a defect is identified in the image, the IPC sends an audible and visual signal through an alarm to notify workers on the assembly line to reject the defective product.

[0219] As can be seen from Figure 14, a high-speed camera 1211' and a light source (e.g., a bar light source 1212') are installed in a position before the mask material cutting machine 125. Since the space in the detection environment is very limited and the speed of moving the mask material on the production line is very fast, the camera needs to capture images in a short time (e.g., at least 25 frames per second). However, when the camera captures images at such a high speed, the image becomes blurred and distorted due to the limitation of the exposure time. Therefore, the embodiment of the present disclosure uses binning to fuse 2*2 pixels into a virtual pixel to enhance the image, enabling the camera to shoot at a very low exposure time without underexposure, thereby avoiding blur and smearing caused by long exposure.

[0220] The detection system proposed in the disclosed embodiments overcomes the limitations of related technologies, which rely too heavily on mechanical structures and lack sufficient description of visual recognition methods. The system utilizes advanced image capture equipment, intelligent recognition technology combined with deep learning algorithms, and efficient data service support to achieve real-time and accurate detection of defects in fabrics used in high-speed sports masks.

[0221] As shown in Figure 15, the defect detection system (also referred to as an intelligent defect detection system) provided by the embodiment of the present disclosure includes hardware, an algorithm layer / component (also referred to as a detection algorithm layer) and a data service layer / component. The hardware components of the system include an industrial computer, a high-speed camera (high-speed camera), a light source and an alarm, which work together to capture images and issue warnings when defects are detected. The algorithm layer includes technologies such as preprocessing, channel pruning, depthwise separable convolution and Bayesian optimization, which effectively improve the detection speed and accuracy. The data service component includes image and data storage, defect data analysis, defect data reporting and automated defect labeling. The detection algorithm is the core part of the detection system, and the hardware and data services make the defect detection system faster, more accurate and easier to use.

[0222] The camera may be, for example, a CIS camera (CMOS Image Sensor, complementary metal oxide semiconductor image sensor), a CCD camera, or the like.

[0223] For example, a camera and light source are responsible for capturing image information from the mask fabric surface. An industrial computer processes the captured image and identifies surface defects. An alarm prompts workers to dispose of defective masks that don't meet quality standards. The camera captures the mask fabric image, which is then processed by the industrial computer for defect detection. If a defect is detected, the industrial computer sends an audible and visual signal through the alarm, notifying workers on the production line to remove the defective product.

[0224] The algorithm layer is the core of the system's intelligence, responsible for defect location and classification. It includes image preprocessing, channel pruning, depthwise separable convolution, and Bayesian optimization modules. Preprocessing preprocesses images before defect detection, enhancing the distinction between defects and background through image enhancement and filtering, thereby improving defect identification. Channel pruning, based on the need for lightweight model design, involves pruning the model's channels. The L2-norm is used to measure the contribution of convolutional channels to the network output, and channels are pruned based on this contribution to reduce model parameters and improve processing speed. Depthwise separable convolution is also a lightweight approach, simplifying the convolution operation through Deepwise Conv and Pointwise Conv. Compared to standard convolution, depthwise separable convolution can reduce computational parameters by 85% while maintaining accuracy. Bayesian optimization applies Bayesian methods to optimize the model. First, a dropout layer is added to the model to create randomness and simulate the Bayesian posterior distribution, enhancing the accuracy and reliability of defect detection. Bayesian methods can quantify the uncertainty of predictions. In a detection system, the model not only predicts outcomes but also provides information about how confident it is in those predictions. This quantification of uncertainty allows the system to process information more intelligently, such as taking a more cautious approach to predictions about which the model is less confident.

[0225] In this embodiment, the detection algorithm can predict four main types of mask material defects: stickers (sticker refers to the connection between each roll of mask fabric, which is attached together on the production line with adhesive tape, hence the term "tape"). Spots (stains), printing issues (some masks have patterns and prints, which may have printing errors), and creases in the mask fabric. During online operation, the entire system is end-to-end. It takes the acquired image as input. The model first removes irrelevant background (i.e., areas outside the mask fabric captured in the image). Efficient image feature extraction is achieved through channel pruning and depthwise separable convolution. Bayesian methods for stochastic and epistemic uncertainty are then incorporated into the detection architecture. Stochastic uncertainty arises from the randomness of the training data itself. In the context of mask fabric defect detection, this may include uncertainty caused by natural variation in the mask fabric, subtle differences in the production process, or noise in the image acquisition process. Stochastic uncertainty is generally irreducible because it reflects the inherent variation in the data itself. Epistemic uncertainty arises from uncertainty in model parameters and is the uncertainty in the model's understanding and inference of the data. In mask fabric defect detection, epistemic uncertainty can stem from the model's uncertainty in identifying specific types of defects or its adaptability to new materials and types of defects. The Bayesian approach involves adding a dropout layer to the backbone's intermediate feature layer to simulate the Bayesian posterior distribution, obtaining different outputs about the features. The distribution of these outputs provides an approximation of predictive uncertainty, where the variability of the predictions reflects the model's uncertainty. The detection head is modified to add the mean and variance of each parameter to the model output, including the bounding box coordinates, category, and confidence level, to represent a measure of predictive uncertainty. The loss function is adjusted to include a negative log-likelihood loss to penalize excessive uncertainty and inaccurate predictions. Specifically, a new negative log-likelihood loss function has been added to the detection head to calculate the Bayesian probability loss. This serves two purposes: first, to penalize excessive uncertainty (because if the model is very uncertain about its predictions, the loss will increase), and second, to penalize inaccurate predictions (large prediction errors increase the loss).

[0226] The disclosed embodiments optimize the defect category, location and uncertainty prediction process of the mask fabric defect detection system. The output of the improved detection head is a probability distribution, and a new Bayesian loss, such as a negative log-likelihood loss, is added to increase the penalty for uncertain results, thereby optimizing back propagation. Dropout is added to the backbone intermediate feature layer to simulate the uncertainty of the model parameters and realize the Bayesian posterior distribution. Data uncertainty is determined by data noise, annotation variance, and sample quality (such as lighting, resolution, and contrast). Through the above improvements, the model is able to estimate the uncertainty of its predictions, thereby improving the robustness and reliability of the model. The implementation of these innovations is important for improving the accuracy and efficiency of the mask fabric defect detection system, especially in a high-speed production environment. Through the combination of these technologies, effective detection of high-speed motion mask fabrics can be achieved, while improving the robustness and reliability of the system.

[0227] The data service layer is the information processing and decision support center in the mask fabric defect detection system. It is responsible for integrating the output of the algorithm layer and performing subsequent data management and decision-making. The data service layer includes defect and image data storage, defect data aggregation, defect data transmission, and automatic decision-making modules. The image and defect data storage is responsible for collecting and storing all raw image data obtained through the image acquisition system and the result data processed by the algorithm layer (including one or more of the location of the defect, the defect type of the defect, the uncertainty of each defect, the image characteristics of the defect area, etc.). Through efficient data organization and indexing mechanisms, it ensures the queryability and accessibility of the data. Among them, the defect data analysis will unify and analyze the various types of detected defect data, and summarize the scattered defect information into intuitive reports to facilitate the monitoring and evaluation of production quality. Defect data transmission ensures that the detected defect information can be quickly and accurately transmitted to the downstream links of the production line or the quality control department, providing support for real-time response and defect processing. Automatic decision-making is based on aggregated analysis of defect data. Combined with the actual situation of the production line, this module uses intelligent decision-making algorithms to analyze the distribution and uncertainty probability of defect detection and generate reports to help producers understand which types of defects are most common in the current mask production, the frequency of serious defects, etc., allowing producers to understand the defects of the production process from a more macro perspective and optimize the production process.

[0228] The data service layer is responsible for storing, analyzing, and transmitting inspection data, and implementing appropriate treatment strategies based on the severity of defects. Furthermore, the data service layer enhances the efficiency of manual inspections by visually displaying the defect level / grade through the thickness of the defect border and providing a search function for defect / flaw information, supporting quality tracking and production process optimization.

[0229] The present disclosure relates to a system for detecting defects in mask fabrics in high-speed production environments. This system integrates high-performance image capture hardware (cameras and light sources), advanced deep learning algorithms (including the deep learning defect detection model described in the above embodiments), and a comprehensive data service layer to enable real-time defect detection and classification of high-speed moving mask fabrics. Overall, this disclosure provides an efficient and reliable solution to meet the demand for high-quality masks on fast production lines.

[0230] The mask cloth defect detection system provided by the embodiment of the present disclosure, by providing an image acquisition component, an algorithm layer, and a data service component, can perform efficient and accurate defect detection on the mask cloth passing through without affecting the speed of the mask cloth production line, and automatically mark the defects when defects are detected, thereby greatly improving production efficiency and product quality. Through the combination of manual and automated detection, the system also ensures a high level of detection reliability. For example, in combination with the Bayesian method, an uncertainty score will be given for each predicted defect point (for example, referring to the above formula (2)). If the uncertainty score of a certain defect point is high, the defect point will be sent to manual marking.

[0231] "High speed" in the disclosed embodiments means meeting the global shutter requirement, with a maximum frame rate exceeding 60 fps, for example. For example, high speed means capturing mask fabric on an assembly line at a production rate of 500 masks per minute without motion blur and without losing information about the mask fabric area during the camera frame sampling interval.

[0232] According to one embodiment of the present disclosure, a system for detecting defects in mask fabric is provided. The system comprises: an image acquisition component comprising a high-speed area array camera, a strip light source, and a camera and light source bracket. The high-speed area array camera is used to capture images of the mask fabric surface, and the light source is used to highlight the visual differences between defects and the background. A high-performance industrial computer component comprises an industrial computer equipped with a high-performance GPU and a display. The industrial computer is used to detect defects in images captured by the camera in real time. The display is installed at the end of the mask production line to facilitate workers to confirm image information of mask defects. An alarm component is installed at the end of the mask production line. When a defect is detected, an alarm is triggered to prompt workers to take timely action. A defect recognition algorithm component analyzes defect characteristics and image characteristics to improve the model and optimize its calculation speed and accuracy. The image characteristics may include one or more of changes in the background in the image, changes in the surface texture of the object being detected, and interference of the surrounding ambient light on the image. Processing related to image characteristics involves adjusting camera exposure and removing image background.

[0233] In the disclosed embodiments, defect characteristics refer to the anchor box distribution characteristics of defects and the Euclidean distance between defect feature classes. For example, the anchor box distribution characteristics and the Euclidean distance between defect feature classes are analyzed. Clustering is performed based on the anchor box distribution characteristics to generate the highest priority anchor boxes. These boxes are then sorted and compared based on the Euclidean distance between defect feature classes to optimize defect classification accuracy.

[0234] In the embodiments of the present disclosure, there are two ways to analyze defect characteristics:

[0235] One method is to generate the best prior anchor frame: for example, cluster the labeled defect anchor frames through k-means to find the best prior anchor frame, and then analyze the length-width ratio distribution of the defects.

[0236] The other is hard sample mining and optimization: First, based on the distribution of anchor boxes, find defects of different proportions. For example, for defects with an aspect ratio of 1:1, for defect A, calculate the Euclidean distance between A and other types of defects with an aspect ratio of 1:1. Then find the closest defect of a different type and mark it as N (Negative), and find the farthest defect of the same type and mark it as P (Positive). These three defect samples then constitute a hard sample triplet. By constructing multiple hard sample triplets of defects with different aspect ratios, a hard sample dataset is finally constructed. The model is then deeply trained on the hard sample dataset to make it more suitable for this hard sample, thereby achieving better detection results. In other words, the most difficult defects to distinguish are selected for targeted optimization.

[0237] The advantages of a bar light source are its high power and intensity. Because cameras need to capture highly moving objects, exposure times cannot be too long, so a strong enough light source is needed to provide sufficient brightness. The bar light source should be selected with a width and length sufficient to cover the area the camera is capturing.

[0238] The data service component records and counts all defects detected on the assembly line, updates the data to the database in real time, and performs data analysis to help workers analyze the causes of defects and thus optimize the process.

[0239] The present disclosure aims to provide a mask fabric defect detection system that aims to, at least to some extent, address existing issues with mask fabric defect detection, such as insufficient detection accuracy. By incorporating advanced image acquisition and processing technologies, this system significantly improves the accuracy and efficiency of defect identification, thereby improving the process flow in related technical fields.

[0240] The disclosed embodiments provide a complete online high-speed defect detection system, which is a systematic improvement and innovation for a specific business scenario (mask cloth moving at high speed on a production line). The system lightweights the model as a whole, introduces Bayesian estimation into the model, enhances training samples through AIGC technology, solves the hard sample problem through quantitative learning to enable better model convergence, improves image acquisition quality through binning, designs a camera automatic exposure algorithm, and designs human-computer interaction logic to optimize the defect decision-making process.

[0241] In an exemplary embodiment, the present disclosure also provides a defect detection system, comprising: an image capture device for capturing the image to be processed of the object to be inspected in the production process; a processor and a memory for storing a computer program; when the processor executes the computer program, the steps of the method described in any of the above embodiments are implemented.

[0242] In an exemplary embodiment, the memory includes a memory for storing a defect detection program (implementing the functions of the above-mentioned algorithm layer) and a memory for storing a service data program (implementing the functions of the above-mentioned data service layer).

[0243] In the above description, the intelligent defect detection system and method are listed as preferred examples. Those skilled in the art will appreciate that modifications, including additions and / or substitutions, may be made without departing from the scope and spirit of the present invention. Specific details may be omitted so as not to obscure the present invention; however, the purpose of this disclosure is to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0244] FIG16 is a schematic diagram of the structure of a defect detection device according to an exemplary embodiment of the present disclosure. As shown in FIG16 , the defect detection device 1500 provided by the embodiment of the present disclosure includes an image acquisition module 1510 and an image processing module 1520 .

[0245] The image acquisition module 1510 can be used to obtain an image to be processed.

[0246] The image processing module 1520 can be used to process the image to be processed through the feature extraction module in the deep learning defect detection model. The feature extraction module includes an intermediate feature layer, and the intermediate feature layer includes a discarding layer. The discarding layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding processing after being processed by the feature extraction module.

[0247] The image processing module can also be used to process multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of defects in the image to be processed.

[0248] The image processing module may be further configured to obtain a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the defect in the image to be processed based on the multiple category prediction results and the multiple position prediction results of the defect.

[0249] In which, the image processing module 1520 is also used to: obtain the first category detection result, first position detection result, first category uncertainty and first position uncertainty of the first defect based on multiple first category prediction results and multiple first position prediction results of the first defect; obtain the second category detection result, second position detection result, second category uncertainty and second position uncertainty of the second defect based on multiple second category prediction results and multiple second position prediction results of the second defect.

[0250] In an exemplary embodiment, the image acquisition module 1510 is further configured to obtain the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is configured to obtain the image to be processed of the object to be inspected during a production process.

[0251] The defect detection apparatus 1500 may further include a speed acquisition module configured to obtain the current production speed of the object to be inspected and the current processing speed of the deep learning defect detection model for the image to be processed. The image processing module 1520 may further be configured to adjust the number of parallel threads or processes used by the deep learning defect detection model to process the image to be processed based on the current production speed and the current processing speed.

[0252] In an exemplary embodiment, the image acquisition module 1510 is also used to: obtain the image to be processed from the image capturing device in the image acquisition component; if the grayscale value and grayscale distribution of the image to be processed do not meet the preset conditions, send a control signal to adjust the exposure parameters of the image capturing device; obtain a new image to be processed from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset conditions is obtained.

[0253] In an exemplary embodiment, the defect detection apparatus 1500 may further include: a sending module, configured to send prompt information regarding the defect if the category uncertainty and / or location uncertainty of the defect is greater than an uncertainty threshold.

[0254] In an exemplary embodiment, the defect detection device 1500 may further include: a data processing module for counting pixel number information of each defect detected in the image to be processed; determining the defect level of each defect based on the pixel number information of each defect; and performing corresponding processing based on the defect level of each defect.

[0255] In an exemplary embodiment, the data processing module may also be configured to: in the image to be processed, mark defects at different defect levels with frames having different thicknesses.

[0256] In an exemplary embodiment, the defect detection device 1500 may also include: a data processing module, which can be used to save the image to be processed and the category detection results, position detection results, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection serial number of the defects detected in the image to be processed; receive and retrieve corresponding defect information based on one or more of the input image features, detection time and detection serial number of the defect area.

[0257] In an exemplary embodiment, the defect detection device 1500 may also include: a model training module, which can be used to obtain a target training data set, wherein the target training data set includes a target sample image and the labeled category and labeled position of the target sample defect in the target sample image; inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defect; obtaining the category detection result, position detection result, category uncertainty and position uncertainty of the target sample defect based on the multiple category prediction results and multiple position prediction results of the target sample defect; constructing a loss function based on the labeled category and category detection result, labeled position and position detection result, and category uncertainty and position uncertainty of the target sample defect to train the deep learning defect detection model.

[0258] In an exemplary embodiment, the model training module may also be used to: obtain an initial sample image and the annotated categories and locations of initial sample defects in the initial sample image; and perform any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an expanded sample image and the annotated categories and locations of expanded sample defects in the expanded gain image. The target sample image includes the initial sample image and the expanded sample image.

[0259] In an exemplary embodiment, the model training module can also be used to: extract initial sample defects from the initial sample image; synthesize the initial sample defects with the amplified background image to generate the amplified sample image; and / or, learn the initial sample image through a deep learning generation model to generate the amplified sample image.

[0260] In an exemplary embodiment, the model training module can also be used to: obtain candidate sample images and the labeled categories and labeled positions of candidate sample defects in the candidate sample images; obtain the output features of the candidate sample images through the feature extraction module; obtain the feature distance between each candidate sample image based on the output features of each candidate sample image; select candidate sample images with a feature distance greater than a first distance threshold among candidate sample images with the same labeled category, and / or candidate sample images with a feature distance less than a second distance threshold among candidate sample images with different labeled categories as target sample images in the target training data set.

[0261] In an exemplary embodiment, the model training module can also be used in the training process of the deep learning defect detection model to: extract the weight tensor of each convolutional layer in the deep learning defect detection model; obtain the Euclidean norm of each channel in the weight tensor; sort each convolutional layer according to the Euclidean norm of its channel, and prune the corresponding channel according to the pruning threshold or pruning ratio.

[0262] For other contents of the embodiment of FIG16 , reference may be made to the other embodiments described above.

[0263] Figure 17 schematically shows a schematic structural diagram of an electronic device 1600 according to an embodiment of the present disclosure. The electronic device 1600 shown in Figure 17 includes a processor 1610, which can call and run a computer program from a memory to implement the method in the embodiment of the present disclosure.

[0264] Optionally, as shown in FIG17 , the electronic device 1600 may further include a memory 1620. The processor 1610 may call and execute a computer program from the memory 1620 to implement the method in the embodiment of the present disclosure.

[0265] The memory 1620 may be a separate device independent of the processor 1610 , or may be integrated into the processor 1610 .

[0266] Optionally, as shown in FIG17 , the electronic device 1600 may further include a transceiver 1630 , and the processor 1610 may control the transceiver 1630 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0267] Optionally, the processor 1610 , the memory 1620 , and the transceiver 1630 may implement bidirectional communication with each other via the communication bus 1640 .

[0268] It should be understood that the processor of the embodiment of the present disclosure may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment may be completed by hardware integrated logic circuits in the processor or software instructions.

[0269] The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in combination with its hardware.

[0270] It is understood that the memory in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be understood that the above-mentioned memory is by way of example and not limitation.

[0271] The embodiments of the present disclosure further provide a computer-readable storage medium for storing a computer program. The computer program enables a computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, no further details are given here. The embodiments of the present disclosure further provide a computer program product, including computer program instructions. The computer program instructions enable a computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, no further details are given here. The embodiments of the present disclosure further provide a computer program. When the computer program is run on a computer, it enables the computer to execute the corresponding processes in the various methods of the embodiments of the present disclosure. For the sake of brevity, no further details are given here.

[0272] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0273] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Industrial Applicability

[0274] The present disclosure is applicable to the field of artificial intelligence technology, and is used to solve the problem in related technologies of inaccurate defect prediction and inability to output category uncertainty and position uncertainty, so as to achieve the effect of quickly and effectively predicting the category and location of defects and evaluating the category uncertainty and position uncertainty of the prediction results.

Claims

1. A defect detection method, comprising: Obtaining an image to be processed; Processing the image to be processed by a feature extraction module in a deep learning defect detection model, wherein the feature extraction module includes an intermediate feature layer, and the intermediate feature layer includes a discarding layer, and the discarding layer is used to randomly discard features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; Processing the multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; According to the multiple category prediction results and the multiple position prediction results of the defect, the category detection result, the position detection result, the category uncertainty and the position uncertainty of the defect in the image to be processed are obtained.

2. The method of claim 1, wherein: The intermediate feature layer includes a first cross-stage convolution block, and the dropout layer includes a first dropout layer; The first cross-stage convolution block includes a first cross-stage input layer, a first convolution layer, a first intermediate layer, a second convolution layer, a first fusion layer, a third convolution layer and the first discard layer; The first convolution layer is used to perform convolution processing on the first cross-stage input feature map input by the first cross-stage input layer, and input the first convolution feature map to the first intermediate layer; The first intermediate layer is used to process the first convolution feature map and input the first feature map to be fused into the first fusion layer; The second convolution layer is used to perform convolution processing on the first cross-stage input feature map, and input the second feature map to be fused into the first fusion layer; The first fusion layer is used to fuse the first feature map to be fused and the second feature map to be fused, and input the first fused feature map to the third convolution layer; The third convolutional layer is used to perform convolution processing on the first fusion feature map, and input the second convolutional feature map to the first discard layer; The first discard layer is used to randomly discard features in the second convolutional feature map to obtain multiple first discarded feature maps. The multiple features obtained after the image to be processed is processed by the feature extraction module and subjected to random discarding processing include multiple first discarded feature maps.

3. The method of claim 2, wherein: The intermediate feature layer further includes a second cross-stage convolution block and a third cross-stage convolution block, and the dropout layer further includes a second dropout layer and a third dropout layer; The input end of the second cross-stage convolution block is connected to the output end of the first cross-stage convolution block to respectively receive a plurality of the first discarded feature maps, process the plurality of the first discarded feature maps, and input the plurality of second discarded feature maps to the third cross-stage convolution block through the second discard layer; The input end of the third cross-stage convolution block is connected to the output end of the second cross-stage convolution block to receive multiple The method further comprises the step of: processing the plurality of second discarded feature maps, and outputting a plurality of third discarded feature maps through the third discarding layer.

4. The method of claim 3, wherein: The deep learning defect detection model also includes a feature pyramid module, and the feature pyramid module includes a first spatial pyramid pooling block, a second spatial pyramid pooling block, and a third spatial pyramid pooling block; The first spatial pyramid pooling block is connected to the first cross-stage convolution block and is used to receive a plurality of the first discarded feature maps; The second spatial pyramid pooling block is connected to the second cross-stage convolution block, and is used to receive a plurality of the second discarded feature maps, and process the plurality of the second discarded feature maps respectively to obtain a plurality of second spatial pooling feature maps; The third spatial pyramid pooling block is connected to the third cross-stage convolution block, and is used to receive the plurality of third discarded feature maps, and process the plurality of third discarded feature maps respectively to obtain a plurality of third spatial pooling feature maps; The first spatial pyramid pooling block is also connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and obtain a plurality of first spatial pooling feature maps according to a plurality of the first discarded feature maps and a plurality of the second spatial pooling feature maps; The second spatial pyramid pooling block is also connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and obtain a plurality of second spatial pooling feature maps according to a plurality of the second discarded feature maps and a plurality of the third spatial pooling feature maps.

5. The method of claim 4, wherein: The detection head includes a first depth-separable convolution block, a second depth-separable convolution block and a third depth-separable convolution block; the defect includes a first defect, a second defect and a third defect; The first depthwise separable convolution block is connected to the first spatial pyramid pooling block, and is used to receive a plurality of the first spatial pooling feature maps, and process the plurality of the first spatial pooling feature maps respectively to obtain a plurality of first category prediction results and a plurality of first position prediction results of the first defect; The second depthwise separable convolution block is connected to the second spatial pyramid pooling block, and is used to receive a plurality of the second spatial pooling feature maps, and process the plurality of the second spatial pooling feature maps respectively to obtain a plurality of second category prediction results and a plurality of second position prediction results of the second defect; The third depthwise separable convolution block is connected to the third spatial pyramid pooling block, and is used to receive a plurality of the third spatial pooling feature maps, and process the plurality of the third spatial pooling feature maps respectively to obtain a plurality of third category prediction results and a plurality of third position prediction results of the third defect; Wherein, obtaining the category detection result, position detection result, category uncertainty and position uncertainty of the defect in the image to be processed according to the multiple category prediction results and the multiple position prediction results of the defect includes: Obtaining a first category detection result, a first position detection result, a first category uncertainty, and a first position uncertainty of the first defect according to a plurality of first category prediction results and a plurality of first position prediction results of the first defect; Obtaining a second category detection result, a second position detection result, a second category uncertainty, and a second position uncertainty of the second defect according to the plurality of second category prediction results and the plurality of second position prediction results of the second defect; According to the plurality of third category prediction results and the plurality of third position prediction results of the third defect, the third A third category detection result, a third position detection result, a third category uncertainty, and a third position uncertainty of the defect.

6. The method of claim 2, wherein: The first intermediate layer includes a first intermediate input layer, a first depth-separable convolutional layer and a first accumulation layer; The first intermediate input layer is used to receive the first convolution feature map output by the first convolution layer, and input the first convolution feature map into the first depthwise separable convolution layer and the first accumulation layer respectively; The first depth-separable convolution layer is used to perform depth-separable convolution processing on the first convolution feature map, and input the first depth-separable convolution feature map into the first accumulation layer; The first accumulation layer performs accumulation processing on the first convolutional feature map and the first depth-separated convolutional feature map to obtain the first feature map to be fused.

7. The method of claim 6, wherein: The first depthwise separable convolutional layer comprises a first depthwise separation input layer, a first depthwise convolutional layer, a first pointwise convolutional layer and a first activation function layer; The first depth separation input layer is used to receive the first convolution feature map output by the first intermediate input layer, and input the first convolution feature map to the first depth convolution layer and the first activation function layer respectively; The first depth convolution layer is used to perform depth convolution processing on the first convolution feature map, and input the first depth convolution feature map to the first point convolution layer; The first point convolution layer is used to perform point convolution processing on the first depth convolution feature map, and input the first point convolution feature map to the first activation function layer; The first activation function layer is used to process the first point convolution feature map and the first convolution feature map to obtain the first depth separation convolution feature map.

8. The method of claim 1, wherein: The feature extraction module also includes an attention block connected to the intermediate feature layer; The attention block is used to process the image to be processed, obtain a downsampled feature map of the image to be processed, and input the downsampled feature map into the intermediate feature layer.

9. The method of claim 1, wherein: The deep learning defect detection model also includes a background removal module; The background removal module is used to remove background information in the image to be processed, and input the image to be processed after the background information is removed into the feature extraction module.

10. The method of claim 1, wherein: Obtain the image to be processed, including: Obtaining the image to be processed from an image capturing device in an image acquisition component, wherein the image acquisition component is used to obtain the image to be processed of the object to be detected in the production process; Wherein, the method further comprises: Obtaining a current production speed of the object to be inspected and a current processing speed of the deep learning defect detection model on the image to be processed; The number of parallel threads or parallel processes used by the deep learning defect detection model to process the image to be processed is adjusted according to the current production speed and the current processing speed.

11. The method of claim 1, wherein: Obtain the image to be processed, including: Obtain the image to be processed from an image capturing device in the image acquisition component; If the grayscale value and grayscale distribution of the image to be processed do not meet the preset conditions, sending a control signal to adjust the exposure parameters of the image capture device; A new image to be processed is obtained from the image capturing device after adjusting the exposure parameters, until the image to be processed that meets the preset condition is obtained.

12. The method of claim 1, wherein: Also includes: If the category uncertainty and / or position uncertainty of the defect is greater than an uncertainty threshold, prompt information for the defect is sent.

13. The method of claim 1, wherein: Also includes: Counting the number of pixels of each defect detected in the image to be processed; Determine a defect level of each defect according to pixel quantity information of each defect; According to the defect level of each defect, corresponding processing is performed.

14. The method of claim 13, wherein: According to the defect level of each defect, corresponding processing is performed, including: In the image to be processed, defects at different defect levels are marked with frames with different thicknesses.

15. The method of claim 1, wherein: Also includes: Saving the image to be processed and the category detection result, position detection result, category uncertainty, position uncertainty, defect level, defect area image, image features of the defect area, detection time and detection sequence number of the defect detected in the image to be processed; Receive and retrieve corresponding defect information based on one or more of the input image features, detection time and detection sequence number of the defect area.

16. The method of claim 1, wherein: Also includes: Obtaining a target training data set, wherein the target training data set includes a target sample image and annotated categories and annotated positions of target sample defects in the target sample image; Inputting the target sample image into the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the target sample defects; Obtaining a category detection result, a position detection result, a category uncertainty, and a position uncertainty of the target sample defect according to a plurality of category prediction results and a plurality of position prediction results of the target sample defect; A loss function is constructed according to the labeled category and category detection result, the labeled position and position detection result of the target sample defect, as well as the category uncertainty and position uncertainty to train the deep learning defect detection model.

17. The method of claim 16, wherein: The loss function includes a first loss function and a second loss function; When the difference between the labeled category and the category detection result of the target sample defect is greater, and the difference between the labeled position and the position detection result is greater, the value of the first loss function is greater; When the category uncertainty and the position uncertainty of the target sample defect are greater, the value of the second loss function is greater.

18. The method of claim 17, wherein: The first loss function L1 is as follows: MSE is the bounding box regression calculation, y i is the marked position of the defect of the i-th target sample, is the predicted position detection result of the i-th target sample defect, N is the number of target sample defects, N is a positive integer greater than or equal to 1, i is a positive integer greater than or equal to 1 and less than or equal to N; CE represents multivariate cross entropy calculation, Represents the category confidence of category c in the predicted category detection result; s c represents the category confidence of category c in the labeled category, C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C; BCE represents binary cross entropy calculation, o pre represents the target confidence for the target sample defect prediction, and o represents the confidence of the target marked in the target sample defect.

19. The method of claim 17, wherein: The second loss function L2 is as follows: μ i is the predicted position detection result of the i-th target sample defect, y i is the marked position of the defect of the i-th target sample; is the predicted position uncertainty of the i-th target sample defect; N is the number of defects in the target sample, N is a positive integer greater than or equal to 1, i is a positive integer greater than or equal to 1 and less than or equal to N; μ c Represents the category detection result of the predicted category c; s c represents the category confidence of category c in the labeled category, represents the category uncertainty of the predicted category c, C represents the number of categories, C is a positive integer greater than or equal to 1, and c is a positive integer greater than or equal to 1 and less than or equal to C.

20. The method of claim 16, wherein: Obtain the target training data set, including: Acquire an initial sample image and a marked category and a marked position of an initial sample defect in the initial sample image; Performing any one or more of random cropping, rotation, scaling, illumination change simulation, and defect synthesis on the initial sample image to generate an expanded sample image and a marked category and a marked position of an expanded sample defect in the expanded gain image; Wherein, the target sample image includes the initial sample image and the amplified sample image.

21. The method of claim 20, wherein: Performing defect synthesis processing on the initial sample image to generate an augmented sample image includes: Extracting initial sample defects from the initial sample image; synthesizing the initial sample defects with the augmented background image to generate the augmented sample image; and / or, The initial sample image is learned by a deep learning generation model to generate the amplified sample image.

22. The method of claim 16, wherein: Obtain the target training data set, including: Obtaining a candidate sample image and a labeled category and a labeled position of a candidate sample defect in the candidate sample image; Obtaining output features of the candidate sample image through the feature extraction module; Based on the output features of each candidate sample image, obtaining the feature distance between each candidate sample image; Select candidate sample images whose feature distance is greater than a first distance threshold among candidate sample images with the same annotated category, and / or select candidate sample images whose feature distance is less than a second distance threshold among candidate sample images with different annotated categories as target sample images in the target training data set.

23. The method of claim 16, wherein: During the training process of the deep learning defect detection model: Extracting a weight tensor for each convolutional layer in the deep learning defect detection model; Obtaining the Euclidean norm of each channel in the weight tensor; Each convolutional layer is sorted according to the Euclidean norm of its channel, and the corresponding channel is pruned according to the pruning threshold or pruning ratio.

24. The method of claim 16, wherein: The target sample images include images of masks with different materials, textures, colors, and print patterns.

25. A defect detection device comprising: An image acquisition module, used for acquiring an image to be processed; An image processing module, used for processing the image to be processed by a feature extraction module in a deep learning defect detection model, wherein the feature extraction module includes an intermediate feature layer, wherein the intermediate feature layer includes a discarding layer, and the discarding layer is used for randomly discarding features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains a plurality of features after random discarding processing after being processed by the feature extraction module; The image processing module is also used to process the multiple features after random discarding through the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; The image processing module is also used to obtain a category detection result, a position detection result, a category uncertainty and a position uncertainty of the defect in the image to be processed according to the multiple category prediction results and the multiple position prediction results of the defect.

26. A defect detection system comprising: An image acquisition component, used for acquiring the image to be processed of the object to be detected in the production process; A defect detection device, connected to the image acquisition component, for obtaining the image to be processed from the image acquisition component; wherein the defect detection device comprises: An image processing module is used to process the image to be processed through a feature extraction module in a deep learning defect detection model, the feature extraction module includes an intermediate feature layer, the intermediate feature layer includes a discard layer, and the discard layer is used to randomly discard the features extracted after the intermediate feature layer processes the image to be processed, so that the image to be processed obtains multiple features after random discarding after being processed by the feature extraction module; the multiple features after random discarding are processed by the detection head in the deep learning defect detection model to obtain multiple category prediction results and multiple position prediction results of the defects in the image to be processed; according to the multiple category prediction results and multiple position prediction results of the defects, the category detection results, position detection results, category uncertainty and position uncertainty of the defects in the image to be processed are obtained.

27. The system of claim 26, wherein: The image acquisition component comprises: An image capturing device, the image capturing device comprising an area array camera; The area array camera is used to collect the original image to be processed, and perform pixel merging processing on the original image to be processed to generate the image to be processed.

28. The system of claim 27, wherein: The image acquisition component also includes: The strip light source is located at one side of the image capturing device and illuminates the object to be detected moving on the production line at a predetermined angle.

29. The system of claim 28, wherein: The predetermined angle is 45 degrees.

30. The system of claim 26, wherein: The system includes an industrial computer component, the industrial computer component includes an industrial computer, and the defect detection device is deployed on the industrial computer.

31. The system of claim 30, wherein: A data service component is also deployed on the industrial computer; The data service component is used to record and count the defect information detected during the production process.

32. The system of claim 26, wherein: Also includes: The alarm component is connected to the defect detection device and is used to receive a trigger signal sent by the defect detection device when the defect detection device detects a defect, and send alarm information according to the trigger signal.

33. The system of claim 26, wherein: Also includes: A display is connected to the defect detection device and is used to display the defect information detected by the defect detection device.

34. A defect detection system comprising: An image capturing device, used for collecting the image to be processed of the object to be inspected in the production process; a processor and a memory for storing a computer program; When the processor executes the computer program, the steps in the method according to any one of claims 1 to 24 are implemented.

35. The system of claim 34, wherein: The memory includes a memory for storing a defect detection program and a memory for storing a service data program.

36. An electronic device comprising: one or more processors; The memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the method according to any one of claims 1 to 24.

37. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 24.

38. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 24.

Images