Two-dimensional and three-dimensional fused fabric defect detection method

Abstract

The invention discloses a two-dimensional and three-dimensional fused fabric defect detection method, and belongs to the technical field of machine vision recognition. The technical problem of limitation of a traditional two-dimensional machine vision system is solved. According to the technical scheme, the method is characterized in that multi-angle light source imaging is carried out, a detected fabric is polished through light sources at different angles, and two-dimensional images of illumination at different angles are obtained through shooting of a linear array camera; and two-dimensional and three-dimensional information fusion: carrying out pixel-level fusion on the shot two-dimensional grey-scale map and the reconstructed gradient map and depth map with the three-dimensional information, carrying out high and low frequency decomposition and fusion on the two-dimensional information and the three-dimensional information through non-subsampled contourlet transformation, and fusing the two-dimensional defect feature information and the three-dimensional defect feature information into one map. After the non-subsampled contourlet transform decomposes the image, the frequency domain space is divided into wedge-shaped directional sub-bands; defects are identified based on the deep learning model. The method has the effect of improving the accuracy of textile quality detection.

Description

Technical Field

[0001] The present application relates to the technical field of machine vision recognition, and in particular to a two-dimensional and three-dimensional fusion fabric defect detection method. Background Art

[0002] In daily life, textile products play an important role in modern society and are also one of the key industries of the country's light industry. However, in the process of fabric manufacturing, surface defects or flaws are common, which directly affects the production efficiency and quality of the entire project. Therefore, it is extremely important to conduct effective quality inspection of fabric products to prevent defective products from being mixed into finished products.

[0003] In traditional manufacturing processes, quality inspection mainly relies on manual visual inspection, which is inefficient and susceptible to operator fatigue and inconsistent subjective judgment standards. With the development of technology, machine vision systems have begun to be used in textile quality inspection, mainly through two-dimensional image recognition to detect defects on the surface of the fabric. This method improves the speed and repeatability of inspection, but there are still limitations in two-dimensional image processing, especially when it comes to identifying defects in depth or subtle textures. In addition, due to the influence of light, camera angle and the characteristics of the fabric itself, the two-dimensional inspection system may also have problems with false detection and missed detection. Therefore, the textile industry is in urgent need of a more advanced technical solution to overcome the limitations of traditional two-dimensional machine vision systems and improve the accuracy of textile quality inspection. Summary of the invention

[0004] In view of the shortcomings (problems) of the prior art mentioned above, a two-dimensional and three-dimensional fused fabric defect detection method is provided. By acquiring the two-dimensional image and three-dimensional image information of the fabric surface, the two-dimensional image and the three-dimensional image are fused so that different defect features can be displayed in the same image. Combined with deep learning technology, defects are identified, thereby improving the detection rate of defects.

[0005] To achieve the above objectives and other related objectives, this application adopts the following technical solutions:

[0006] A two-dimensional and three-dimensional fusion fabric defect detection method comprises the following steps:

[0007] Step S1, multi-angle light source imaging, lighting the fabric under inspection through light sources at different angles, photographing through a line array camera, and obtaining two-dimensional images of different angles of illumination;

[0008] Step S2, fusion of two-dimensional and three-dimensional information, pixel-level fusion of the captured two-dimensional grayscale image and the reconstructed gradient image and depth image with three-dimensional information, high-low frequency decomposition and fusion of the two-dimensional information and the three-dimensional information by non-subsampled contourlet transform, fusion of the two-dimensional defect feature information and the three-dimensional defect feature information into one image, after the non-subsampled contourlet transform decomposes the image, the frequency domain space is divided into wedge-shaped directional sub-bands;

[0009] Step S3, identifying defects based on the deep learning model.

[0010] Preferably, step S1 includes the following steps:

[0011] S11: First, place the red and green light sources symmetrically, keep the incident light and the camera light plane collinear, and make the incident angles of the two light sources relative to the camera the same;

[0012] S12: placing the sample to be tested on the translation stage and fixing it to ensure that the relative position of the sample and the translation stage does not deviate when the sample is driven by the translation stage;

[0013] S13: The moving translation stage carries the sample to be tested to a preset position, and at the same time, the encoder triggers a signal to the linear array camera, and the linear array camera collects the fabric image illuminated by the red and green light sources at the same time. By separating the red and green channels of the camera, images illuminated by different light sources can be obtained;

[0014] S14: Photometric stereo is used to reconstruct the three-dimensional information of the sample under test. Two symmetrical red and green line light sources are used. The incident angle α of the light source can be obtained by calibrating the reflection model. Then the direction vector of the light source can be obtained as l1=(0, sinα, cosα), l2=(0, -sinα, cosα). The y-direction gradient q of the sample surface is obtained. According to the model: I=ρEL·n, where I is the image brightness; E is the incident light intensity; n is the unit normal vector; L is the unit direction vector of the light source; ρ is the reflectivity. Calculate:

[0015] S15: Based on the gradient q reconstructed by the photometric stereo method, the surface gradient is integrated using a global iterative optimization method to restore the surface depth information, so that the iterative error is within a preset range, and the three-dimensional depth information z of the sample surface can be obtained;

[0016] S16: Normalize the gradient q and the depth z to [0, 255] to obtain the gradient map q and the depth map z.

[0017] Preferably, step S2 includes the following steps:

[0018] S21: q is a gradient image normalized to [0,255]. The gradient image is decomposed into two scales based on rolling guided filtering to obtain the rolling guided filter image qRGF And Gaussian filter filter graph q GF , transformed into the logarithmic domain to obtain the base layer q b and detail layer q d , image enhancement is performed in the logarithmic domain, and the enhanced gradient map q can be obtained by converting the logarithmic domain into the spatial domain;

[0019] S22: The grayscale image I, gradient image q, and depth image z are decomposed at multiple scales using a non-subsampled pyramid, and multi-directionally decomposed using a non-subsampled directional filter group. The number of decomposition layers is 4, and the corresponding low-frequency subband image I1 is decomposed. 0 ,q1 0 , z1 0 , and the corresponding high-frequency subband image I l k ,q l k , z l k , l represents the number of decomposition layers, and k represents the number of high-frequency subbands decomposed at layer l;

[0020] S23: Fuse the low-frequency sub-bands of the grayscale image, gradient image, and depth image. The fusion rule adopts the absolute maximum principle to preserve the energy of the low-frequency sub-band as much as possible. The low-frequency sub-band mainly contains most of the energy and a small amount of details of the original image, where energy is defined as the sum of the squares of the pixel intensities in the image.

[0021] S24: The high-frequency subband mainly contains the edge and texture details of the image, and the high-frequency subband coefficient with strong edge preservation ability is selected;

[0022] S25: performing an inverse transformation of the non-subsampled contourlet transform on the fused low-frequency and high-frequency sub-bands to obtain a fused image.

[0023] Preferably, step S3 also includes the following steps:

[0024] S31: Dataset preparation: crop the fusion image into 8 low-resolution images of 2048 pixel × 2048 pixel size, use YOLOv5 automatic enhancement, use LabelImg annotation, follow the VOC2007 dataset format, and use K-means++ algorithm to cluster the prior frame. In the initialization stage, select a target frame with a larger area as the first cluster center, and then use the cosine similarity metric to gradually select the target frame with the smallest similarity to the current cluster center as the new cluster center; then, in the iterative process, select the target frame with the smallest similarity metric to the current cluster center as the new cluster center, so as to generate a prior frame size that is more suitable for the defect size and shape.

[0025] Preferably, step S3 also includes the following steps:

[0026] S32: Model training, using YOLOv5s as the defect detection model and training. The main structure of YOLOv5s consists of four parts: Input input end, Backbone backbone network, Neck network layer and Head detection end.

[0027] At the input end, batch normalization is used to accelerate model convergence.

[0028] The Backbone network is responsible for extracting features from the image. The Neck layer is mainly responsible for fusing the feature information extracted by the backbone layer and inputting it into the subsequent prediction layer.

[0029] The Head detection head parses the feature maps of different scales output by the Neck network, and trains them through the loss function and NMS non-maximum suppression, and finally obtains the output y1, y2, y3 of feature maps of different scales to achieve multi-scale prediction.

[0030] Preferably, in step S32, y1 is the coarsest-grained feature map with a size of 14×14, which is used to detect large-sized targets; y2 is the medium-grained feature map with a size of 28×28, which is used to detect medium-sized targets; y3 is the finest-grained feature map with a size of 56×56, which is used to detect small-sized targets;

[0031] Each feature map divides the input image into S×S cells, where S is 56, 28, and 14 respectively. These grids help the model locate targets at different size levels. Three prior boxes are set for each grid to predict three bounding boxes. The output dimension is S×S×(3×(4+1+M)), where 3 means that each grid has three prior boxes. 4+1+M are the four coordinates, confidence, and M types of defects of the bounding box respectively. Finally, non-maximum suppression is used to eliminate duplicate bounding boxes to achieve sample defect detection.

[0032] Preferably, the sample to be tested includes leather, wherein the types of defects are various and include at least one of holes, three threads, knots, broken spandex and coarse fibers, the resolution of the input fusion image is 8192 pixels × 4096 pixels, and the minimum detected defect size is required to be 10 pixels × 10 pixels.

[0033] Preferably, for a 8192 pixel×4096 pixel fused image to be detected, it is first cropped into 8 images of 2048 pixel×2048 pixel size, which are simultaneously fed into the model for detection.

[0034] In summary, the present application includes at least one of the following beneficial technical effects:

[0035] By acquiring the 2D and 3D image information of the fabric surface, the 2D and 3D images are fused so that different defect features can be displayed in the same image. In combination with deep learning technology, defects can be identified, thereby improving the defect detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of the testing equipment;

[0037] Figure 2 It is a schematic diagram of photometric stereo;

[0038] Figure 3(a) is a schematic diagram of the decomposition structure of NSCT;

[0039] Figure 3(b) is a diagram of the NSCT decomposition frequency domain partition structure.

[0040] Description of main reference numerals:

[0041] 1. 3CCD linear array camera; 2. Red line light source; 3. Green line light source; 4. Translation stage. DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0044] The specific implementation methods of the present application will be further described below in conjunction with the accompanying drawings.

[0045] Example:

[0046] The present application embodiment discloses a two-dimensional and three-dimensional fusion fabric defect detection method, comprising the following steps:

[0047] Step S1, multi-angle light source imaging, lighting the inspected fabric with light sources at different angles, photographing with a line array camera, and obtaining two-dimensional images with different angles of illumination.

[0048] Step S1 includes the following steps:

[0049] S11: First, place the red and green light sources symmetrically, keep the incident light and the camera light plane collinear, and make the incident angles of the two light sources relative to the camera the same; Figure 1 shown.

[0050] S12: The sample to be tested is placed on the translation stage and fixed to ensure that the relative position of the sample and the translation stage does not deviate when the sample is driven by the translation stage.

[0051] S13: The moving translation stage carries the sample to be tested to a preset position, and at the same time, the encoder triggers a signal to the linear array camera, and the linear array camera collects the fabric image illuminated by the red and green light sources at the same time. By separating the red and green channels of the camera, images illuminated by different light sources can be obtained.

[0052] S14: Photometric stereo is used to reconstruct the three-dimensional information of the sample under test. Two symmetrical red and green line light sources are used. The incident angle α of the light source can be obtained by calibrating the reflection model.

[0053] The incident angle α is obtained by performing reflection model calibration. The following is the detailed calibration process.

[0054] The photometric stereo vision theory is based on the reflection model in photometry. Therefore, the reasonable calibration of the reflection model parameters is of great significance for the design of the imaging scheme, among which the incident angle α of the light source is an important design parameter. This paper adopts a linear LED light source, which is symmetrical in the long axis direction. The reflection model can be simplified to a single degree of freedom function f_bd(λ). The long axis of the light source is perpendicular to the paper surface, α is the incident light angle of the light source, and λ is the inclination angle of the measured surface along the long axis of the light source.

[0055] Since the degrees of freedom irrelevant to the measurement are excluded, it is convenient to use simple experimental equipment to measure the reflection model function, and the reflection model measurement equipment is simplified. The reflection model measurement system is placed horizontally, the optical axis of the camera is parallel to the work surface, and an electric turntable is used to carry the surface sample of the material to be measured. The sample stands vertically on the turntable surface, framing the turntable to produce a rotation of the specified deflection angle λ. The strip LED lighting source is located on one side of the turntable, and the angle between the incident direction (i.e., the line connecting the light-emitting surface and the axis of the turntable) and the camera optical axis is α, and the angle set for the test is 45°. The camera is used for photometric measurement to obtain the functional relationship of the average grayscale of the central area of ​​the image with respect to the deflection angle λ I = f_bd(λ).

[0056] The deflection angles from -30° to 30° were photographed, and a curve of the change in image grayscale and deflection angle was established.

[0057] It can be seen that there is no obvious mutation in the grayscale change curve of the leather, and its reflection model can be approximated as the Lambertian reflection model. The deflection angle is selected in the range of [-10°, 0°], and the incident angle is more suitable in the range of [35°, 45°]. Just select a suitable incident angle in the range of [35°, 45°].

[0058] Then we can get the direction vector of the light source as l1=(0,sinα,cosα),l2=(0,-sinα,cosα), and get the sample surface y-direction gradient q. According to the model: I=ρEL·n, where I is the image brightness; E is the incident light intensity; n is the unit normal vector; L is the unit direction vector of the light source; ρ is the reflectivity. Calculate:

[0059] The traditional photometric stereo method requires three or more light sources in different planes to illuminate the surface of the object to be measured from different angles. The camera collects images with different brightness distributions, and solves the normal gradient information of each spatial point on the surface of the object based on a specific illumination model. The depth information of the object surface is obtained through integral reconstruction, such as Figure 2 shown.

[0060] The premise of the photometric stereo method is that the surface of the object needs to satisfy the Lambertian reflection characteristic, requiring the surface of the object to diffusely reflect the incident light and the intensity of the reflected light in each direction is consistent. The diffuse reflection intensity of this model conforms to the following formula: I = ρEL·n (1),

[0061]

[0062] The gradient q of the leather surface in the y direction can be obtained by dividing the two equations.

[0063] S15: Based on the gradient q reconstructed by the photometric stereo method, the surface gradient is integrated using the global iterative optimization method to restore the surface depth information. The global iterative optimization method solves the height of a certain point by only considering the four adjacent points up, down, left, and right. The four points up, down, left, and right are used as the starting points for integration. The integration is performed along the inner cross path, and the integrated value is used to update the depth matrix. The depth matrix is ​​gradually optimized by multiple iterations so that the final error falls within the allowed range. The algorithm uses the following formula for iteration:

[0064]

[0065] Set the gradient error function to: By making the iterative error within a preset range, the three-dimensional depth information z of the sample surface can be obtained.

[0066] S16: Normalize the gradient q and the depth z to [0, 255] to obtain the gradient map q and the depth map z.

[0067] Step S2, fusion of two-dimensional and three-dimensional information, pixel-level fusion of the captured two-dimensional grayscale image and the reconstructed gradient image and depth image with three-dimensional information, high- and low-frequency decomposition and fusion of the two-dimensional information and the three-dimensional information through NSCT (NonsubsampledContourlet) non-subsampled contourlet transform, and fusion of the two-dimensional defect feature information and the three-dimensional defect feature information into one image. After the non-subsampled contourlet transform decomposes the image, the frequency domain space is divided into wedge-shaped directional sub-bands, refer to Figure 3 for understanding.

[0068] Step S2 includes the following steps: performing image enhancement on the gradient image to improve the contrast of tiny defects.

[0069] S21: q is a gradient image normalized to [0,255]. The gradient image is decomposed into two scales based on rolling guided filtering to obtain the rolling guided filter image q RGF And Gaussian filter filter graph q GF , transformed into the logarithmic domain to obtain the base layer q b and detail layer q d .

[0070] q b =ln(q RGF -q GF +1) (6)

[0071] q d =ln(q+1)-q b (7)

[0072] Image enhancement is performed in the logarithmic domain, and E represents the enhanced image in the logarithmic domain.

[0073] E=λq b +q d +γ (8),

[0074] It can be seen from the above formula that the size of the detail layer does not change, and the detail information of the image can be retained. The present invention defines λ so that λ can achieve dynamic range compression in the base layer. T is defined as the target base contrast, which can be set according to needs.

[0075]

[0076] Define γ to restore the overall contrast and ensure that the output does not exceed the maximum grayscale value intensity of the base layer.

[0077] γ=(1-λ)max(q b ) (10),

[0078] Transforming the logarithmic domain into the spatial domain can obtain the enhanced gradient map q.

[0079] S22: The grayscale image I, gradient image q, and depth image z are decomposed at multiple scales using the non-subsampled pyramid (NSP), and multi-directionally decomposed using the non-subsampled directional filter bank (NSDFB). The number of decomposition layers is 4, and the corresponding low-frequency subband image I1 is decomposed. 0 ,q1 0 , z1 0 , and the corresponding high-frequency subband image I l k ,q l k , z l k , l represents the number of decomposition layers, and k represents the number of high-frequency sub-bands decomposed at layer l.

[0080] S23: Fuse the low-frequency subband coefficients of the grayscale image I, gradient image q, and depth image z respectively. Low-frequency subband fusion: The low-frequency subband coefficients (not the low-frequency subband image) mainly contain most of the energy and a small amount of details of the original image, where energy is defined as the sum of the squares of the pixel intensities in the image. In order to retain the energy of the low-frequency subband, the low-frequency subband coefficient with the largest absolute value of energy is selected as the low-frequency subband fusion coefficient. The fusion rule adopts the absolute maximum principle to retain the energy of the low-frequency subband as much as possible. The low-frequency subband mainly contains most of the energy and a small amount of details of the original image, where energy is defined as the sum of the squares of the pixel intensities in the image. The low-frequency subbands of the grayscale image, gradient image, and depth image are fused, and the fusion rule adopts the absolute maximum principle to retain the energy of the low-frequency subband as much as possible.

[0081] S24: The high-frequency subband mainly contains the edge and texture details of the image. In order to preserve the edge and texture details of the image, a high-frequency subband coefficient with strong edge preservation capability is selected. The edge preservation capability is expressed by the following formula.

[0082] Q=(w1*I h ) 2 +(w2*I h ) 2 +(w3*I h ) 2 +(w4*I h ) 2 (11)

[0083]

[0084] Among them, w1, w2, w3, w4 are operators that retain edge details, I h is the high-frequency sub-band coefficient. Calculate the maximum fusion coefficient and fuse the high-frequency sub-bands of the grayscale image, gradient image, and depth image.

[0085] Function of high-frequency subband coefficients: High-frequency subbands mainly contain edge and texture details in the image. By selecting high-frequency subband coefficients with strong edge preservation capabilities, it can ensure that important detail information is preserved during the image fusion process.

[0086] How to use the high-frequency sub-band coefficients, whether the high-frequency sub-band coefficients are related to the maximum fusion coefficient: by comparing the coefficient Q calculated by formula (11), select the high-frequency sub-band coefficient with the largest Q as the high-frequency sub-band fusion coefficient. (After decomposition, there are multiple high-frequency sub-bands, and there are also multiple calculated coefficients. The largest coefficient is selected as the fusion coefficient).

[0087] Low-frequency subband coefficients: represent the information of the low-frequency part of the image, which contains most of the energy and main features of the image, but with fewer details.

[0088] High-frequency subband coefficients: refers to the coefficients extracted during the transformation process that contain image details, texture, and edge information. Compared with low-frequency subbands, high-frequency subband coefficients reflect the rapidly changing parts of the image.

[0089] S25: performing an inverse transformation of the non-subsampled contourlet transform on the fused low-frequency and high-frequency sub-bands to obtain a fused image.

[0090] Step S3, identifying defects based on the deep learning model.

[0091] Step S3 also includes the following steps:

[0092] S31: The dataset used in this method contains defective and normal leather images, where various types of defects exist, including holes, three threads, knots, broken spandex, and rough fibers. The resolution of the input fusion image is 8192 pixels × 4096 pixels, and the minimum defect size required to be detected is 10 pixels × 10 pixels.

[0093] During the dataset preparation process, the original high-resolution image cannot be directly used for model training. We crop the fusion image into 8 low-resolution images with a size of 2048 pixels × 2048 pixels.

[0094] We use YOLOv5's AutoAugment to effectively enrich the data set and introduce more data changes during the training process, making the model better adapted to target detection tasks in various scenarios, helping to improve the model's robustness and generalization capabilities, and improving the accuracy and reliability of defect detection.

[0095] LabelImg annotation is used, following the VOC2007 dataset format. K-means++ algorithm is used to cluster the prior box. In the initialization stage, we select a target box with a larger area as the first cluster center, and then use the cosine similarity metric to gradually select the target box with the smallest similarity to the current cluster center as the new cluster center. Subsequently, in the iterative process, the target box with the smallest similarity metric to the current cluster center is selected as the new cluster center to better capture the defect characteristics of each cluster, thereby generating a prior box size that is more suitable for the defect size and shape.

[0096] In addition, in view of the characteristics of tiny defects, we have also modified the three loss functions of YOLOv5 to better adapt to the detection of small objects. Specifically, we adjusted the weights and calculation methods of positioning loss, confidence loss, and classification loss, which can improve the model's detection performance for tiny defects, making it more accurate and robust.

[0097] S32: Model training, taking into account the model recognition accuracy and real-time performance, YOLOv5s is used as the defect detection model and trained. The main structure of YOLOv5s consists of four parts: Input input end, Backbone backbone network, Neck network layer and Head detection end.

[0098] The input end is mainly responsible for preprocessing the input image, which mainly consists of three methods: data expansion, adaptation, anchor frame calculation and image stretching. Data expansion is performed by randomly scaling 4 randomly selected images, then randomly distributing and cropping them, and finally randomly splicing the cropped images to expand the data set, reduce the high memory usage requirements during network training, and improve the ability to detect weak defects. In order to improve training efficiency, we use batch normalization to accelerate model convergence and reduce the problem of the model being sensitive to initial weights in the early stages of training. Batch normalization also helps prevent overfitting and improve the model's generalization ability on unknown data.

[0099] The Backbone network is mainly responsible for feature extraction of images, and is usually composed of modules such as FOCUS, CBS, CSP, and SPP. First, for the input image (2048×2048×3), the FOCUS module specific to YOLOv5 divides it into multiple sub-images (640×640×3), and then performs feature extraction. Next, these sub-images are fused in sequence along the channel dimension, and a convolution operation is performed using a convolution kernel of size 64×64, and an image of size (320×320×64) is output. Subsequently, a down-sampling feature extraction operation is performed through a CBS convolution module of size 3×3 and a stride of 2. Next, the feature map is processed by three CSP and CBS convolution modules in sequence. Finally, the output feature map is processed by the SpatialPyramidPooling (SPP) module for multi-scale feature fusion to improve accuracy, and the pooled feature map is fused and input into the Neck layer through the Concat operation.

[0100] The Neck layer is mainly responsible for fusing the feature information extracted by the backbone layer and inputting it into the subsequent prediction layer. The Neck layer is generally divided into two parts: Feature Pyramid Network (FPN) and Path Aggregation Network (PAN). The Feature Pyramid Network transfers and fuses high-level semantic feature information through upsampling in a top-down manner to obtain a prediction feature map, while the Path Aggregation Network transfers the positioning feature information from the bottom up.

[0101] The head detection head analyzes the feature maps of different scales output by the Neck network and trains them through the loss function and NMS non-maximum suppression. Finally, the feature maps of different scales are output y1, y2, and y3 to achieve multi-scale prediction.

[0102] Among them, y1 is the coarsest feature map with a size of 14×14, which is used to detect large-sized objects. y2 is a medium-sized feature map with a size of 28×28, which is used to detect medium-sized objects. Y3 is the finest feature map with a size of 56×56, which is used to detect small-sized objects. Each feature map divides the input image into S×S cells, where S is 56, 28, and 14 respectively. These grids help the model locate objects at different size levels. Each grid sets 3 prior boxes to predict 3 bounding boxes, and the output dimension is S×S×(3×(4+1+M)), where 3 means that each grid has three prior boxes, and 4+1+M are the four coordinates, confidence, and M types of defects of the bounding box, respectively. Finally, non-maximum suppression is used to eliminate duplicate bounding boxes to achieve leather defect detection.

[0103] The loss function during training consists of three parts: bounding box error, confidence error, and category error. The bounding box error uses GIOU (intersection-over-union) loss, and the confidence error and category error use balanced cross entropy loss. The total loss function is:

[0104] L=λ bbox ·L bbox +λ conf ·L conf +λ cls ·L cls (11)

[0105] λ bbox , conf , cls is the weight factor.

[0106]

[0107] |B p ∩B gt )| is the intersection area of ​​the predicted box and the real box, |B p ∪B gt | is the union area of ​​the predicted box and the true box, |C| is the area of ​​the smallest enclosing box, |C-(B p ∪B gt )| is the area difference between the minimum enclosing box and the union of the predicted box and the true box.

[0108]

[0109] N is the number of prediction boxes, Indicates whether the i-th prediction box contains the true label of the target, p i Predict the confidence that the i-th prediction box contains the target for the model.

[0110]

[0111] C is the total number of categories, w c is the weight for category c, which is set inversely according to the frequency of occurrence of the category, y c is the true label, if the target belongs to category c, it is 1, otherwise it is 0. The model predicts the probability that the sample belongs to c.

[0112] According to the above data set, model, and loss function, a defect detection model for fused images can be trained.

[0113] On the other hand, the tested samples include leather, wherein the types of defects are various and include at least one of holes, three threads, knots, broken spandex and coarse fibers, the resolution of the input fusion image is 8192 pixels × 4096 pixels, and the minimum defect size to be detected is required to be 10 pixels × 10 pixels.

[0114] For a 8192pixel×4096pixel fusion image to be detected, it is first cropped into 8 images of 2048pixel×2048pixel size and sent to the model for detection at the same time.

[0115] The above embodiments are only illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A two-dimensional and three-dimensional fusion fabric defect detection method, characterized in that: The steps include: Step S1, multi-angle light source imaging, lighting the fabric under inspection through light sources at different angles, photographing through a line array camera, and obtaining two-dimensional images of different angles of illumination; Step S2, fusion of two-dimensional and three-dimensional information, pixel-level fusion of the captured two-dimensional grayscale image and the reconstructed gradient image and depth image with three-dimensional information, high-low frequency decomposition and fusion of the two-dimensional information and the three-dimensional information by non-subsampled contourlet transform, fusion of the two-dimensional defect feature information and the three-dimensional defect feature information into one image, after the non-subsampled contourlet transform decomposes the image, the frequency domain space is divided into wedge-shaped directional sub-bands; Step S3, identifying defects based on the deep learning model.

2. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 1 is characterized in that: Step S1 includes the following steps: S11: First, place the red and green light sources symmetrically, keep the incident light and the light plane of the camera collinear, and make the incident angles of the two light sources relative to the camera the same; S12: placing the sample to be tested on the translation stage and fixing it to ensure that the relative position of the sample and the translation stage does not deviate when the sample is driven by the translation stage; S13: The moving translation stage carries the sample to be tested to a preset position, and at the same time, the encoder triggers a signal to the linear array camera, and the linear array camera collects the fabric image illuminated by the red and green light sources at the same time. By separating the red and green channels of the camera, images illuminated by different light sources can be obtained; S14: Photometric stereo is used to reconstruct the three-dimensional information of the sample under test. Two symmetrical red and green line light sources are used. The incident angle α of the light source can be obtained by calibrating the reflection model, and then the direction vector of the light source can be obtained as l1=(0, sinα, cosα), l2=(0, -sinα, cosα). The y-direction gradient q of the sample surface is obtained. According to the model: I=ρEL·n, where I is the image brightness; E is the incident light intensity; n is the unit normal vector; L is the unit direction vector of the light source; ρ is the reflectivity. Calculate: S15: Based on the gradient q reconstructed by the photometric stereo method, the surface gradient is integrated using a global iterative optimization method to restore the surface depth information, so that the iterative error is within a preset range, and the three-dimensional depth information z of the sample surface can be obtained; S16: Normalize the gradient q and the depth z to [0, 255] to obtain the gradient map q and the depth map z.

3. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 1 is characterized in that: Step S2 includes the following steps: S21: q is a gradient image normalized to [0,255]. The gradient image is decomposed into two scales based on rolling guided filtering to obtain the rolling guided filter image q RGF And Gaussian filter filter graph q GF , transformed into the logarithmic domain to obtain the base layer q b and detail layer q d , image enhancement is performed in the logarithmic domain, and the enhanced gradient map q can be obtained by converting the logarithmic domain into the spatial domain; S22: The grayscale image I, gradient image q, and depth image z are decomposed at multiple scales using a non-subsampled pyramid, and multi-directionally decomposed using a non-subsampled directional filter group. The number of decomposition layers is 4, and the corresponding low-frequency subband image I1 is decomposed. 0 ,q1 0 , z1 0 , and the corresponding high-frequency subband image I l k ,q l k , z l k , l represents the number of decomposition layers, and k represents the number of high-frequency subbands decomposed at layer l; S23: Fuse the low-frequency sub-bands of the grayscale image, gradient image, and depth image. The fusion rule adopts the absolute maximum principle to preserve the energy of the low-frequency sub-band as much as possible. The low-frequency sub-band mainly contains most of the energy and a small amount of details of the original image, where energy is defined as the sum of the squares of the pixel intensities in the image. S24: The high-frequency subband mainly contains the edge and texture details of the image, and the high-frequency subband coefficient with strong edge preservation ability is selected; S25: performing an inverse transformation of the non-subsampled contourlet transform on the fused low-frequency and high-frequency sub-bands to obtain a fused image.

4. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 1 is characterized in that: Step S3 also includes the following steps: S31: Dataset preparation: crop the fusion image into 8 low-resolution images of 2048 pixel × 2048 pixel size, use YOLOv5 automatic enhancement, use LabelImg annotation, follow the VOC2007 dataset format, and use K-means++ algorithm to cluster the prior frame. In the initialization stage, select a target frame with a larger area as the first cluster center, and then use the cosine similarity metric to gradually select the target frame with the smallest similarity to the current cluster center as the new cluster center; then, in the iterative process, select the target frame with the smallest similarity metric to the current cluster center as the new cluster center, so as to generate a prior frame size that is more suitable for the defect size and shape.

5. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 4 is characterized in that: Step S3 also includes the following steps: S32: Model training, using YOLOv5s as the defect detection model and training. The main structure of YOLOv5s consists of four parts: Input input end, Backbone backbone network, Neck network layer and Head detection end. At the input end, batch normalization is used to accelerate model convergence. The Backbone network is responsible for extracting features from the image. The Neck layer is mainly responsible for fusing the feature information extracted by the backbone layer and inputting it into the subsequent prediction layer. The Head detection head parses the feature maps of different scales output by the Neck network, and trains them through the loss function and NMS non-maximum suppression, and finally obtains the output y1, y2, y3 of feature maps of different scales to achieve multi-scale prediction.

6. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 5 is characterized in that: In step S32, y1 is the coarsest-grained feature map with a size of 14×14, which is used to detect large-sized targets; y2 is the medium-grained feature map with a size of 28×28, which is used to detect medium-sized targets; y3 is the finest-grained feature map with a size of 56×56, which is used to detect small-sized targets; Each feature map divides the input image into S×S cells, where S is 56, 28, and 14 respectively. These grids help the model locate targets at different size levels. Three prior boxes are set for each grid to predict three bounding boxes. The output dimension is S×S×(3×(4+1+M)), where 3 means that each grid has three prior boxes. 4+1+M are the four coordinates, confidence, and M types of defects of the bounding box respectively. Finally, non-maximum suppression is used to eliminate duplicate bounding boxes to achieve sample defect detection.

7. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 1 is characterized in that: The tested sample includes leather, wherein the defects are various and include at least one of holes, three threads, knots, broken spandex and coarse fibers. The resolution of the input fusion image is 8192 pixels × 4096 pixels, and the minimum defect size to be detected is required to be 10 pixels × 10 pixels.

8. The two-dimensional and three-dimensional fusion fabric defect detection method according to claim 7 is characterized in that: For a 8192pixel×4096pixel fusion image to be detected, it is first cropped into 8 images of 2048pixel×2048pixel size and sent to the model for detection at the same time.

Images