Cross-scale fabric defect sample generation method and system based on comparative learning mutual information mechanism

Abstract

The invention discloses a cross-scale fabric defect sample generation method based on a comparative learning mutual information mechanism, and the method comprises the steps: S1, constructing a comparative learning mutual information network, constructing a pixel-level fabric defect-background microscopic stripper, setting a PatchNCE-based structural similarity loss function, and generating an IFD-GAN fabric defect image generation model; according to the acquired fabric image, extracting an input image block with a preset size; performing flaw image stripping on the input image block by using an IFD-GAN fabric flaw image generation model to generate a high-fidelity flaw image block; and embedding the generated high-fidelity defect image block into the original industrial-size fabric image in a sliding window mode, and outputting an industrial-size target fabric defect image. The problems that real flaw images are difficult to collect, diversity and inter-class balance are insufficient, industrial size images cannot be directly processed through an existing method, and continuous warp and weft fabric flaws cannot be directly generated can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of defect image processing, in particular to a cross-scale fabric defect sample generation method and system based on a contrast learning mutual information mechanism. BACKGROUND

[0002] Fabric anomaly detection and quality control in the textile industry play an important role in improving product core competitiveness and meeting the growing market requirements. Fabric defects not only affect the appearance and performance of products, but also easily lead to economic and reputation losses for enterprises. Common types of industrial defects and their texture details in close-up images are shown in FIG. 1. Industrial fabric defects have obvious cross-scale characteristics, and how to accurately and efficiently detect various cross-scale defects has become a key problem to be solved. Traditional defect detection mainly relies on manual visual inspection, which is inefficient and easily affected by subjective factors, and cannot meet the needs of large-scale automated industrial applications. Intelligent defect detection based on artificial intelligence has gradually become a research hotspot. Figure 1

[0003] Deep learning technology has greatly promoted the intelligent application of industrial fabric defect detection. Target detection algorithms based on deep learning have outstanding defect recognition, classification and positioning capabilities, and are recognized as a key solution to realize intelligent defect detection. However, even though deep learning has achieved good results in the field of defect detection in recent years, related improvements have been proven to be effective, but there is still a great gap with the requirements of industrial applications, and there are insurmountable bottlenecks. The effectiveness of the detection model and the industrial generalization depend to a great extent on the size, balance and diversity of the data set, and a high-quality industrial fabric defect data set is of great significance to improve the comprehensive performance of the detection model and the effectiveness of industrial applications. Due to the limited and single defect samples provided by the actual industrial environment, poor inter-class balance, and many image imaging interference factors, it is often difficult to completely rely on field collection to construct a diversified high-quality industrial fabric defect data set.

[0004] ​The industrial fabric flaw size span is large, the shape is complex, the dispersibility is strong, and the long-diameter ratio of the warp and weft through type flaw accounts for the majority. The typical characteristics of such flaw images are that the target features are only distributed along one dimension and penetrate the whole image in this dimension, the long-diameter ratio is extremely large, and the pixel ratio is extremely small, which is essentially different from local flaws. When facing the industrial fabric flaw detection, the existing deep learning technology often faces the difficulties of image acquisition difficulty and inaccurate flaw feature extraction, and the detection difficulty is further increased in the complex industrial scene, which restricts the industrial application effect. Therefore, how to generate high-fidelity cross-scale fabric flaw images under the industrial size and provide sufficient and diversified balanced training data set for the detection algorithm has become a key problem to be solved. Limited by computing power and model capability, the existing image generation technology cannot directly process and generate fabric flaw images of the size of the industrial camera imaging; for the warp and weft flaws with large long-diameter ratio, the existing model and its improved scheme cannot directly and completely realize the high-fidelity image generation.

[0005] Therefore, there is an urgent need for a sample generation method which can solve the problems of real flaw image acquisition difficulty, lack of diversity and inter-class balance, and the existing method cannot directly process industrial size images and cannot directly generate continuous warp and weft fabric flaws. SUMMARY

[0006] The purpose of the application is to overcome the above shortcomings, and the purpose of the application is to provide a cross-scale fabric flaw sample generation method and system based on a contrast learning mutual information mechanism. The application proposes a high-fidelity fabric flaw image generation algorithm IFD-GAN for industrial applications, which solves the problems of real flaw image acquisition difficulty, lack of diversity and inter-class balance, and the existing method cannot directly process industrial size images and cannot directly generate continuous warp and weft fabric flaws.

[0007] To solve the above technical problems, the application provides a cross-scale fabric flaw sample generation method based on a contrast learning mutual information mechanism, comprising: S1: constructing a contrast learning mutual information network comprising a generator and a discriminator, constructing a pixel-level fabric flaw-background micro-dissector for extracting fine-grained texture information of the long-diameter ratio warp and weft flaws and generating a pixel-level flaw mask; S2: setting a structure similarity loss function based on PatchNCE and combining the contrast learning mutual information network, the pixel-level fabric flaw-background micro-dissector and the warp and weft attention mechanism to generate an IFD-GAN fabric flaw image generation model; S3: extracting an input image block of a preset size according to the collected fabric image; S4: using the IFD-GAN fabric flaw image generation model to perform flaw image dissection on the input image block to generate a high-fidelity flaw image block; S5: The generated high-fidelity defect image block is embedded into the original industrial-size fabric image through a sliding window method, and an industrial-size target fabric defect image is output.

[0008] As a preferred mode of the present application, when constructing the contrastive learning mutual information network comprising the generator and the discriminator, the method comprises: S11: For the input-output image domain , , , , , respectively represent any image elements of the image domains X and Y, H and W represent the image resolution size height and width, respectively, and C represents the number of channels, , represents the image set; S12: Set the PatchNCE loss function: , , , represents the number of spatial positions of the l layer, , respectively represent the corresponding position features of the input and output images, , represents other position features, wherein is the number of feature channels for each layer, , represents a multi-layer perceptron; S13: Set the contrastive loss function with PatchNCE as the core : , , is the classical adversarial loss: , , represents the model generator, , represents the model discriminator, , represents the average value, D(y) represents the probability of a real sample, G(x) and D(G(x)) represent the generated false image and its probability generated by the generator, respectively, , , are weight coefficients, and the value range is set to 10 -1 to 10 2 .

[0009] As a preferred mode of the present application, when constructing the contrastive learning mutual information network comprising the generator and the discriminator, the method further comprises: S16: Calculate the cross-entropy loss: , , is the feature vector of the output image random position region block, is a feature vector of a region block in the same position of the input image, , is a feature vector of a region block in the remaining part of the input image, is a scale hyperparameter, which is used to control key scale relations such as model structure, data distribution or regularization strength.

[0010] As a preferred mode of the present application, when constructing the pixel-level fabric defect-background micro-peeler, the method comprises: S101: setting the pixel-level fabric defect-background micro-peeler as a U-net structure; S102: introducing a CARAFE operator in the up-sampling process and compressing the feature map channel number through 1x1 convolution, and performing feature map group convolution; S103: performing sub-pixel convolution on the output feature map, splitting the channel number of each pixel point into a preset number of sub-pixel channels and rearranging, expanding the channel dimension in the spatial dimension, and obtaining the up-sampling feature map after normalization using softmax, and then performing feature fusion on the input feature map and the up-sampling feature map; S104: setting a skip connection to directly pass the shallow features of the encoder to the decoder and introducing a spatial channel attention mechanism in the depth channel; S105: setting the defect conversion process formula between different images: , , wherein, represents an input defect window image, represents a first-order defect window image directly generated by the improved defect generator, represents a pixel-level defect mask generated by the DBS, and a final generated defect window image is and respectively, dot product operation and fusion results, represents spatial multiplication.

[0011] As a preferred mode of the present application, when the warp and weft attention mechanisms are combined, the method comprises: S21: obtaining the intermediate feature map of the fabric image from the generator as the input of the warp and weft self-attention mechanism; S22: cutting the input feature map along the warp and weft two spatial dimensions respectively to form two branches for separately processing the warp and weft information, and applying 1xN convolution in the warp branch and Nx1 convolution in the weft branch; S23: One-dimensional attention operation is performed on the corresponding direction features in two branches respectively, and the weight response of each spatial position is calculated to obtain the meridional and latitudinal attention results; S24: The attention weight is obtained by using 1x1 convolution and Softmax function, and the output feature map is obtained by weighting and summing the meridional and latitudinal attention results and the input feature map; S25: The output feature map is connected with the original input feature map in residual connection, and the enhanced meridional and latitudinal attention features are output, and then the enhanced feature map of the meridional and latitudinal self-attention is transmitted to the pixel-level fabric defect-background micro-detacher.

[0012] As a preferred mode of the present application, when setting the PatchNCE-based structural similarity loss function, the method comprises: S21: Set the formula for calculating the structural similarity between features: , , , , , , , , , , , , , -2 , , , , , , , , -2 , ,

[0013] As a preferred mode of the present application, after generating the high-fidelity defect image block, the method further comprises: The peak signal-to-noise ratio and the structural similarity index are used to evaluate the quality and fidelity of the generated image; The mean average precision index under the condition that the accuracy, recall rate and IoU are respectively set to 50% and 50%:95% is used to comprehensively evaluate the detection precision before and after enhancement, and the frame rate is used to evaluate the detection speed.

[0014] As a preferred mode of the present application, when the peak signal-to-noise ratio and the structural similarity index are used to evaluate the quality and fidelity of the generated image, the method comprises: The peak signal-to-noise ratio is defined by the mean square error: and the peak signal-to-noise ratio is expressed by the logarithm: wherein, is the image size, is the original image, is the reconstructed image, represents the maximum color value of the image point; The structural similarity index evaluation uses the mean value as the estimate of the brightness, the standard deviation as the estimate of the contrast, and the covariance as the measure of the structural similarity, and the calculation formula is: wherein, and represent the mean value in the direction of image I and J respectively, and are the standard deviations, is the covariance of I and J, and are the dimension stability constants, , wherein is the dynamic range of the pixel value, k1=0.01, k2=0.03.

[0015] As a preferred mode of the present application, when the detection accuracy and the detection speed before and after enhancement are comprehensively evaluated and the frame rate is used to evaluate the detection speed, the method comprises: The accuracy rate index calculation formula is: wherein is the number of correctly determined defect samples, is the number of negative samples that are incorrectly determined as defect samples, i.e. the false detection number; The recall rate index calculation formula is: wherein is the number of defect samples that are incorrectly determined as negative samples, i.e. the missed detection number; The mean average precision index calculation formula of the IoU is set to 50% and 50%:95% respectively: , , wherein is the total number of categories, is the average precision, which is the area surrounded by the PR curve with the recall rate R as the horizontal axis and the accuracy rate P as the vertical axis and the coordinate axes, is the intersection over union, and represent the predicted frame and the real frame respectively; The frame rate index calculation formula is: , wherein is the number of model processing pictures, is the total consumption time.

[0016] The application also designs a cross-scale fabric defect sample generation system based on a contrast learning mutual information mechanism using the above method, comprising: A defect model construction module is configured to construct a contrast learning mutual information network comprising a generator and a discriminator, construct a pixel-level fabric defect-background micro-separator for extracting fine-grained texture information of large aspect ratio warp and weft defects and generating a pixel-level defect mask, set a PatchNCE-based structural similarity loss function, and combine the contrast learning mutual information network, the pixel-level fabric defect-background micro-separator, and the warp and weft attention mechanism to generate an IFD-GAN fabric defect image generation model. A fabric image processing module is configured to extract input image blocks of a predetermined size according to the collected fabric images. A defect image generation module is configured to use the IFD-GAN fabric defect image generation model to perform defect image separation on the input image blocks to generate high-fidelity defect image blocks, and then embed the generated high-fidelity defect image blocks into the original industrial-size fabric images through a sliding window method to output an industrial-size target fabric defect image.

[0017] The above technical solutions of the application have the following advantages compared with the prior art: 1. The application designs a pixel-level defect-background separator for large aspect ratio warp and weft defects. Thanks to the high-efficiency decoupling and extraction capability of the defect separator for warp and weft defect texture detail features, the IFD-GAN realizes high-fidelity generation and positioning of defects across the background domain, and precisely controls the fusion degree of defect detail features and background texture.

[0018] 2. The application proposes a new warp and weft self-attention mechanism, which adopts two one-dimensional self-attention parallel branches along the warp and weft directions to respectively focus on the target defects in the corresponding dimensions, realizes industrial defect detail feature focusing, cooperates with the defect separator to accurately extract defect texture information, and further ensures high-fidelity generation of defects while effectively avoiding a large amount of redundant calculation.

[0019] 3. The application designs a PatchNCE-based structural similarity loss function, which efficiently regulates defect generation details and their association with the overall image, balances the consistency of IFD-GAN generated defect features and background texture, and balances the consistency of IFD-GAN generated defect features and background texture.

[0020] 4、The application can effectively realize high-fidelity generation of cross-scale fabric defects in industrial size images, and further realize data set scale expansion, inter-class balance and diversity expansion functions, breaking the constraint of common models limited by image size processing. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0022] Figure 1 is a schematic diagram of common types and characteristics of cross-scale industrial fabric defects provided by the prior art.

[0023] Figure 2 is a schematic diagram of IFD-GAN design and generation strategy for various defects provided by the embodiments of the present application.

[0024] Figure 3 is a schematic diagram of introducing a contrast learning mutual information mechanism provided by the embodiments of the present application.

[0025] Figure 4 is a schematic diagram of N+1 classification problem conversion mechanism and PatchNCE loss evaluation based on regional block contrast learning provided by the embodiments of the present application.

[0026] Figure 5 is a schematic diagram of IFD-GAN pixel-level defect-background stripper (DBS) provided by the embodiments of the present application.

[0027] Figure 6 is a schematic diagram of introducing warp and weft direction self-attention mechanism provided by the embodiments of the present application.

[0028] Figure 7 is a contrastive schematic diagram of mutual information mechanism and PatchNCE loss strategy applied to different generation tasks provided by the embodiments of the present application.

[0029] Figure 8 is a flowchart of a cross-scale fabric defect sample generation method based on a contrast learning mutual information mechanism provided by the embodiments of the present application.

[0030] Figure 9 is a schematic diagram of module connection of a cross-scale fabric defect sample generation system based on a contrast learning mutual information mechanism provided by the embodiments of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described below in detail with reference to the accompanying drawings, wherein the same or similar components are denoted by the same or similar reference numerals throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and are not to be understood as limiting the present application.

[0032] In the present application, the measurement angle and standard of high-quality industrial fabric defect dataset are as follows: (1) High quality and high fidelity of target defects: The quality of the industrial fabric defect dataset is reflected in the presentation of the target defect features in the image. Industrial fabric defects have obvious characteristics such as cross-scale characteristics, irregularity, and dispersion, so the target defects in the high-quality dataset image should be clear and faithful, especially the large aspect ratio of warp and weft defects should be coherent and complete, so as to truly and objectively reflect and present the defect features. This helps the detection algorithm to better grasp the target defect features, so as to have better detection effect and robustness.

[0033] (2) Dataset size and diversity: High-quality industrial fabric defect dataset should have a certain size and diversity. For industrial applications, diversity not only reflects in the fabric defect categories and different scales and shapes, but also in the corresponding relationship between defects and background textures. This can make the model have better adaptive ability and generalization ability when facing different industrial application scenarios.

[0034] (3) Balance between categories: In the multi-class target positioning and classification task of industrial fabric defect detection, the dataset should provide as balanced as possible class data information for the algorithm model. In the training process, the relative balanced distribution of each defect in the dataset helps to improve the model detection accuracy. If the number of samples of several categories of defects is too small to destroy the balance between categories, it is easy to lead to model overfitting and reduce its industrial practical value.

[0035] (4) Accuracy of labeling: The extraction of fabric defect image foreground target is highly dependent on accurate foreground object labels, so the images in the dataset need to be accurately labeled so that the model can learn the correct features and implement accurate foreground stripping. The labeling of different categories of target defects in the high-quality industrial fabric defect dataset should follow the corresponding rules to compensate for the influence of cross-scale and irregular features, reduce the interference of fabric background, and improve the learning reliability of the algorithm model.

[0036] Thus, to realize high-fidelity generation of fabric defect images of cross-scale under the imaging size of an industrial camera, to construct a diversified industrial fabric defect dataset with high quality, large scale, and strong balance, and to improve the application effect of algorithms in industrial complex scenes, the present technology proposes a high-fidelity industrial fabric defect image generation model named IFD-GAN (Industrial Fabric Defects-Generative Adversarial Network), and a technical flowchart of the model is shown in Figure 2 IFD-GAN is a new second-order generation algorithm based on contrastive learning, aiming to realize high-quality generation of cross-scale fabric defects in an unsupervised environment.

[0037] For the input of 1920x1080 high-resolution cross-scale fabric defect images imaged by an industrial camera, IFD-GAN first judges the defect category and selects the model processing window size (such as 256x256). For local class defects, the synthesis of defects and different background images (or other defect images) is realized through a defect generation structure. For large-length-diameter-ratio weft and warp penetrating fabric defects, to realize high-quality migration of this type of defects across the background domain, IFD-GAN decomposes the task into two sub-stages: first, a specially designed defect-background stripper is used to accurately extract the defect texture from the fabric background texture, and then the generated defect is fused with the original background to realize high-fidelity migration of the target area defect. For the industrial characteristics of the close coupling of fabric defects and background texture, this innovative weft and warp defect stripper can effectively realize pixel-level segmentation of defects. By fusing appropriate attention mechanisms and feature reorganization operators and other strategies, the model's attention to the underlying features of the image is enhanced, accurate positioning and detail fidelity of the defect are realized, and the defect migration effect is improved. To cooperate with the defect stripper and maximize its benefits, an innovative weft and warp self-attention mechanism is specially designed for the IFD-GAN generator, which cooperatively realizes high-fidelity migration of weft and warp defects while reducing redundant calculations and background interference. IFD-GAN uses an improved structure similarity loss function based on PatchNCE to coordinate the generation details of the defect and its relationship with the background texture. After fine-tuning the color information of the window image, IFD-GAN implants it into the industrial image, and combines it with the window sliding step iteration to realize the complete generation of weft and warp defects, and finally outputs the target fabric defect image of industrial size.

[0038] In the present application, a contrastive learning mutual information mechanism is also introduced. The non-paired image one-way translation architecture based on contrastive learning adopts the strategy of maximizing the mutual information of the corresponding positions of the input and output image blocks and the multi-layer, block-based idea, so that the generation network can supervise the discovery of the common parts of the two corresponding positions while ignoring the characteristic parts of the different positions, as shown in Figure 3 For the task of high-fidelity fabric defect image generation, it is beneficial to focus on key detail information such as the appearance and texture of the defect target domain, and has important application potential.

[0039] Especially for the industrial fabric defect image generation task, under the mutual information contrast learning strategy, the key detail information such as the appearance and texture of the defect target domain is focused on and the background domain information is ignored, which is conducive to realizing high-fidelity defect migration. This method adopts a multi-layer, block-based idea and extracts negative samples from the input image itself, realizes one-sided translation in the unpaired image-to-image translation setting, and can be extended to a training setting where each domain is only a single image, which has outstanding performance in image migration and translation tasks, especially has important application potential for industrial fabric defect image generation.

[0040] Therefore, in order to realize high-fidelity image generation of cross-scale fabric defects under industrial size, the image one-sided translation framework based on contrast learning mutual information is used as the backbone network. The contrast loss function PatchNCE loss is used to make the encoder have the ability to associate corresponding image blocks and separate other image blocks, so that the encoder can focus on the common part between domains and ignore the difference part between domains.

[0041] For the input and output image domains , where , , where and represent any image elements of image domains X and Y respectively, H and W represent image resolution sizes high and wide respectively, and C represents the number of channels, represents the image collection; CUTGAN adopts a new contrast loss function PatchNCE as the core , which is expressed as: , where is the classic adversarial loss: , where represents the model generator, represents the model discriminator, represents the average value, D(y) represents the probability of a real sample, G(x) and D(G(x)) represent the generated false image and its probability generated by the generator respectively, , are weight coefficients, and the value range is set to 10 -1 to 10 2 .

[0042] The framework adopts a multi-layer and region block-based idea, uses multi-level region blocks as learning targets and adopts multi-layer features. Different layers and spatial positions in the encoder feature encoding layer represent different region blocks, and the deeper the layer, the larger the region block. After the input image ( ) passes through the encoder, L layer features are selected, and two-layer MLP network Hl A series of features are generated : wherein represents the output feature of the l-th layer, Likewise, the output image ( ) is encoded by the encoder and the multi-layer perception layer to obtain the feature : .

[0043] To match the image blocks at the corresponding positions of the input and output, other image blocks at other positions of the same image or other images are extracted as negative samples, as shown in Figure 4 , and the loss is denoted as PatchNCE loss, which is calculated as follows: wherein , represents the number of spatial positions of the l-th layer, l , respectively represent the features of the input and output images at the corresponding positions, , represents the features at other positions, wherein is the number of feature channels of each layer, represents the multi-layer perception. Further, to maximize the mutual information, the key of the contrastive learning strategy lies in constructing positive and negative samples to calculate the mutual information. The Noise Contrastive Estimation (NCE) framework ingeniously regards the target region block in the output image as an object (query), the region block at a random position in the output image as an anchor point, and the region block at the same position in the input image as a positive sample (positive), and the region blocks of the remaining N parts in the input image as negative samples (negatives), which are respectively mapped into K-dimensional vectors, so as to transform it into a classification problem of N+1 classes, the mechanism of which is shown in

[0044] , so as to calculate the cross-entropy loss, thereby narrowing the distance between the target region block and the positive sample in the feature space while pushing away the distance between the target region block and the negative sample in the feature space, as shown in the following formula: Figure 4 wherein , is the feature vector of the region block at a random position in the output image, is the feature vector of the region block at the same position in the input image, , is the feature vector of the region block of the remaining part in the input image, ​The proportionality parameter is used to control the key proportionality relationship of the model structure, data allocation, or regularization strength, etc. .

[0045] For images from the domain , the target image y is usually input into the generator to generate an image , and the loss y and are calculated to prevent the generator from making unnecessary changes, which is equivalent to the CycleGAN cycle consistency loss.

[0046] In this application, a pixel-level fabric defect-background micro-stripper (DBS) is also designed. Specifically, when generating an industrial fabric defect image, attention should be paid to local detail fidelity, and background harmony should also be considered. The texture detail features of industrial fabric defects are often closely coupled with background information. Therefore, the model is required to have more accurate detail capturing and processing capabilities, and also to accurately control the fusion degree of defect features and background texture to achieve more natural and realistic high-fidelity industrial defect image generation.

[0047] To realize efficient extraction of defect texture detail features and decoupling of defect texture detail features and background information in the process of cross-scale industrial fabric defect high-fidelity image generation, overcome the defects such as poor defect quality, artifacts near the target domain, and significant lack of fidelity in conventional defect image generation, and distinguish from traditional binary segmentation masks, this application proposes a pixel-level defect-background stripping technology for large aspect ratio warp and weft defects. A defect stripper is additionally added in the IFD-GAN generator for pixel-level defect stripping and defect positioning, and its structure is shown in Figure 5 .

[0048] The pixel-level fabric defect-background micro-stripper adopts a U-net structure. To match the cross-scale features of industrial fabric defects, overcome the defects such as small perception domain and insufficient utilization of defect semantic information of conventional U-net, and realize high-fidelity generation of warp and weft defect details, a CARAFE operator is introduced in the downsampling process of the defect stripper. The feature map channel number is compressed through 1x1 convolution to reduce the computational complexity, and then feature group convolution is performed. Further, the output feature map is sub-pixel convolved, each pixel channel is split into 4 sub-pixel channels and rearranged, the channel dimension is unfolded in the spatial dimension, and after normalization using softmax, the upsampled feature map is obtained. The input feature map and the upsampled feature map are fused to better utilize the context semantic information of the input feature map.

[0049] In the fabric defect-background stripping task, the target defect foreground texture features are more worthy of attention, and the deep features of the image after experiencing multiple convolution layers are almost unable to retain effective defect texture information. To meet the high-fidelity demand of defect generation, the decoder of the last layer of the IFD-GAN generator is connected with the corresponding features of the encoder to enable the decoder to obtain more context information at a higher level. The defect stripper can not only retain high-level semantic information but also fully utilize shallow information to obtain defect texture detail features. A spatial channel attention mechanism CBAM is set in the depth feature output channel of the defect stripper, which aims to suppress the noise features contained in the depth features and convey effective global context information. Finally, a softmax function is used to generate a pixel-level mask label map.

[0050] The IFD-GAN pixel-level fabric defect-background microstripper focuses on more accurate extraction of defect texture detail information and defect positioning. Subsequently, the IFD-GAN will use the generated mask to implement fine decoupling and pixel-level stripping of defects and backgrounds, and fuse with the generated defect foreground to achieve high-fidelity warp and weft defect image generation. For the typical cross-scale features of industrial fabric defects, DBS can independently strip and encode the deep semantic information of large-aspect-ratio defects with irregularity and dispersion in an unsupervised environment, i.e., DBS can not only effectively avoid overfitting of defect and background textures, but also significantly reduce the possibility of defect artifacts in the image conversion process, because the generated artifacts only remain in the defect foreground image and cannot be screened by the mask attention mechanism. Similarly, DBS can fully retain the background properties and avoid distortion. The designed DBS gives the IFD-GAN excellent pixel-level decoupling capability between defects and background textures, effectively prevents artifacts in non-defect areas, enhances the controllability of defect generation, and fully guarantees the fidelity of generated defects in terms of morphology and texture details while maintaining the characteristics of the target background domain, indicating that the pixel-level DBS plays a key role in improving the quality and fidelity of generated defects, and is better for warp and weft defects. This not only guarantees the high authenticity of industrial defect generation, but also gives it cross-background domain portability. The process of defect conversion between different images can be represented as: , , wherein, represents the input defect window image, represents the first-order defect window image generated directly by the improved defect generator, represents the pixel-level defect mask generated by DBS, and the final generated defect window image is and are respectively dot product operation and fusion result, denotes spatial multiplication.

[0051] This migratability enables the IFD-GAN model to flexibly adapt to different fabric backgrounds when generating longitudinal and latitudinal defect images, significantly enhancing the diversity of generated defect images. Thanks to the pixel-level fabric defect-background micro-detacher, IFD-GAN can not only accurately capture local detail information of defects when generating longitudinal and latitudinal defects, but also fully consider the fusion relationship between defect texture and background, thereby realizing more natural and faithful defect image generation with industrial application value.

[0052] In the present application, a longitudinal and latitudinal self-attention mechanism (LLSA) is also introduced.

[0053] Specifically, the industrial fabric defect has obvious cross-scale characteristics, and is mostly a longitudinal and latitudinal penetrating fabric defect with a large aspect ratio. If a conventional N x N convolution is still used to obtain the feature of such target defect, it will inevitably cover a high proportion of irrelevant background area, and also disperses the attention to the detail features of the target defect, making it difficult to accurately extract the effective texture features of the industrial defect. To meet the special requirements of the IFD-GAN industrial-level generation algorithm for focusing on longitudinal and latitudinal defect details and high-fidelity generation, a special self-attention mechanism is designed for the typical features of longitudinal and latitudinal defects to match the characteristics of the target defect with a large aspect ratio, which is named longitudinal and latitudinal self-attention, and its structure is shown in Figure 6 This one-dimensional self-attention focuses on encoding long-range context along the horizontal or vertical spatial dimension, and cuts the input sequence along different dimensions, calculates the attention in each dimension and sums it to obtain the global attention. This longitudinal and latitudinal one-dimensional self-attention greatly reduces the computational complexity of the model and allows it to perform attention operations in a larger or even global area. To address the differences in individual size and appearance of longitudinal and latitudinal target defect areas, a longitudinal and latitudinal strip convolution of 1 x N or N x 1 is used before each branch to encode multi-scale context information along the spatial dimension to better locate the target defect area. During the fusion of the longitudinal and latitudinal attention results obtained by the two parallel branches, the weight of each spatial position will be based on the combination of importance in the longitudinal and latitudinal directions. Finally, the weight is obtained through 1 x 1 convolution and Softmax, and then weighted summed with the original feature map to obtain the output feature map.

[0054] The longitudinal and latitudinal self-attention mechanism proposed in the application can effectively focus on the target defect detail features, accurately extract defect texture information, and effectively avoid the large amount of redundant interference and attention dispersion phenomenon caused by the conventional convolution, thereby improving the model operation efficiency. The longitudinal and latitudinal self-attention mechanism is integrated into the feature extraction link after the IFD-GAN coding and follows the residual mode, and a new longitudinal and latitudinal feature extraction function is constructed. The longitudinal and latitudinal self-attention mechanism can efficiently cooperate with the pixel-level fabric defect-background microstripper to realize longitudinal and latitudinal defect detail extraction, and make an important contribution to high-fidelity defect generation.

[0055] Therefore, in the actual implementation process, the parameter amount of the basic model is 14703300, and the parameter amount of the model after only adding the designed LLSA is 12945366, which decreases by nearly 12%. It can be seen that the lightweight design of compressing the channel number of the feature map by 1x1 convolution in the LLSA upsampling process can locally limit the increase of the calculation complexity. In addition, the training time of the model after adding the LLSA is reduced by 8.24%. Essentially, although the LLSA is a kind of self-attention mechanism and is added multiple times in the defect generator of the model, it not only does not increase the calculation parameter amount, but also reduces it by 11.96%, and saves about 8.24% of the training time. This is mainly due to the targeted design of the LLSA which utilizes the longitudinal and latitudinal strip convolution of 1xN or Nx1 to encode in parallel along the spatial dimension. The longitudinal and latitudinal attention decomposition and focusing along one-dimensional distribution and the unique operation strategy of the LLSA not only alleviate the calculation complexity of the model, reduce the parameter amount, but also effectively improve the calculation efficiency.

[0056] In the application, a structure similarity loss function based on PatchNCE is also designed. There is an essential difference between the fabric defect image generation and the natural image generation studied in the technical research. Under the mutual information contrast learning mechanism, the features of the targets in the natural image are obviously different, and the relationship between the targets and the background is clearly segmented. As shown in (a) of Figure 7 , the features of the horse's head and legs are obviously different, and the horse and the grass are easily segmented, so the mutual information is strong. In the fabric defect image generation task, as shown in (b) of Figure 7 , if the PatchNCE method is used, the region block at a random position in the generated image is selected as a positive sample, and other region blocks are selected as negative samples, then the mutual information is very weak. Therefore, considering the particularity of the same features at different positions of the defect, if the cross-entropy loss of the PatchNCE strategy is still used, the adjustment direction of the feature distance in the feature space between the positive and negative samples and the selected features will be confused, thereby seriously affecting the fidelity of the defect generation.

[0057] In fact, in this application, the patch structure similarity in the same position of the image is more worthy of attention. Therefore, based on the multi-layer and block-based idea and the use of PatchNCE multi-layer features, the feature structure similarity is further considered to strengthen the feature contrast learning effect under the mutual information mechanism. In view of the special nature of the task of generating cross-scale fabric defect images, in order to match the typical features of industrial applications and actual needs, a special loss function is designed based on the PatchNCE loss strategy to regulate the defect generation details and their consistency with the background texture, striving to achieve high-fidelity defect image generation.

[0058] Since there is little difference in the features of each part of the fabric defect, if the PatchNCE strategy is still used, it cannot narrow the distance between the selected features and the positive samples in the feature space, while pushing away the distance from the negative samples. In order to realize high-fidelity generation of defects, the patch structure similarity in the same position is more worthy of attention. However, if L1 or L2 distance is used as the loss function, it is easy to cause the model to pay too much attention to pixel-level accurate matching, while ignoring the overall correlation and consistency of the image. In view of the special nature of the task, IFD-GAN uses the method of cosine similarity to mark the randomly selected patch of the generated image as X and the patch of the input image in the same position as Y to calculate the structural similarity between the features, so as to ensure the convergence direction of the distance between the positive and negative samples and the selected features in the feature space. The calculation formula is as follows: , , , , , , , , , , , , , -2 , ,

[0059] Finally, considering the special nature of the same features at each position of the defect under the PatchNCE negative sample strategy, only the introduction of patch structure similarity loss can strengthen the feature contrast learning effect under the mutual information mechanism, while also helping IFD-GAN to learn more robust feature representation, overcoming mode collapse and improving its training stability. Therefore, the structure similarity contrast loss function based on improved PatchNCE is specially designed for IFD-GAN may be expressed as: wherein, , are weight coefficients, the value range of the weight coefficients is 10 -2 to 10, to optimize the coupling loss as a whole, and model the multiple loss dependency relationships.

[0060] In the present application, after the high-fidelity flaw image block is generated, a plurality of evaluation methods are set, which are as follows: 1. High-fidelity evaluation: two objective indexes of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to evaluate and quantify the quality and fidelity of the generated image.

[0061] The peak signal-to-noise ratio is the ratio of the maximum possible power of a signal to the power of the destructive noise affecting its representation accuracy, reflecting the size of the image pixel error, commonly denoted by n. Since many signals have a very wide dynamic range, the peak signal-to-noise ratio is often expressed in logarithmic decibel units. The peak signal-to-noise ratio is often used as a measurement method for signal reconstruction quality in the field of image compression, which is simply defined by the mean square error (MSE). Two m x n monochrome images I and K, if one is the noise approximation of the other, then the mean square error is defined as: And the PSNR is often expressed in logarithmic form: wherein, is the image size, is the original image, is the reconstructed image; denotes the maximum value of the image point color, if each sampling point is represented by 8 bits, then it is 255. A more general representation is that if each sampling point is represented by B bits of linear pulse code modulation, then is 2B-1. For a color image with RGB values for each point, the definition of the peak signal-to-noise ratio is similar, except that the mean square error is the sum of all variances divided by the image size and then divided by 3.

[0062] Typically, the PSNR value is between 30 and 40 dB. A PSNR in this range means that the generated image has less distortion and better quality. The relationship between different PSNR value ranges and image distortion levels and quality is described in the following table: Table: Relationship between PSNR value range and image quality SSIM is used to measure the similarity between images and reflect the degree of distortion. As an implementation of structural similarity theory, structural similarity index defines structural information as independent of brightness and contrast from the perspective of image composition, reflects the properties of object structure in the scene, and models distortion as a combination of three different factors of brightness, contrast and structure. The mean value is used as the estimate of brightness, the standard deviation is used as the estimate of contrast, and the covariance is used as the measure of structural similarity.

[0063] The calculation formula of SSIM is: wherein, and and and are the mean values of the images I and J in the direction, is the covariance of I and J, and are the standard deviations, , wherein is the dynamic range of pixel value, k1=0.01, k2=0.03. The range of SSIM is-1 to 1, and the closer the value is to 1, the higher the similarity and the smaller the distortion. When two images are exactly the same, the value of SSIM is equal to 1.

[0064] The above PSNR and SSIM indexes are calculated for the whole image. In order to further verify the high fidelity of the large aspect ratio of the IFD-GAN generated longitudinal and latitudinal through-type defects and local defects, spatial and frequency domain analysis is carried out for the industrial high-resolution image during evaluation. Image spatial analysis can effectively display the pixel-level texture and spatial information of the defect, and frequency domain analysis can reflect the gray scale change caused by the defect in the image and extract the high-frequency defect texture component. The spatial and frequency domain analysis of the defect image can further intuitively evaluate the local details of the defect on the basis of the PSNR and SSIM indexes, and then comprehensively verify the high fidelity of the IFD-GAN generated defects.

[0065] 2. Diversity evaluation: In order to evaluate the diversity of the IFD-GAN enhanced ZD001_FD data set from multiple angles, the generated defect size, morphology, position and background matching relationship are shown.

[0066] To evaluate the diversity of the data set enhanced by IFD-GAN, the size, appearance, location, background and matching relationship of the generated industrial fabric defects can be displayed. Multi-angle display and evaluation of defect diversity of the data set is crucial for building a powerful, reliable and universal deep learning defect detection model, which directly affects the performance and application value of the model. The excellent diversity of the defect data set provides key materials for model training, which helps to comprehensively improve the robustness of the model, accelerate the convergence of the model and effectively avoid overfitting. The generalization ability of the model directly determines the effect of industrial fabric defect detection and its practical value for industrial applications.

[0067] 3. Detection effect evaluation: To prove the positive effect of the industrial fabric defect images generated by IFD-GAN on the actual detection accuracy and speed after being used to enhance and balance the original data set, the accuracy Precision (P / %), recall Recall (R / %), and mean Average Precision (mAP / %) are selected as indicators to evaluate the detection accuracy before and after enhancement. Frame rate FPS (f / s) is selected as the evaluation index of detection speed, which is commonly used to represent the real-time detection capability of the model. The calculation formula is as follows: ， ， ， ， , , wherein is the number of correctly identified defect samples, is the number of negative samples that are incorrectly identified as defect samples, i.e. false positives; is the number of defect samples that are incorrectly identified as negative samples, i.e. false negatives; is the total number of classes, is the average precision, which is the area enclosed by the PR curve and the coordinate axes with recall R as the horizontal axis and accuracy P as the vertical axis, is the intersection over union, and represent the predicted box and the true box respectively; is the number of pictures processed by the model, is the total time consumed.

[0068] Therefore, as shown in FIG. 1, Figure 8 in some embodiments, the application designs a cross-scale fabric defect sample generation method based on a contrast learning mutual information mechanism, which comprises: S1: Construct a contrastive learning mutual information network containing a generator and a discriminator, construct a pixel-level fabric defect-background micro-separator for extracting fine-grained texture information of large aspect ratio warp and weft defects and generating a pixel-level defect mask.

[0069] S2: Set a structure similarity loss function based on PatchNCE and combine the contrastive learning mutual information network, the pixel-level fabric defect-background micro-separator and the warp and weft attention mechanism to generate an IFD-GAN fabric defect image generation model.

[0070] S3: According to the collected fabric image, an input image block of a preset size is extracted; wherein the preset size is set by a designer according to actual needs, for example, 256x256.

[0071] S4: The IFD-GAN fabric defect image generation model is used to perform defect image separation on the input image block to generate a high-fidelity defect image block.

[0072] S5: The generated high-fidelity defect image block is embedded into the original industrial size fabric image through a sliding window method to output an industrial size target fabric defect image.

[0073] Thus, with reference to Figure 9 It is shown that in some embodiments, the present application also relates to a cross-scale fabric defect sample generation system based on a contrastive learning mutual information mechanism using the above method, comprising: A defect model construction module 201 is configured to construct a contrastive learning mutual information network containing a generator and a discriminator, construct a pixel-level fabric defect-background micro-separator for extracting fine-grained texture information of large aspect ratio warp and weft defects and generating a pixel-level defect mask, set a structure similarity loss function based on PatchNCE and combine the contrastive learning mutual information network, the pixel-level fabric defect-background micro-separator and the warp and weft attention mechanism to generate an IFD-GAN fabric defect image generation model.

[0074] A fabric image processing module 202 is configured to extract an input image block of a preset size according to a collected fabric image.

[0075] A defect image generation module 203 is configured to use the IFD-GAN fabric defect image generation model to perform defect image separation on the input image block to generate a high-fidelity defect image block, and then embed the generated high-fidelity defect image block into the original industrial size fabric image through a sliding window method to output an industrial size target fabric defect image.

[0076] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0077] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A cross-scale fabric defect sample generation method based on contrastive learning mutual information mechanism, characterized by: The following steps are involved: S1: Construct a contrastive learning mutual information network consisting of a generator and a discriminator, and build a pixel-level fabric defect-background micro-stripper for extracting fine-grained texture information of large aspect ratio warp and weft defects and generating pixel-level defect masks; S2: Set the PatchNCE-based structural similarity loss function and combine it with the contrastive learning mutual information network, pixel-level fabric defect-background micro-stripper and warp and weft attention mechanism to generate the IFD-GAN fabric defect image generation model; S3: extracting an input image block of a preset size according to the collected fabric image; S4: using the IFD-GAN fabric defect image generation model to perform defect image stripping on the input image block to generate a high-fidelity defect image block; S5: The generated high-fidelity defect image block is embedded into the original industrial-sized fabric image through a sliding window method, and the industrial-sized target fabric defect image is output.

2. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 1, characterized in that: When constructing a contrastive learning mutual information network including a generator and a discriminator, the method includes: S11: For input and output image domains ,in , ,in and Represents any image element in the image domain X and Y, H and W represent the image resolution height and width, C represents the number of channels, Represents a collection of images; S12: Set PatchNCE loss function: ,in , Indicates the l The number of spatial locations of the layer, , respectively represent the corresponding position features of the input and output images, , represents other location features, where is the number of feature channels per layer, represents a multilayer perceptron; S13: Setting the contrast loss function with PatchNCE as the core : ,in For classic adversarial loss: ,in represents the model builder, represents the model discriminator, represents the average value, D(y) represents the probability of the real sample, G(x) and D(G(x)) represent the false images generated by the generator and their probabilities, respectively. 、 are weight coefficients, and the value range is set to 10 -1 to 10 2 .

3. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 2, characterized in that: When constructing a contrastive learning mutual information network including a generator and a discriminator, the method further includes: S16: Calculate cross entropy loss: ,in , is the feature vector of the random position area block of the output image, is the feature vector of the area block at the same position in the input image, , is the feature vector of the remaining area block of the input image, It is a scale hyperparameter used to control key scale relationships such as model structure, data distribution, or regularization strength.

4. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 3, characterized in that: When constructing a pixel-level fabric defect-background microscopic stripper, the method includes: S101: Setting the pixel-level fabric defect-background microscopic stripper to a U-net structure; S102: In the upsampling process, the CARAFE operator is introduced and the number of feature map channels is compressed by 1×1 convolution, and feature map group convolution is performed; S103: Perform sub-pixel convolution on the output feature map, split the number of channels of each pixel into a preset number of sub-pixel channels and rearrange them, expand the channel dimension in the spatial dimension, and obtain an upsampled feature map after normalization using softmax, and then perform feature fusion on the input feature map and the upsampled feature map; S104: Set up skip connections to pass shallow features of the encoder directly to the decoder and introduce a spatial channel attention mechanism in the depth channel; S105: Set the defect conversion process formula between different images: ， ， ,in, represents the input defect window image, represents the first-order defect window image directly generated by the improved defect generator, Represents the pixel-level defect mask generated by DBS, and the final defect window image for and Respectively The result of fusion after dot product operation, Represents spatial multiplication.

5. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 4, characterized in that: When combined with the warp and weft attention mechanism, the method includes: S21: Obtain the intermediate feature map of the fabric image from the generator as the input of the warp and weft self-attention mechanism; S22: Split the input feature map along the longitudinal and latitudinal spatial dimensions to form two branches for processing the longitudinal and latitudinal information separately, and apply 1×N convolution in the longitudinal branch and N×1 convolution in the latitudinal branch; S23: Perform one-dimensional attention operations on the features of the corresponding directions in the two branches respectively, calculate the weight response of each spatial position, and obtain the longitudinal and latitudinal attention results; S24: Use 1×1 convolution and Softmax function to obtain the attention weight and add the longitudinal and latitudinal attention results to the input feature map to obtain the output feature map; S25: Perform a residual connection between the output feature map and the original input feature map, output the enhanced warp and weft attention features, and then pass the warp and weft self-attention enhanced feature map to the pixel-level fabric defect-background micro-stripper.

6. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 5, characterized in that: When setting the PatchNCE-based structural similarity loss function, the method includes: S21: Set the formula for calculating the structural similarity between features: , ,in and represent the average values ​​of X and Y respectively. and represent the variance of X and Y respectively, represents the covariance of X and Y, and is a constant, , , where L is the dynamic range of pixel values, 、 Can be taken from 10 according to dynamic pixels -2 to 1; S22: Set the PatchNCE multi-layer feature calculation formula: , and then set the structural similarity loss function based on PatchNCE: ,in, 、 are weight coefficients respectively, and the value range of weight coefficient is 10 -2 To 10.

7. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 1 or 6, characterized in that: After the high-fidelity defect image block is generated, the method further includes: Peak signal-to-noise ratio and structural similarity indicators are used to evaluate the quality and fidelity of generated images; The detection accuracy before and after enhancement is comprehensively evaluated using the mean average precision index when the precision, recall and IoU are set to 50% and 50%:95% respectively, and the detection speed is evaluated using the frame rate.

8. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 7, characterized in that: When using peak signal-to-noise ratio and structural similarity indicators to evaluate the quality and fidelity of the generated image, the method includes: The peak signal-to-noise ratio is defined by the mean square error: , and then use the logarithm to express the peak signal-to-noise ratio: ,in, is the image size, is the original image, To reconstruct the image, Indicates the maximum color of the image point; The structural similarity index evaluation uses the mean as the estimate of brightness, the standard deviation as the estimate of contrast, and the covariance as the measure of structural similarity. The calculation formula is: ,in, and Represent the mean of the image in the I and J directions respectively, and is the standard deviation, is the covariance of I and J, and is the stabilization constant, , ,in is the dynamic range of pixel values, k1=0.01, k2=0.

03.

9. The method for generating cross-scale fabric defect samples based on contrastive learning mutual information mechanism according to claim 7, characterized in that: When comprehensively evaluating the detection accuracy and detection speed before and after enhancement and using frame rate to evaluate the detection speed, the method includes: The accuracy index calculation formula is: ,in To determine the correct number of defect samples, is the number of negative samples that are mistakenly judged as defective samples, that is, the number of false detections; The calculation formula of the recall rate indicator is: ,in is the number of defective samples that are mistakenly judged as negative samples, that is, the number of missed detections; The calculation formula for the mean average precision index when the IoU is set to 50% and 50%:95% is: , , ,in is the total number of categories, is the average precision, which is the area enclosed by the PR curve with the recall rate R as the horizontal axis and the accuracy rate P as the vertical axis and the coordinate axis. is the intersection and union ratio, and Represent the predicted box and the true box respectively; The frame rate indicator calculation formula is: ,in The number of images processed by the model, is the total elapsed time.

10. A cross-scale fabric defect sample generation system based on contrastive learning mutual information mechanism using the method according to any one of claims 1 to 9, characterized in that: include: The defect model construction module is used to build a contrastive learning mutual information network consisting of a generator and a discriminator, and to construct a pixel-level fabric defect-background micro-stripper for extracting fine-grained texture information of large aspect ratio warp and weft defects and generating pixel-level defect masks. The IFD-GAN fabric defect image generation model is generated by setting a PatchNCE-based structural similarity loss function and combining the contrastive learning mutual information network, the pixel-level fabric defect-background micro-stripper, and the warp and weft attention mechanism. A fabric image processing module, configured to extract an input image block of a preset size based on the collected fabric image; The defect image generation module is used to use the IFD-GAN fabric defect image generation model to perform defect image stripping on the input image block to generate a high-fidelity defect image block; the generated high-fidelity defect image block is then embedded into the original industrial-sized fabric image through a sliding window method to output an industrial-sized target fabric defect image.

Images