Fabric image pattern detection and restoration method based on multi-model fusion

Abstract

The invention discloses a fabric image pattern detection and restoration method based on multi-model fusion, and the method comprises the steps: collecting a large number of images with fabric patterns, constructing an annotation data set, and carrying out the automatic feature modeling of the fabric patterns through employing an improved YOLO11 model. The trained target detection model can quickly identify the positions of various patterns in the fabric image, and provides accurate regional information for subsequent instance segmentation and image restoration. The instance segmentation model carries out accurate mask extraction on the target area, and the boundary of the pattern area can be accurately extracted from the fabric image. And inputting the mask of the pattern area into an image restoration model, and performing image restoration and filling. According to the invention, through deep fusion target detection, instance segmentation and image restoration technologies, pattern identification, region extraction and background restoration in a complex background are realized. Not only is the image processing precision improved, but also the feasibility of application scenes such as fabric pattern analysis, pattern elimination and image enhancement is expanded.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and image processing, and particularly relates to a fabric image pattern detection and repair method based on multi-model fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence, especially deep learning technology, computer vision has been widely applied in multiple industrial fields. In the textile industry, the automatic recognition and intelligent processing of fabric images have gradually become key technical means to promote production efficiency and product quality improvement. Especially in the aspects of textile detection, pattern design, garment manufacturing and quality control, the demand for accurate recognition and repair of fabric patterns is increasingly prominent. However, due to the diversity and complexity of fabrics and the noise interference in the image acquisition process, traditional image processing methods face many challenges in practical applications.

[0003] Traditional fabric image processing methods usually rely on manually designed feature extraction algorithms, such as edge detection, texture analysis or frequency domain transformation techniques. Such methods perform well when dealing with simple structure and regular texture fabric images, but when processing complex background, irregular patterns, rich colors or multi-scale patterns, they often fail to achieve accurate and effective recognition and repair. Specifically, these methods are easily disturbed by factors such as light changes, folding shadows, occlusions, and pattern overlaps, resulting in inaccurate pattern segmentation, blurred boundaries, unnatural texture after repair, and even background residues, which seriously affect the subsequent image analysis and application results.

[0004] In recent years, breakthroughs in deep learning technology have greatly promoted the development of computer vision tasks, especially in object detection, semantic segmentation, instance segmentation, and image repair. The introduction of convolutional neural network (CNN), generative adversarial network (GAN), and transformer architecture enables models to automatically learn complex feature representations from large amounts of data and has strong robustness and generalization ability. These technologies have been widely applied in medical image analysis, autonomous driving, remote sensing image processing and other fields.

[0005] However, existing research work mostly focuses on single task scenarios, such as only image segmentation or only image repair, lacking comprehensive solutions for the special application scenario of pattern processing in fabric images. In the real application scenario of fabric images, the distribution of patterns usually presents high density, multi-direction and multi-scale characteristics, and single segmentation or repair method is difficult to meet the actual demand. In addition, existing methods rarely consider the collaborative optimization between recognition, segmentation and repair processes, resulting in low efficiency of the overall processing flow and difficulty in meeting the actual requirements of precision and naturalness. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a pattern recognition and repair integrated processing method for fabric images, which can realize accurate recognition, effective segmentation and natural repair of the pattern region in a complex fabric image, thereby improving the automation level and the final repair effect of image processing, and solving the problems of inaccurate recognition, fuzzy boundary and unnatural repair in the prior art in fabric pattern processing.

[0007] The technical solution adopted by the present application to solve the technical problem is: a fabric pattern processing method based on deep learning, which fuses target detection, instance segmentation and image repair modules to construct an end-to-end integrated model, specifically including the following steps:

[0008] Obtaining a pattern image;

[0009] Constructing a deep learning model for detecting and repairing target patterns in the pattern image; the deep learning detection model is constructed based on three network models, including:

[0010] A first target detection network is constructed based on a YOLO11 image detection model, which is used to detect target patterns in the pattern image and obtain target bounding boxes;

[0011] A second image segmentation network is constructed based on a SAM image segmentation model, which is used to extract target images according to the target bounding boxes, and then accurately extract masks from the target images, so as to accurately obtain the pattern boundary in the fabric image;

[0012] A third image repair network is constructed based on a LAMA image repair model, which uses the original image and the pattern mask image to repair the pattern image, generates a clear background image without patterns, and realizes a natural transition effect.

[0013] Further, the YOLO11 network includes:

[0014] A C3k2 module is introduced to improve the network calculation efficiency.

[0015] A C2PSA module combining CSP segmentation and PSA attention is introduced to improve the attention ability to key areas while making the model more lightweight.

[0016] Further, the C3k2 module mainly refers to:

[0017] The input feature map is first expanded to twice the channel through a convolution layer, and then is cut into two parts along the channel dimension. The first part is reserved for jump connection, and the second part is sequentially passed through multiple stacked sub-modules (such as C3k module). The output of each sub-module is saved. Finally, the jump connection and the outputs of all sub-modules are spliced in the channel dimension, and an output feature is generated through a convolution layer.

[0018] Further, the C2PSA module refers to:

[0019] The C2PSA module is an improved convolution structure fused with position-sensitive attention (PSA), which has stronger feature modeling and enhancement capabilities, mainly including three stages of feature separation, attention enhancement and feature fusion. Among them, the feature map is divided into two parts, one part is directly transmitted, and the other part is processed through the PSA attention module, and finally spliced and fused. To improve the attention ability to key areas while making the model more lightweight.

[0020] Let the input feature map be:

[0021] X∈R B×C×H×W

[0022] First, the channel compression and separation are performed:

[0023] Use a 1x1 convolution to compress the channel to 2c, and then separate it according to the channel:

[0024] X1=Conv 1×1 (X)∈R B×2c×H×W

[0025] X=[A,B],A,B∈R B×c×H×W

[0026] Send B to multiple stacked PSABlock for feature modeling:

[0027] B'=PSABlock n (···(PSABlock2(PSABlock1(B))))

[0028] Finally, A and B' are spliced along the channel dimension:

[0029] X2=Concat(A,B')∈R B×2c×H×W

[0030] Use a 1x1 convolution to restore the channel to the original input channel number C:

[0031] Y=Conv 1×1 (X2)∈R B×C×H×W

[0032] Further, the image segmentation network is constructed based on a SAM (Segment Anything Model) image segmentation model, including:

[0033] The SAM model receives the target bounding box output by the target detection network as prompt information, and extracts the target image region according to the bounding box;

[0034] The SAM model generates an accurate pattern mask based on image content and boundary cues, and the pattern mask is used to depict the boundary shape of the pattern region in the fabric image.

[0035] Further, the image inpainting network is constructed based on the LAMA (Look At My Assumptions) image inpainting model, and the image inpainting network comprises,

[0036] The LAMA model uses the pattern image and its corresponding mask image to automatically predict the context information of the missing region and generate filling content,

[0037] The inpainting process realizes semantic consistency and texture continuity through a multi-scale context fusion module to generate a background image with natural transition.

[0038] Further, the target detection model is constructed by the following method:

[0039] Obtain a plurality of training image samples;

[0040] Construct a target detection model based on the number of preset detection categories;

[0041] Label the class label and position label of the preset detection category in each training image sample using a rectangular frame;

[0042] Analyze the training image samples using the target detection model to obtain a prediction result;

[0043] Compare the prediction result with the class label and the position label, and optimize the network parameters of the first target detection network and the second target detection network according to the comparison result.

[0044] The present application proposes a fabric pattern processing method based on deep learning, which is an end-to-end solution integrating multiple deep learning technologies. The method integrates key modules such as target detection, instance segmentation and image inpainting, and realizes intelligent recognition, accurate extraction and high-quality removal of patterns through a unified architecture, which can significantly improve the automation level and result quality of fabric image processing. The proposed technical solution not only improves the processing efficiency and accuracy, but also provides strong support for the digital upgrade of the textile industry, and has wide application prospect and important technical popularization value.

[0045] Advantages

[0046] Compared with the prior art, the application has stronger adaptability, accuracy and intelligent level in pattern recognition, segmentation and repair of fabric images. By introducing a multi-task cooperative mechanism in deep learning, the application effectively fuses three sub-tasks of target detection, instance segmentation and image repair, and realizes accurate identification, fine segmentation and natural removal of complex pattern regions in fabric images in the same framework, significantly improving the efficiency and quality of the overall processing. Unlike traditional processing methods based on artificial features or single networks, this method can still maintain stable and high-precision processing performance when facing actual problems such as diverse pattern shapes, complex color changes and strong background interference, and has stronger robustness and universality. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a structural schematic diagram of the YOLO11 module of the application;

[0048] Figure 2 is another structural schematic diagram of the YOLO11 module of the application;

[0049] Figure 3 is a flowchart of the first embodiment of the application. DETAILED DESCRIPTION

[0050] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope defined by the appended claims of the application.

[0051] The application provides a fabric image pattern detection and repair method based on multi-model fusion, which fuses target detection, instance segmentation and image repair modules to construct an end-to-end integrated model, specifically including the following steps:

[0052] Step S1: model training: mainly complete annotation of the same data set, including annotation of target pattern, put the data set and annotation results into the model for training; the specific steps are as follows:

[0053] Step S1.1: after obtaining the image set, use a rectangular frame to annotate the class and position coordinates of each picture;

[0054] Step S1.2: use the YOLO11 network to train the picture; specifically including:

[0055] Step S1.2.1: the YOLO11 uses a C3K2 module, as shown in Figure 1As shown, the input feature map first passes through a convolutional layer to generate twice the number of channels, and then is split into two parts along the channel dimension. The first part is reserved as a jump connection, and the second part is sequentially passed through multiple stacked sub-modules (such as C3k modules), and the output of each sub-module is saved. Finally, the jump connection and the outputs of all sub-modules are spliced in the channel dimension, and a convolutional layer is used to fuse and generate the output features.

[0056] Specifically, the calculation process of the C3k2 module is represented as:

[0057] First, the feature map X first passes through the first convolutional layer Conv1, which expands the channels of the input to obtain the feature map:

[0058] X' = Conv1(X)

[0059] Then X' is split into two parts along the channel dimension:

[0060] X' = [Y1, Y2]

[0061] Where Y1 is reserved as a jump connection part, and Y2 is used as the input of the subsequent module.

[0062] Next, Y2 is sequentially passed through n stacked sub-modules, each of which is a C3K module or a Bottleneck module:

[0063] Z1 = M1(Y2), Z2 = M2(Z1),... Z n = M n (Z n-1 )

[0064] The split Y1 and all processed Z i are spliced in the channel dimension:

[0065] Z = Concat(Y1, Z1, Z2,..., Z n )

[0066] Finally, the spliced features are fused and output by the second convolutional layer Conv2:

[0067] Y = Conv2(Z)

[0068] Step S1.2.2: The C2PSA module combining CSP segmentation and PSA attention is introduced in YOLO11 to improve the attention ability to key regions and make the model more lightweight. As shown in Figure 2As shown, the feature map is divided into two parts, one part is directly transmitted, and the other part is processed through the PSA attention module, and finally spliced and fused. The PSA attention module processing can be stacked multiple times PSA (pyramid slice attention) module, which is stacked and improved on the basis of SE attention mechanism, and the attention ability to key areas is improved.

[0069] Let the input feature map be:

[0070] X∈R B×C×H×W

[0071] First, the channel compression and separation are performed:

[0072] A 1x1 convolution is used to compress the channel to 2c, and then the channel is separated:

[0073] X1=Conv 1×1 (X)∈R B×2c×H×W

[0074] X=[A,B],A,B∈R B×c×H×W

[0075] B is sent to multiple stacked PSA blocks for feature modeling:

[0076] B'=PSABlock n (···(PSABlock2(PSABlock1(B))))

[0077] Finally, A and B' are spliced along the channel dimension:

[0078] X2=Concat(A,B')∈R B×2c×H×W

[0079] A 1x1 convolution is used to restore the channel to the original input channel number C:

[0080] Y=Conv 1×1 (X2)∈R B×C×H×W

[0081] Step S1.3: Store the weights of the YOLO11 model generated by step S1.2 as a pth format file.

[0082] Step S2: As shown in Figure 3 , the coordinates of the target detection frame are obtained through the YOLO11 network, and the specific steps are as follows:

[0083] Step S2.1: Define the coordinate array boxes[], the label array class_ids[], and the confidence array confidences[];

[0084] Step S2.2: Object detection network initialization and image input preprocessing; specific steps include:

[0085] Step S2.2.1: Define the object detection model class ObjectDetection. In this step, load the YOLO pre-trained model through the __init__ function to perform object detection. When initializing the ObjectDetection class, load the pre-trained YOLO11 model (load through the YOLO(model_path) method).

[0086] Step S2.2.2: Image input preprocessing; process the input image to ensure it is in RGB format, and if it is a grayscale image, convert it to RGB format.

[0087] Step S2.3: Image prediction and bounding box coordinate extraction.

[0088] Step S2.3.1: Image size adjustment, adjust the size of the input image to ensure it meets the input requirements of the model (640x640 pixels).

[0089] Step S2.3.2: Use the YOLO model to perform object detection on the image and obtain the coordinates, class ID, confidence, etc. of the predicted bounding box.

[0090] Step S2.3.3: Use the non-maximum suppression (NMS) algorithm to remove duplicate detection boxes and keep the most representative target box. NMS will remove those boxes with high overlap and low confidence based on the confidence of each box, and only keep the most suitable box.

[0091] Step S2.4: Return the image and predicted bounding box information

[0092] Step S2.4.1: Return the image after object detection and the coordinates of each box for further processing.

[0093] Step S3: As shown in Figure 3 , obtain the mask of the image detection pattern through the SAM image segmentation network, and the specific steps are as follows:

[0094] Step S3.1: Image segmentation model initialization and image preprocessing.

[0095] Step S3.1.1: Image input needs to be preprocessed before entering the model. The specific steps are to convert the input image to RGB format, and if the input image is a grayscale image, it needs to be converted to RGB format to meet the requirements of network input.

[0096] ImageRGB = ConvertColor(GBR→RGB)

[0097] Step S3.2: target bounding box input and segmentation operation;

[0098] Step S3.2.1: prepare input box and image data,

[0099] In order to perform image segmentation, the coordinates of the target detection box need to be input together with the processed image to the SAM model. The input box coordinates are usually [x1, y1, x2, y2], which represent the target region detected in the image.

[0100] Step S3.2.2: run the SAM model for inference and obtain the mask.

[0101] The processed image and box coordinates are input to the SAM model for inference. The model outputs the precise mask of the target region according to the input box position and image content.

[0102] Step S3.2.3: extract the predicted mask and convert it to a binary image.

[0103] The mask is extracted from the model output and converted to a standard binary image format (mask region is 1, other regions are 0) for subsequent operations.

[0104] The generated segmentation mask can be represented as a binary image M(x, y) defined as follows:

[0105]

[0106] Step S4: repair the binary mask obtained from the original image and step S3 by the LAMA image inpainting network, the specific steps are as follows:

[0107] Step S4.1: image and mask preprocessing;

[0108] Step S4.1.1: original image and mask reading and size normalization:

[0109] Input image:

[0110] I∈R H×W×3

[0111] It is a three-channel color image.

[0112] Mask image:

[0113] M∈{0,1} H×W

[0114] Where 1 represents the repair area (generated by SAM), and 0 represents the reserved area.

[0115] If the original image size is not the target model input size H x W, use bilinear interpolation:

[0116] I resized = Resize(I, H, W), M resized = Resize(M, H, W)

[0117] Where bilinear interpolation is used for resizing, keeping the image structure and mask edge smooth.

[0118] Step S4.1.1: Construct the inpainting input image (masking with mask)

[0119] I masked = I O (1 - M resized )

[0120] Where O denotes element-wise multiplication, i.e., zeroing the area that needs to be inpainted.

[0121] Step S4.2: LaMa inpainting network inference;

[0122] Step S4.2.1: Feature extraction (encoder stage)

[0123] Take the mask image as a double input, extract local and global features through the LaMa encoder:

[0124] F = Encoder(I masked , M resized )

[0125] LaMa uses attention modules based on Fourier Convolution to enhance the ability to model long-distance context information.

[0126] Step S4.2.2: Inpainting image generation;

[0127] Generate the inpainting image output using the features F extracted by the encoder:

[0128] I' = Decoder(F)

[0129] Step S4.3: Inpainting image post-processing

[0130] Step S4.3.1: Image denormalization

[0131] LaMa model input is usually standardized, so the output needs to be denormalized:

[0132] I output = I' · σ + μ

[0133] Where μ, σ are the ImageNet mean and standard deviation, respectively.

[0134] Step S4.3.2: Restore the original image size and save

[0135] I final =Resize(I output ,H,W)。

[0136] The application proposes a new fabric pattern processing method through deep learning technology, combining three modules of image recognition, instance segmentation and image inpainting, aiming to improve the efficiency and accuracy of pattern recognition, extraction and removal in fabric image processing, especially the application effect in complex patterns and background.

[0137] The fabric image pattern detection and repair method based on multi-model fusion of the application includes the following three main parts:

[0138] The first part is to collect a large number of images with fabric patterns, construct a labeled data set, and use an improved YOLO11 model to automatically model the fabric patterns. The trained target detection model can quickly identify the positions of various patterns in the fabric image, providing accurate area information for subsequent instance segmentation and image inpainting.

[0139] The second part uses an instance segmentation model (SAM) to accurately extract the mask of the target area. Through this step, the system can accurately extract the boundary of the pattern area from the fabric image, ensuring the accuracy and naturalness of subsequent repair.

[0140] The third part inputs the mask of the pattern area into the image inpainting model (LaMa) for image inpainting and filling. This model uses the context information of the image to perform semantic-level repair and background recovery on the pattern area, generating a clear background image without patterns and achieving a natural transition effect. This step significantly improves the shortcomings of traditional repair methods in complex backgrounds, providing a more realistic and detailed repair effect.

[0141] The application realizes pattern recognition, area extraction and background recovery in complex backgrounds by deeply integrating target detection, instance segmentation and image inpainting technology. This method not only improves the accuracy of image processing, but also expands the feasibility of application scenarios such as fabric pattern analysis, pattern elimination and image enhancement.

[0142] The protection content of the application is not limited to the above embodiments. Changes and advantages that can be thought of by those skilled in the art without departing from the spirit and scope of the application are included in the application, and the appended claims are protected.

Claims

1. A fabric image pattern detection and repair method based on multi-model fusion, characterized in that, The method comprises the following steps: acquiring a pattern image; constructing a deep learning model for detecting and repairing a target pattern in the pattern image; the deep learning detection model is constructed based on three network models, including: a first target detection network constructed based on a YOLO11 image detection model, used for detecting a target pattern in the pattern image and obtaining a target bounding box; a second image segmentation network constructed based on a SAM image segmentation model, used for extracting a target image according to the target bounding box, and then performing accurate mask extraction on the target image, so as to accurately obtain the pattern boundary in the fabric image; a third image repair network constructed based on a LAMA image repair model, used for repairing the pattern image by using an original image and a pattern mask image, generating a clear background image without patterns, and realizing a natural transition effect.

2. The multi-model fusion-based fabric image pattern detection and repair method according to claim 1, characterized in that, The target detection network is constructed by using a YOLOv11 network model.

3. The multi-model fusion based fabric image pattern detection and repair method according to claim 2, characterized in that, The C3k2 module is introduced into the YOLO11 network. Specifically, the calculation process of the C3k2 module is represented as: First, the feature map X is first subjected to a first convolutional layer Conv1, which expands the channels of the input to obtain a feature map: X' = Conv1(X) Then X' is split into two parts along the channel dimension: X' = [Y1, Y2] Where Y1 is reserved as a skip connection part, and Y2 is used as the input of the subsequent module; Then, Y2 is sequentially subjected to n stacked sub-modules, each of which is a C3K module or a Bottleneck module: Z1 = M1(Y2), Z2 = M2(Z1),... Z n = M n (Z n-1 ) Splitting out Y1 and all processed Z i In channel dimension concatenation: Z = Concat(Y1, Z1, Z2,..., Z n ) Finally, the second convolutional layer Conv2 is used to fuse and output the spliced features: Y = Conv2(Z).

4. The multi-model fusion based fabric image pattern detection and repair method according to claim 2, characterized in that, The C2PSA module combining CSP segmentation and PSA attention is introduced into the YOLO11 network: Let the input feature map be: X e R B×C×H×W First, the channels are compressed and separated: Use a 1x1 convolution to compress the channels to 2c, and then separate them by channel: X1= Conv 1×1 (X) ∈ R B×2c×H×W X = [A, B], A, B e R B×c×H×W Send B into multiple stacked PSABlock for feature modeling: B' = PSA Block n (···(PSABlock2(PSABlock1(B)))) Finally, A and B' are spliced along the channel dimension: X2 = Concat(A, B') e R B×2c×H×W Use a 1x1 convolution to restore the channel to the original number of channels C of the input: Y = Conv 1×1 (X2) ∈ R B×C×H×W Where the feature map is divided into two parts, one of which is directly transmitted, and the other of which is processed by the PSA attention module, and finally spliced and fused.

5. The multi-model fusion based fabric image pattern detection and repair method according to claim 1, characterized in that, The image segmentation network is constructed based on a SAM image segmentation model, The SAM model receives the target bounding box output by the target detection network as prompt information, and extracts the target image region according to the bounding box; The SAM model generates an accurate pattern mask based on image content and boundary prompts, which is used to depict the boundary shape of the pattern region in the fabric image; The generated mask can be represented as a binary image M(x, y), defined as follows: Where Ω is the detected pattern region.

6. The multi-model fusion based fabric image pattern detection and repair method according to claim 5, characterized in that, The SAM model uses a prompt control mechanism to achieve accurate segmentation of the pattern region, The prompt control mechanism includes: Based on the target bounding box, foreground prompt points are generated, and attention masks are formed combined with image semantic features, thereby improving the extraction accuracy of the pattern region boundary and ensuring the accuracy of boundary positioning in a complex fabric texture background.

7. The multi-model fusion based fabric image pattern detection and repair method according to claim 1, characterized in that, The image repairing network is constructed based on a LAMA image repairing model, context information of a missing area is automatically predicted and filling content is generated by the LAMA model by using the pattern image and a corresponding mask image, and the repairing process realizes semantic consistency and texture continuity through a multi-scale context fusion module to generate a background image with natural transition.

Images