Knitting pattern color code generation optimization method and device based on deep learning

Abstract

The present invention provides a knitting pattern color code generation optimization method and device based on deep learning. The present invention performs the following key optimizations to improve performance: multi-level feature fusion is performed on the RefinerNet refinement preprocessing network to improve the recognition ability of complex textures; the Img2Prog network is integrated with PANet to realize multi-level information reuse, and the traditional downsampling method is replaced by SPD convolution to effectively reduce the loss of structural information; the CBAM attention mechanism is introduced into the residual structure, the Resblock structure is optimized, and the accuracy of low-frequency color code generation is improved. Through a series of quantitative evaluation and qualitative analysis experiments on the improved algorithm, the improved algorithm has better accuracy. These optimizations not only improve the design efficiency, but also ensure the high quality and weavability of the pattern, meeting the needs of the modern textile industry for rapid response to market changes.

Description

Technical Field

[0001] The present invention belongs to the field of image generation, and in particular relates to a knitting pattern color code generation optimization method and device based on deep learning. Background Art

[0002] The current textile industry is undergoing an industrial transformation driven by intelligent manufacturing technology. The application of fully fashioned knitting technology enables modern knitting machines to automatically produce textiles with complex patterns and surface structures. Pattern design, as the core of the artistic expression of knitted fabrics, relies on plate-making software to realize the pattern color code design. This process requires not only the plate-makers to have a deep knowledge of textiles, but also to master the corresponding programming skills to cope with variable factors such as yarn and texture. Faced with the complexity of the production process and the dual demands for efficiency and cost, automated knitting pattern design has become a major research focus in the intersection of textiles and computer science.

[0003] Traditional knitting pattern design relies on fixed design rules, machine instructions, and physical simulation. Although Shida et al. provided many hand-knitted patterns, these methods are difficult to adapt to the needs of industrial knitting machines. Yuksel et al. designed complex patterns through stitch networks and realized yarn-level garment modeling. Wu et al. developed a fully automatic process to build stitch grids directly on three-dimensional models to ensure the knittability of patterns. McCann et al. introduced a parametric design tool based on shape primitives to achieve direct conversion from three-dimensional meshes to machine instructions. Hofmann et al.'s KnitPick system converts knitted images into machine and hand knitting instructions. The CAD tools developed by Kaspar et al. further simplified the process of creating and customizing knitted patterns. Although these technologies have achieved results in three-dimensional knitting design, they have not yet provided a simple solution for two-dimensional knitting image design.

[0004] With the development of deep learning technology, researchers have begun to explore its application in two-dimensional knitting image design. Melnyk et al. used generative adversarial networks (GANs) to create punched card patterns and converted knitting actions into binary patterns for actual knitting proofing, but these methods are limited to specific models. Wenhua et al. proposed a fabric parameter estimation model based on VGG19, which can estimate fabric parameters from a single fabric image and color it through a micro-piece bidirectional reflectance distribution function (BRDF) model. Scheidt et al. proposed a pattern color code generation method based on a long short-term memory network (LSTM), which serialized the color code and generated novel knitting patterns with high-level style specifications. However, this method has problems of randomness and unweavability in the generation of pattern color codes. Kaspar et al. proposed an image generation algorithm from knitting images to pattern color codes based on theoretical inspiration, but the generation results of low-frequency color codes are not ideal, and color code misalignment and missing are prone to occur.

[0005] In summary, the existing pattern color code generation method still has the problems of low accuracy and conversion efficiency. Summary of the invention

[0006] In view of the problems existing in the prior art, in order to improve the accuracy of pattern color code generation, the present invention has carried out the following key optimizations to improve performance: multi-level feature fusion of the RefinerNet refinement preprocessing network to improve the recognition ability of complex textures; integrating the Img2Prog network with PANet to achieve multi-level information reuse, and replacing the traditional downsampling method with SPD convolution to effectively reduce the loss of structural information; introducing the CBAM (Convolutional Block Attention Module) attention mechanism in the residual structure, optimizing the Resblock structure, and improving the accuracy of low-frequency color code generation. Through a series of quantitative evaluation and qualitative analysis experiments on the improved algorithm, the improved algorithm has better accuracy. These optimizations not only improve the design efficiency, but also ensure the high quality and weavability of the pattern, meeting the needs of the modern textile industry for rapid response to market changes.

[0007] The technical solution of the present invention is: a knitting pattern color code generation optimization method based on deep learning, comprising the following steps:

[0008] Step 1: Based on the collected knitted fabric image, the knitting plate making software draws the corresponding pattern color code map; the knitted fabric image is used as input, and the pattern color code image is used as a label to form a data set, and the data set is divided into a training set and a test set;

[0009] Step 2, randomly scaling, cropping, horizontally flipping and data preprocessing the knitted image data in the data set;

[0010] Step 3, constructing an InverseKnit++ model, wherein the InverseKnit++ model includes a RefinerNet refinement preprocessing network and an Img2Prog pattern color code generation network;

[0011] Step 4, using a RefinerNet refinement preprocessing network to refine the received knitted image, thereby obtaining a refined image with obvious loop structure and texture features; the RefinerNet refinement preprocessing network is a RefinerNet refinement preprocessing network that applies multi-scale fusion;

[0012] Step 5, predicting the pattern color code of the grayscale image through the Img2Prog pattern color code generation network, thereby obtaining a pattern color code image; the Img2Prog pattern color code generation network is an Img2Prog pattern color code generation network using an SPD convolution module and an attention mechanism module CBAM;

[0013] Step 6, using the preprocessed training set to train the constructed InverseKnit++ model, and inputting the test set into the trained InverseKnit++ model to generate a pattern color code image;

[0014] Step 7: Adjust the size of the pattern color code image by using the nearest neighbor interpolation algorithm without changing the number of color codes, and combine each color code with the corresponding knitting instruction to generate the final pattern color code image.

[0015] Furthermore, the RefinerNet refinement preprocessing network is built based on the feature pyramid network FPN, which includes three processes: bottom-up, top-down, and lateral connection. The bottom-up path is a convolutional network for feature extraction. In the process from bottom to top, the spatial resolution is downsampled to extract and compress more semantic feature information. Through the top-down path, FPN can reconstruct feature information. The lateral connection between the bottom-up and top-down channels is used to supplement high-resolution details and help locate the pattern of knitted fabrics.

[0016] The specific processing process of the bottom-up path is as follows:

[0017] The input knitted image passes through three groups of convolutional units consisting of convolutional layers and residual blocks, and one convolutional layer, and obtains feature maps p1-p4 respectively;

[0018] The specific process of horizontal connection is as follows:

[0019] The four feature maps p1, p2, p3, and p4 of different sizes are respectively passed through a horizontally connected convolutional layer to obtain feature maps c1, c2, c3, and c4;

[0020] The specific processing process of the top-down path is as follows:

[0021] Feature map n4, i.e. c4, first passes through upsampling layer 1, then performs unit addition with feature map c3 in summing layer 1, and then passes through convolution layer 5 to obtain feature map n3; feature map n3 first passes through upsampling layer 2, then performs unit addition with feature map c2 in summing layer 2, and then passes through convolution layer 6 to obtain feature map n2; feature map n2 first passes through upsampling layer 3, then performs unit addition with feature map c1 in summing layer 3, and then passes through convolution layer 7 to obtain feature map n1;

[0022] The size of feature map n4 is enlarged by 4 times, and the size of feature map n3 is enlarged by 2 times, and they are input into the Concatenation layer together with feature map n2. Feature map F2 is obtained through convolution layer 8. Feature map F2 is passed through upsampling layer 4, and then added with feature map n1 in summing layer 4. Feature map F1 is obtained through convolution layer 9. Feature map F1 is passed through upsampling layer 5, and then through convolution layer 10 to obtain the final output refined image.

[0023] Furthermore, the Img2Prog pattern color code generation network is composed of a backbone feature extraction network and a PANet feature fusion network. The attention mechanism module CBAM is introduced into the residual block of the backbone feature extraction network to form the RBCM residual, and the SPD convolution module is used instead of the cross-row convolution.

[0024] The specific processing process of the backbone feature extraction network is as follows:

[0025] First, the input refined image passes through the first convolution layer, the second convolution layer, the third convolution layer, and the residual block RBCM in sequence. The feature map Q1 is obtained after the first convolution layer, the feature map Q2 is obtained after the second convolution layer, the feature map Q3 is obtained after the third convolution layer, and the feature map Q4 is obtained after the residual block RBCM.

[0026] Then, feature maps Q1, Q2, and Q3 are respectively passed through a horizontally connected convolutional layer to obtain feature maps k1, k2, and k3;

[0027] Finally, feature map P3, i.e. k3, first passes through the first upsampling layer, performs unit addition with feature map k2 in the first summing layer, and obtains feature map P2 through the fourth convolution layer. Feature map P2 first passes through the second upsampling layer, performs unit addition with feature map k1 in the second summing layer, and obtains feature map P1 through the fifth convolution layer. The specific processing process of the PANet feature fusion network is as follows:

[0028] First, the feature map N1, that is, P1, first passes through the sixth convolution layer, and is unit-added with the feature map P2 in the third addition layer, and passes through the seventh convolution layer to obtain the feature map N2. The feature map N2 first passes through the eighth convolution layer, and is unit-added with the feature map P1 in the fourth addition layer, and passes through the ninth convolution layer to obtain the feature map N3.

[0029] Next, the feature maps N1, N2, and N3 are downsampled through the SPD modules with step sizes of 4, 2, and 1, respectively, and then input into the Concatenation connection layer together with the feature map Q4 through the jump connection for feature fusion. The feature map F is obtained through the tenth convolution layer, and then the output pattern color code image is obtained through the eleventh convolution layer.

[0030] Furthermore, the first convolution layer to the eleventh convolution layer each include a convolution kernel, a Relu activation function layer and a batch normalization BatchNorm layer.

[0031] Furthermore, the specific processing process of the SPD module with a step size of 2 is as follows:

[0032] The input feature map F is divided into 4 feature maps along the spatial dimension, and then the 4 feature maps are concat-connected along the channel dimension to obtain the feature map P. The feature map F is transformed from space to depth to obtain the feature map P, whose spatial dimension is reduced by 2 times and the number of channels is increased by 4 times. Then a convolution with a step size of 1 is applied to the feature map P to reduce the channel dimension to 1 / 4, and the processed feature map M is obtained.

[0033] Furthermore, the processing of the attention mechanism module CBAM is as follows:

[0034] The CBAM module contains two attention modules, channel and space. The channel attention mechanism firstly processes the feature map F of size C×H×W through two different pooling methods, namely, global maximum pooling (MaxPool) and global average pooling (AvgPool), to obtain two C×1×1 channel features. Then, the two channel features are sent to a shared two-layer neural network composed of a multi-layer perceptron (MLP) and a hidden layer. The number of neurons in the first layer is C / r, where r is the scaling factor, the number of neurons in the second layer is C, and the activation function is ReLU. After performing product addition operations on the elements of the two output features, the required channel attention map M can be obtained through the Sigmoid activation function. c , and finally M c Perform tensor multiplication with the original input feature map F to obtain the final channel attention feature map F', the output of the channel attention module M c (F) is calculated by the following formula:

[0035] Mc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1)

[0036] F′=M c (F)·F (2)

[0037] Where F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, MLP represents multi-layer perceptron, σ represents Sigmoid activation function, · represents element-wise multiplication, and the broadcast mechanism is used in the middle for dimensional transformation and matching;

[0038] Then the spatial attention mechanism uses average pooling and maximum pooling to compress the channels of the input F' in dimension to aggregate the channel information. The two H×W×1 channel features obtained are fused through a 7×7 convolutional layer, and finally activated by the Sigmoid function to obtain the spatial attention map M. s , and then multiply it with the original input feature map F' to get the final feature fusion map F''; the output M of the spatial attention module s (F') is calculated by the following formula:

[0039] M s (F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)])) (3)

[0040] F″=M s (F′)·F′ (4)

[0041] Among them, f 7x7 Represents a 7×7 convolution operation, [AvgPool(F'); MaxPool(F')] means concatenating the average pooling and maximum pooling results along the channel axis.

[0042] Furthermore, the loss function used in the training process of step 6 is as follows:

[0043]

[0044] Where Z CE represents the normalization factor, N s is the value range of the spatial domain, that is, the value range of the pixel position (i, j), S is the two-dimensional vector matrix of the true label, y is the two-dimensional vector matrix of the predicted label, and d represents the possible pixel offset on the image. By taking the minimum value of the cross entropy loss under all offset conditions, the model can better adapt to the annotation error of the real data to improve the accuracy of the conversion.

[0045] Furthermore, the method further includes step 8, wherein the difference between the finally generated pattern color code image and the real pattern color code image is evaluated by using the accuracy rate, and the accuracy rate calculation formula is:

[0046]

[0047] Where: N inst is the normalization constant, y GT Is a true color code label, s k (i,j) is the softmax value of the color code of the kth category at position (i, j), d is the possible translation, and for each pixel position (i, j) in the entire data set, a displacement d is found so that the generated color code s k (i,j)+d With real color code y GT Closest.

[0048] The present invention also provides a knitting pattern color code generation optimization device based on deep learning, comprising:

[0049] one or more processors;

[0050] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the knitting pattern color code generation optimization method based on deep learning as described in the above technical solution.

[0051] The present invention proposes an improved InverseKnit++ algorithm to achieve high-precision conversion from knitted fabric patterns to pattern color codes. Multi-scale feature fusion is optimized from the aspects of feature learning, feature expression ability, sample weight distribution, etc. By replacing the traditional cross-row convolution layer and global pooling layer with the SPD module, the loss of fine-grained information and the problem of insufficient feature learning are effectively avoided. In addition, different attention mechanisms are applied to shallow detail features and high-level semantic features, respectively, to enhance the information acquisition ability of the model in channel and spatial dimensions. The results of comparison experiments and ablation experiments show that the improved model not only improves the low-frequency color code recognition accuracy and overall conversion efficiency in the process of knitted fabric pattern digitization, but also significantly reduces the phenomenon of missed generation and erroneous generation of pattern color codes, meeting the needs of modern textile industry for rapid design and production. However, there is still room for improvement in the real-time performance of the model. Future work will comprehensively consider accuracy, complexity and inference speed to achieve a balance between high accuracy, lightweight and real-time performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the improved InverseKnit++ structure in the embodiment of the present invention.

[0053] Figure 2 This is the optimized RefinerNet network structure in the embodiment of the present invention.

[0054] Figure 3 This is the SPD convolution when the step size is 2 in the embodiment of the present invention.

[0055] Figure 4 1 is a comparison of two different residual structures in an embodiment of the present invention.

[0056] Figure 5 This is a structural diagram of the CBAM attention mechanism module in an embodiment of the present invention.

[0057] Figure 6 The results generated by the Kaspar algorithm in the embodiment of the present invention are compared with those generated by the algorithm of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0059] The InverseKnit++ network proposed in the embodiment of the present invention consists of three parts, namely, a refinement preprocessing network (RefinerNet), a backbone feature extraction network (Img2Prog) and a PANet feature fusion network. Figure 1 As shown. In view of the problem that the existing refinement preprocessing network is prone to information loss when processing knitted image features, resulting in residual noise in the image after refinement, the present invention introduces the CBAM module into the Resblock structure of the original residual network, and uses the method of dynamically adjusting the feature map weights to more accurately capture the pattern structure features in the knitted image, so that the shallow features can be better transmitted to the subsequent deeper network layers. At the same time, in order to avoid the loss of pattern structure information and poor feature learning during the model downsampling process, the SPD convolution module is used instead of the downsampling cross-row convolution layer.

[0060] The knitting pattern color code generation optimization method based on deep learning provided by an embodiment of the present invention comprises the following steps:

[0061] Step 1: Based on the collected knitted fabric image, the knitting plate making software draws the corresponding pattern color code map; the knitted fabric image is used as input and the pattern color code image is used as a label to form a data set, and the data set is divided into a training set and a test set.

[0062] Step 2, randomly scaling, cropping, horizontally flipping and data preprocessing the knitted image data in the data set;

[0063] Step 3, constructing an InverseKnit++ model, wherein the InverseKnit++ model includes a RefinerNet refinement preprocessing network and an Img2Prog pattern color code generation network;

[0064] Step 4, using the RefinerNet refinement preprocessing network to refine the received knitted image, thereby obtaining a grayscale image with obvious loop structure and texture features; the RefinerNet refinement preprocessing network is a RefinerNet refinement preprocessing network that applies multi-scale fusion.

[0065] The specific processing process of RefinerNet refinement preprocessing network is as follows:

[0066] A two-stage image conversion strategy is adopted to reduce the impact of realistic factors such as lighting and tension on the generation of pattern color codes. When the network receives an image input of 160x160 pixels, the RefinerNet network is first used to refine the knitted image. After the refinement process is completed, the image size is restored to 160x160 pixels by upsampling. The refined image has better detail performance in the loop and texture structure. The network architecture of RefinerNet is as follows: Figure 2 shown.

[0067] The network structure of RefinerNet is built based on the Feature Pyramid Network (FPN), and the construction of features includes three processes: bottom-up, top-down, and lateral connection. The bottom-up path is a convolutional network for feature extraction. In the process from bottom to top, the spatial resolution is downsampled, but more semantic feature information can be extracted and compressed. On the other hand, through the top-down path, FPN can reconstruct feature information. The lateral connection between the bottom-up and top-down channels supplements the high-resolution details and helps to locate the pattern structure of knitted fabrics.

[0068] The specific processing process of the bottom-up path is as follows:

[0069] Input a knitted image (160×160), and pass through convolution layer 1, residual block 1, convolution layer 2, residual block 2, convolution layer 3, residual block 3, and convolution layer 4 in sequence; after passing through residual block 1, feature map p1 (80×80) is obtained, after passing through residual block 2, feature map p2 (40×40) is obtained, after passing through residual block 3, feature map p3 (20×20) is obtained, and after passing through convolution layer 4, feature map p4 (10×10) is obtained.

[0070] The convolution layer here is composed of a convolution kernel with a size of 3×3, a stride of 2, and a feature channel of 64, a Relu activation function layer and a batch normalization Batch Norm layer, which is used for downsampling; the residual block is composed of two convolution kernels with a size of 3×3, a stride of 1, and a feature channel of 64, a CBAM attention module, a Relu activation function layer and two batch normalization Batch Norm layers, which are used to enhance the expressiveness of the feature map.

[0071] The specific process of horizontal connection is as follows:

[0072] The four feature maps p1, p2, p3, and p4 of different sizes are respectively passed through a horizontally connected convolutional layer to obtain feature maps c1, c2, c3, and c4.

[0073] The convolution layer here consists of a convolution kernel with a size of 1×1, a stride of 1, and 64 feature channels, a Relu activation function layer, and a batch normalization layer, which are used to transform the feature dimension.

[0074] The specific processing process of the top-down path is as follows:

[0075] Feature map n4, i.e. c4, first passes through upsampling layer 1, then unit-adds with feature map c3 in summing layer 1, and then passes through convolution layer 5 to obtain feature map n3. Feature map n3 first passes through upsampling layer 2, then unit-adds with feature map c2 in summing layer 2, and then passes through convolution layer 6 to obtain feature map N2. Feature map n2 first passes through upsampling layer 3, then unit-adds with feature map c1 in summing layer 3, and then passes through convolution layer 7 to obtain feature map n1.

[0076] The size of feature map n4 is enlarged by 4 times, and the size of feature map n3 is enlarged by 2 times. They are input into the concatenation layer together with feature map n2, and the feature map F2 (40×40) is obtained through convolution layer 8. Feature map F2 passes through upsampling layer 4, and then is added with feature map n1 in summing layer 4, and then passes through convolution layer 9 to obtain feature map F1 (80×80), which contains semantic information at different levels. Feature map F1 passes through upsampling layer 5, and then passes through convolution layer 10 to obtain the final output refined image (160×160). Upsampling layer 5 and convolution layer 10 are used to restore clear images and remove artifacts.

[0077] The convolution layer here is composed of a convolution kernel with a size of 3×3, a stride of 1, and 64 feature channels, a Relu activation function layer, and a batch normalization layer, which are used for smooth feature processing; the upsampling layer uses bilinear interpolation to upsample the feature map. The sum layer is used for summation operations; the connection layer is used to splice the feature map in the channel dimension.

[0078] RefinerNet is built on the Feature Pyramid Network (FPN), balancing speed and accuracy, and achieving multi-scale feature fusion through bottom-up and top-down paths. The network contains a discriminator for distinguishing between real images and refined images and a generator for image restoration. The discriminator uses Patch-GAN to deal with blurring problems, while the generator uses FPN to extract multi-scale feature maps, and achieves feature fusion through lateral links and 1x1 convolution operations. The upsampling layer and convolution layer are used to restore clear images and reduce artifacts, ensuring that the generated refined images can provide more accurate input for the subsequent backbone feature extraction network. This design effectively retains high-frequency details and improves the quality of image refinement.

[0079] Step 5, predicting the pattern color code of the grayscale image through the Img2Prog pattern color code generation network, thereby obtaining a pattern color code image; the Img2Prog pattern color code generation network is an Img2Prog pattern color code generation network that applies an SPD convolution module and a CBAM attention mechanism;

[0080] The specific processing process of the Img2Prog pattern color code generation network is as follows:

[0081] The feature maps extracted by the shallow network have more pixel-level information, but less semantic information and more noise; while the deep network has rich semantic information, but may ignore detailed features. The original Img2Prog network only generates pattern color codes on the final feature map, which will lead to unsatisfactory generation results when processing small-sized pattern tissue samples or downsampled fabric images. Therefore, it is necessary to integrate the information of feature maps at all levels to accurately capture the texture characteristics of knitted fabric images.

[0082] By integrating the PANet network at the end of the Img2Prog network, multi-scale features can be fully utilized. PANet is similar to FPN and adopts a bottom-up path to connect feature maps of different resolutions through upsampling "N1", "N2", "N3" and downsampling "P1", "P2", "P3" to achieve feature fusion. In this process, the SPD convolution module replaces the cross-row convolution to perform feature space conversion, and cascades the results of the fusion of feature maps at different levels. At the same time, a non-cross-row convolution is added after each SPD module to avoid excessive downsampling of the image and retain more fine-grained information. At the end of the network, 3x3 convolution is used to adjust the feature channel, and 1x1 convolution outputs the mapping matrix. This design takes into account the connection between each patch and is better than pixel-by-pixel processing. The internal structure of SPD is as shown above. Figure 3 shown.

[0083] The improved Img2Prog network is mainly composed of the backbone feature extraction network and the PANet feature fusion network. The CBAM module is introduced into the original residual block to form the RBCM residual, and the SPD convolution module is used instead of the cross-row convolution.

[0084] The specific processing process of the backbone feature extraction network is as follows:

[0085] First, the input refined image (160×160) passes through the first convolution layer, the second convolution layer, the third convolution layer, and the residual block RBCM in sequence. After the first convolution layer, the feature map Q1 (80×80) is obtained, after the second convolution layer, the feature map Q2 (40×40) is obtained, after the third convolution layer, the feature map Q3 (20×20) is obtained, and after the residual block RBCM, the feature map Q4 (20×20) is obtained.

[0086] The convolution layer here is composed of a convolution kernel with a size of 3×3, a stride of 2, and a feature channel of 64, a Relu activation function layer and a batch normalization Batch Norm layer, which is used for downsampling; the residual block is an RBCM composed of two convolution kernels with a size of 3×3, a stride of 1, and a feature channel of 64, a CBAM attention module, a Relu activation function layer and two batch normalization Batch Norm layers, which are used to enhance the expressiveness of the feature map.

[0087] Then, feature maps Q1, Q2, and Q3 are respectively passed through a horizontally connected convolutional layer to obtain feature maps k1, k2, and k3.

[0088] The convolution layer here consists of a convolution kernel with a size of 1×1, a stride of 1, and 64 feature channels, a Relu activation function layer, and a batch normalization layer, which are used to transform the feature dimension.

[0089] Finally, feature map P3, i.e. k3, first passes through the first upsampling layer, and then unit-adds it with feature map k2 in the summing layer, and then passes through the fourth convolutional layer to obtain feature map P2. Feature map P2 first passes through the second upsampling layer, and then unit-adds it with feature map k1 in the second summing layer, and then passes through the fifth convolutional layer to obtain feature map P1.

[0090] The convolution layer here is composed of a convolution kernel with a size of 3×3, a stride of 1, and 64 feature channels, a Relu activation function layer, and a batch normalization layer, which are used for smooth feature processing; the upsampling layer uses bilinear interpolation to upsample the feature map. The summation layer is used for summation operations;

[0091] The specific processing process of the PANet feature fusion network is as follows:

[0092] First, feature map N1, that is, P1, passes through the sixth convolution layer, performs unit addition with feature map P2 in the third summing layer, and passes through the seventh convolution layer to obtain feature map N2. Feature map N2 passes through the eighth convolution layer, performs unit addition with feature map P1 in the fourth summing layer, and passes through the ninth convolution layer to obtain feature map N3.

[0093] The sixth and eighth convolutional layers here are composed of a convolution kernel with a size of 3×3, a stride of 2, and a feature channel of 64, a Relu activation function layer, and a batch normalization Batch Norm layer, which are used for downsampling; the seventh and ninth convolutional layers are composed of a convolution kernel with a size of 3×3, a stride of 1, and a feature channel of 64, a Relu activation function layer, and a batch normalization Batch Norm layer, which are used for smooth feature processing.

[0094] Next, the feature maps N1, N2, and N3 are downsampled through the SPD modules with step sizes of 4, 2, and 1, respectively, and then input into the Concatenation connection layer together with the feature map Q4 through the jump connection for feature fusion. The feature map F is obtained through the tenth convolution layer, and then through the eleventh convolution layer to obtain the output pattern color code image (20×20).

[0095] The tenth convolution layer here is composed of a convolution kernel with a size of 3×3, a stride of 1, and 64 feature channels, a Relu activation function layer and a batch normalization BatchNorm layer, which is used for smooth feature processing; the eleventh convolution layer is composed of a convolution kernel with a size of 1×1, a stride of 1, and 17 feature channels, a Relu activation function layer and a batch normalization BatchNorm layer, which is used for classification prediction.

[0096] The SPD module consists of two parts. The first part is the feature conversion operation from space to depth, which can convert the spatial dimension of the feature map into the depth dimension and retain more information by increasing the number of channels. The second part is the non-strided convolution layer, which maintains the spatial dimension and reduces the number of channels. By using the SPD module to replace the traditional downsampling method (cross-row convolution), the loss of feature information is avoided and the performance of the model on complex tasks is improved. Among them, the network architecture of the SPD module is as follows Figure 3 As shown, the specific processing process is as follows (taking the SPD module with a step size of 2 as an example):

[0097] The input feature map F (40×40×64) is divided into 4 feature maps c1, c2, c3, c4 (20×20×64) along the spatial dimension, and then the feature maps c1, c2, c3, c4 are concat-connected along the channel dimension to obtain the feature map P ((20×20×256). The feature map F is transformed from space to depth to obtain the feature map P, whose spatial dimension is reduced by 2 times and the number of channels is increased by 4 times. Then a convolution with a step size of 1 is applied to the feature map P to reduce the channel dimension to 1 / 4, and the processed feature map M (20×20×64) is obtained.

[0098] The attention mechanism is a key technology to improve the performance of deep learning models. It enables the model to focus on the key features in the image and ignore irrelevant information, thereby improving the efficiency and accuracy of feature extraction. However, the original Kaspars algorithm fails to fully learn the relevant information between features due to the use of fixed convolution kernels and number of channels. To overcome this limitation, the present invention introduces the CBAM module in the residual block to more deeply understand the correlation and importance of feature channels and spatial positions, and compares the original structure with the improved residual structure, such as Figure 4 shown.

[0099] Although the SE (Squeeze-and-Excitation) module performs well in enhancing channel feature representation, it mainly focuses on channel information and ignores spatial information, which limits its ability to process complex images. The CA (CoordinateAttention) module enhances feature expression by integrating spatial coordinate information into channel attention. It can capture local information, but is not enough to integrate global information. The CBAM module contains two attention modules, channel and space. It strengthens features by element-by-element multiplication, effectively suppresses noise, and retains key information, more comprehensively capturing the global and local features of the image, such as Figure 5 shown.

[0100] The channel attention mechanism firstly processes the feature map F of size C×H×W through two different pooling methods, namely, global maximum pooling (MaxPool) and global average pooling (AvgPool), to obtain two C×1×1 channel features. Then, the two channel features are sent to a shared two-layer neural network composed of a multi-layer perceptron (MLP) and a hidden layer. The number of neurons in the first layer is C / r, where r is the scaling factor, the number of neurons in the second layer is C, and the activation function is ReLU. After performing product addition operations on the elements of the two output features, the required channel attention map M can be obtained through the Sigmoid activation function. c , and finally M c Perform tensor multiplication with the original input feature map F to obtain the final channel attention feature map F'. The output M of the channel attention module c (F) can be calculated by the following formula:

[0101] Mc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1)

[0102] F′=M c (F)·F (2)

[0103] Where F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, MLP represents multi-layer perceptron, and σ represents Sigmoid activation function. · represents element-wise multiplication, and the broadcast mechanism is used in the middle for dimension transformation and matching.

[0104] Then the spatial attention mechanism uses average pooling and maximum pooling to compress the channels of the input F' in dimension to aggregate the channel information. The two H×W×1 channel features obtained are fused through a 7×7 convolutional layer, and finally activated by the Sigmoid function to obtain the spatial attention map M. s , and then multiply it with the original input feature map F' to get the final feature fusion map F''. The output M of the spatial attention module s (F') can be calculated by the following formula:

[0105] M s (F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)])) (3)

[0106] F″=M s (F′)·F′ (4)

[0107] Among them, f 7x7Represents a 7×7 convolution operation, [AvgPool(F'); MaxPool(F')] means concatenating the average pooling and maximum pooling results along the channel axis.

[0108] Step 6, using the processed training set to train the constructed InverseKnit++ model, and inputting the test set into the trained InverseKnit++ model to generate a pattern color code image;

[0109] Pattern color code generation is a type of image generation task, so the present invention adopts a supervised learning approach and selects CE loss, i.e., cross-entropy loss, as the loss function of the model. In view of the fact that manually labeled images may cause slight spatial misalignment of some color codes, the present invention considers all possible single pixel offsets when calculating the cross-entropy loss to accommodate the potential alignment deviation between the knitted image and the pattern color code. The complete cross-entropy loss formula is:

[0110]

[0111] Where Z CE represents the normalization factor, N s is the value range of the spatial domain, S is the two-dimensional vector matrix of the real label, y is the two-dimensional vector matrix of the predicted label, and d represents the possible pixel offset (up, down, left, right, diagonal) on the image. By taking the minimum value of the cross entropy loss under all offset conditions, the model can better adapt to the annotation error of the real data and improve the accuracy of the conversion.

[0112] Step 7: Adjust the size of the pattern color code image by using the nearest neighbor interpolation algorithm without changing the number of color codes, and combine each color code with the corresponding knitting instruction to obtain the final pattern color code image.

[0113] The technical effect of the present invention is described below by experiments:

[0114] Experimental environment: The present invention uses Intel(R) Xeon(R) Platinum 8255C CPU@2.50GHz, 40G memory, NVIDIA GeForce RTX 2080Ti graphics card, 64-bit Windows 10 operating system. The model is built based on the deep learning framework Tensorflow1.0, the programming language is Python3.7, and the GPU computing platform is CUDA9.0. The number of iterations is 15K, the batch size is 2, the initial learning rate is 0.0005, and the Adam optimizer is used for optimization, and the decay learning rate is 0.3%.

[0115] Dataset: This paper selects the public knitted fabric dataset released by Kasper to conduct a series of comparative experiments. The dataset consists of 12,392 knitted images and pattern color code images. The knitted images are divided into 8 categories according to the texture structure: Cable, Full, Hem, Links, Mesh, Miss, Move, Tuck. The ratio of training set, test set and validation set is 8:0.5:1.5. The pattern color code images are composed of 17 basic color codes selected from Shima Seikis software.

[0116] Training settings and evaluation indicators: For the refinement effect, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to evaluate the image quality. For the instruction mapping results, the full picture accuracy (FULL) and foreground accuracy (FG) are used to evaluate the accuracy of the relevant algorithms and analyze the advancement and effectiveness of the algorithms.

[0117] FULL refers to calculating the average of all instructions, and FG refers to calculating the average of all instructions except the background. Considering that manual labeling may cause slight spatial misalignment between the knitting image and the pattern color code image, this study uses an accuracy measurement method with a definition similar to the loss function. The accuracy calculation formula is:

[0118]

[0119] Where: N inst is the normalization constant, which is consistent with Z in the loss function CE Same. GT Is a true color code label, s k (i,j) is the softmax value of the color code of the kth category at position (i, j), d is the possible translation (within 1 pixel), and for each pixel position (i, j) in the entire dataset, a displacement d is found so that the generated color code s k (i,j)+d With real color code y GT This approach quantifies the accuracy of the model's generation of color-coded images, taking into account possible small alignment differences between the real color-coded images and the generated images.

[0120] Comparison experiment of different attention mechanisms: The essence of the attention mechanism is to calculate the corresponding weight values ​​to allow the convolutional neural network to identify useful feature vectors that need to be focused on and ignore unimportant feature information. This can avoid useless features from interfering with the fitting results while also improving the operation speed to a certain extent. Based on the baseline Kaspar network, four different types of attention mechanisms are introduced to observe significant changes in performance. The comparison results are shown in Table 1.

[0121] Table 1 Performance comparison of different attention mechanisms

[0122]

[0123] The data analysis in Table 1 shows that the introduction of SE attention mechanism leads to a slight decrease in the overall accuracy (FullAccuracy) to 93.69% and the foreground accuracy (FGAccuracy) to 79.61%, although the perceptual indicators (such as SSIM) remain stable, but the PSNR signal-to-noise ratio is improved. The addition of CA increases the FG accuracy to 80.13%, and although the overall accuracy decreases slightly, both SSIM and PSNR are improved. The introduction of the GAM mechanism leads to a decrease in performance on all evaluation indicators. In contrast, CBAM shows the best performance on all evaluation indicators, with the highest overall accuracy of 94.48%, FG accuracy of 82.06%, SSIM and PSNR reaching 0.91 and 24.13dB respectively. These results emphasize the high efficiency of CBAM in enhancing model feature recognition and utilization, and highlight the potential of attention mechanism in improving model accuracy and generation quality, especially when processing complex knitted image data.

[0124] Ablation experiments of different modules: Each improvement strategy is trained and evaluated on the Kasper dataset. The impact of different improvement strategies on the baseline model is shown in Table 2. The experimental results show that after the preprocessing network and the feature extraction network are multi-scale fused, the network feature extraction capability is enhanced and the knitting pattern structure is clearer; after replacing the cross-row convolution structure with the SPD module, the model has improved the accuracy of low-frequency color code generation on low-resolution and fabric images with complex texture structures; adding the spatial channel attention mechanism to the residual structure, the network can better identify and locate the color code position information, and to a certain extent improve the color code loss and misalignment of the generated pattern color code. After combining all the improvement methods, the FG value increased by 3.17%, verifying that all modules have played a certain role.

[0125] Table 2 Ablation experiment results of different modules

[0126]

[0127] Comparative experiments between different models: The improved model of the present invention is compared with classic models such as UNet, S+U, Scene Parsing, and Kaspar on the test set, and the comparison results are shown in Table 3. Compared with other classic generation models, the network model proposed in the present invention achieves a better mapping effect, not only in terms of accuracy, but also in terms of PSNR, which is 3.17% higher than the Kaspar algorithm, and 0.7% higher, so as to better meet the actual scene.

[0128] Table 3 Performance comparison results of different models

[0129]

[0130] Comparative experiment of color codes with different frequencies: In order to compare the accuracy of the algorithm of the present invention and the Kaspar algorithm in generating different color codes, an experimental comparison was conducted on the test set, and the experimental results are shown in Table 4. It can be observed that there are significant differences in the frequency of occurrence of different color codes in the test set. Compared with the original InverseKnit model, the network model proposed in the present invention not only shows higher accuracy in the generation of high-frequency color codes, but also has a significant improvement in the generation accuracy of low-frequency color codes, indicating that the model has better robustness.

[0131] Table 4 Performance comparison results of different frequency color codes

[0132]

[0133] Qualitative analysis: In order to intuitively demonstrate the improvement of network performance, the present invention conducts a visual comparative analysis of 7 groups of pattern color code images and their reconstructed images, such as Figure 6 As shown. By observing the first row of images, it is found that the original network made mistakes when generating the color code of the twisted structure, while the model proposed by the present invention can generate the color code more accurately; the second row of images shows that the original network has the problem of color code loss when processing the flanging structure, while the model of the present invention performs better even when processing color codes with lower frequency; the third and fourth rows respectively show the color codes of the needle turning structure and the mesh structure, and it is obvious that the algorithm of the present invention has more complete and accurate structural information when generating pattern color codes, reducing the color code misalignment; by comparing the fifth to seventh rows, it can be seen that when processing different yarn colors and texture structures, the improved model performs better in the accuracy of pattern color code generation, and the reconstructed knitting image is more complete in the coil structure.

[0134] In order to reduce the cost of knitted fabric pattern design and development, shorten the sample cycle, and meet the needs of current fast fashion. The present invention proposes an improved InverseKnit++ algorithm, which realizes the automatic design of knitted fabric to pattern color code. By multi-level feature fusion of RefinerNet refinement preprocessing network, the recognition ability of complex textures is improved; multi-level information reuse is realized by combining Img2Prog network and PANet (Path Aggregation Network), and SPD (space to depth) convolution is used instead of cross-row convolution; the improvement measures such as attention mechanism are introduced in the residual structure to improve the accuracy of low-frequency color code generation, which effectively improves the problems of color code loss and insufficient accuracy in the original algorithm. The improved algorithm is quantitatively evaluated and qualitatively analyzed on the test set. The experimental results show that compared with the Kaspar algorithm, the improved InverseKnit++ algorithm is improved to 95% in overall accuracy and 83.17% in foreground accuracy, which verifies that the algorithm has a good effect in the generation of pattern color codes of knitted images.

[0135] The above description is only a preferred specific implementation manner of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention should be included in the scope of protection of the invention.

Claims

1. A knitting pattern color code generation optimization method based on deep learning, characterized in that: The steps include: Step 1: Based on the collected knitted fabric image, the knitting plate making software draws the corresponding pattern color code map; the knitted fabric image is used as input, and the pattern color code image is used as a label to form a data set, and the data set is divided into a training set and a test set; Step 2, randomly scaling, cropping, horizontally flipping and data preprocessing the knitted image data in the data set; Step 3, constructing an InverseKnit++ model, wherein the InverseKnit++ model includes a RefinerNet refinement preprocessing network and an Img2Prog pattern color code generation network; Step 4, using a RefinerNet refinement preprocessing network to refine the received knitted image, thereby obtaining a refined image with obvious loop structure and texture features; the RefinerNet refinement preprocessing network is a RefinerNet refinement preprocessing network that applies multi-scale fusion; Step 5, predicting the pattern color code of the refined image through the Img2Prog pattern color code generation network, thereby obtaining a pattern color code image; the Img2Prog pattern color code generation network is an Img2Prog pattern color code generation network using an SPD convolution module and an attention mechanism module CBAM; Step 6, using the preprocessed training set to train the constructed InverseKnit++ model, and inputting the test set into the trained InverseKnit++ model to generate a pattern color code image; Step 7: Adjust the size of the pattern color code image by using the nearest neighbor interpolation algorithm without changing the number of color codes, and combine each color code with the corresponding knitting instruction to generate the final pattern color code image.

2. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 1, characterized in that: The RefinerNet refinement preprocessing network is built based on the feature pyramid network FPN, which includes three processes: bottom-up, top-down, and lateral connection. The bottom-up path is a convolutional network for feature extraction. In the process from bottom to top, the spatial resolution is downsampled to extract and compress more semantic feature information. Through the top-down path, FPN can reconstruct feature information. The lateral connection between the bottom-up and top-down channels is used to supplement high-resolution details and help locate the pattern of knitted fabrics. The specific processing process of the bottom-up path is as follows: The input knitted image passes through three groups of convolutional units consisting of convolutional layers and residual blocks, and one convolutional layer, and obtains feature maps p1-p4 respectively; The specific process of horizontal connection is as follows: The four feature maps p1, p2, p3, and p4 of different sizes are respectively passed through a horizontally connected convolutional layer to obtain feature maps c1, c2, c3, and c4; The specific processing process of the top-down path is as follows: Feature map n4, i.e. c4, first passes through upsampling layer 1, then performs unit addition with feature map c3 in summing layer 1, and then passes through convolution layer 5 to obtain feature map n3; feature map n3 first passes through upsampling layer 2, then performs unit addition with feature map c2 in summing layer 2, and then passes through convolution layer 6 to obtain feature map n2; feature map n2 first passes through upsampling layer 3, then performs unit addition with feature map c1 in summing layer 3, and then passes through convolution layer 7 to obtain feature map n1; The size of feature map n4 is enlarged by 4 times, and the size of feature map n3 is enlarged by 2 times, and they are input into the Concatenation layer together with feature map n2. Feature map F2 is obtained through convolution layer 8. Feature map F2 is passed through upsampling layer 4, and then added with feature map n1 in summing layer 4. Feature map F1 is obtained through convolution layer 9. Feature map F1 is passed through upsampling layer 5, and then through convolution layer 10 to obtain the final output refined image.

3. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 1, characterized in that: The Img2Prog pattern color code generation network is composed of a backbone feature extraction network and a PANet feature fusion network. The attention mechanism module CBAM is introduced into the residual block of the backbone feature extraction network to form the RBCM residual, and the SPD convolution module is used instead of the cross-row convolution. The specific processing process of the backbone feature extraction network is as follows: First, the input refined image passes through the first convolution layer, the second convolution layer, the third convolution layer, and the residual block RBCM in sequence. The feature map Q1 is obtained after the first convolution layer, the feature map Q2 is obtained after the second convolution layer, the feature map Q3 is obtained after the third convolution layer, and the feature map Q4 is obtained after the residual block RBCM. Then, feature maps Q1, Q2, and Q3 are respectively passed through a horizontally connected convolutional layer to obtain feature maps k1, k2, and k3; Finally, feature map P3, i.e. k3, first passes through the first upsampling layer, performs unit addition with feature map k2 in the first summing layer, and passes through the fourth convolutional layer to obtain feature map P2. Feature map P2 first passes through the second upsampling layer, performs unit addition with feature map k1 in the second summing layer, and passes through the fifth convolutional layer to obtain feature map P1. The specific processing process of the PANet feature fusion network is as follows: First, the feature map N1, that is, P1, first passes through the sixth convolution layer, and is unit-added with the feature map P2 in the third addition layer, and passes through the seventh convolution layer to obtain the feature map N2. The feature map N2 first passes through the eighth convolution layer, and is unit-added with the feature map P1 in the fourth addition layer, and passes through the ninth convolution layer to obtain the feature map N3. Next, the feature maps N1, N2, and N3 are downsampled through the SPD modules with step sizes of 4, 2, and 1, respectively, and then input into the Concatenation connection layer together with the feature map Q4 through the jump connection for feature fusion. The feature map F is obtained through the tenth convolution layer, and then the output pattern color code image is obtained through the eleventh convolution layer.

4. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 3, characterized in that: The first convolution layer to the eleventh convolution layer each include a convolution kernel, a Relu activation function layer and a batch normalization BatchNorm layer.

5. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 3, characterized in that: The specific processing process of the SPD module with a step size of 2 is as follows: The input feature map F is divided into 4 feature maps along the spatial dimension, and then the 4 feature maps are concat-connected along the channel dimension to obtain the feature map P. The feature map F is transformed from space to depth to obtain the feature map P, whose spatial dimension is reduced by 2 times and the number of channels is increased by 4 times. Then a convolution with a step size of 1 is applied to the feature map P to reduce the channel dimension to 1 / 4, and the processed feature map M is obtained.

6. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 1, characterized in that: The processing process of the attention mechanism module CBAM is as follows: The CBAM module contains two attention modules, channel and space. The channel attention mechanism firstly processes the feature map F of size C×H×W through two different pooling methods, namely, global maximum pooling MaxPool and global average pooling AvgPool, to obtain two C×1×1 channel features. Then, the two channel features are sent to a shared two-layer neural network composed of a multi-layer perceptron MLP and a hidden layer. The number of neurons in the first layer is C / r, r is the scaling factor, the number of neurons in the second layer is C, and the activation function is ReLU. After performing product addition operation on the elements of the two output features, the required channel attention map M can be obtained through the Sigmoid activation function. c , and finally M c Perform tensor multiplication with the original input feature map F to obtain the final channel attention feature map F ＇ , the output M of the channel attention module c (F) is calculated by the following formula: (1)； F′=M c (F)·F (2); Where F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, MLP represents multi-layer perceptron, σ represents Sigmoid activation function, • represents element-wise multiplication, and the broadcast mechanism is used in the middle for dimension transformation and matching; Then the spatial attention mechanism is applied to the input F ＇ Each channel of uses average pooling and maximum pooling to compress the dimension to aggregate the channel information, and the two H×W×1 channel features are fused through a 7×7 convolution layer, and finally activated by the Sigmoid function to obtain the spatial attention map M. s , and then with the original input feature map F ＇ Multiply to get the final feature fusion map F ＇＇ ; Output M of the spatial attention module s (F ＇ ) is calculated by the following formula: (3)； F″=M s (F′)·F′ (4); Among them, f 7x7 Represents a 7×7 convolution operation, [AvgPool(F ＇ );MaxPool(F ＇ )] means concatenating the average pooling and maximum pooling results along the channel axis.

7. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 1, characterized in that: The loss function used in step 6 training is as follows: (5)； Where Z CE represents the normalization factor, N s is the value range of the spatial domain, that is, the value range of the pixel position (i, j), S is the two-dimensional vector matrix of the true label, y is the two-dimensional vector matrix of the predicted label, and d represents the possible pixel offset on the image. By taking the minimum value of the cross entropy loss under all offset conditions, the model can better adapt to the annotation error of the real data to improve the accuracy of the conversion.

8. The knitting pattern color code generation optimization method based on deep learning as claimed in claim 1, characterized in that: The method further includes step 8, wherein the difference between the finally generated pattern color code image and the real pattern color code image is evaluated by using the accuracy rate, and the accuracy rate calculation formula is: (6)； Where: N inst is the normalization constant, y GT Is a true color code label, s k (i,j) is the softmax value of the color code of the kth category at position (i, j), d is the possible translation, and for each pixel position (i, j) in the entire data set, a displacement d is found so that the generated color code s k (i,j)+d With real color code y GT Closest.

9. A knitting pattern color code generation and optimization device based on deep learning, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the knitting pattern color code generation optimization method based on deep learning as described in any one of claims 1 to 8.

Images