Yarn color prediction method and system based on multi-modal learning

Abstract

The invention provides a yarn color prediction method and system based on multi-modal learning, and belongs to the technical field of textile printing and dyeing, and the method comprises the steps: firstly, comprehensively collecting blended cotton fiber data, process parameters, fiber color data, yarn appearance images and color data in yarn production, and carrying out the alignment, cleaning and normalization preprocessing of the data; then, a color prediction network composed of a feature extraction module, a compensation fusion module and an output module is constructed, after main modal feature extraction and preliminary prediction are completed through 1D-CNN, auxiliary modal features are extracted with the help of VGG16, and correlation is calculated through the compensation fusion module to achieve compensation for preliminary prediction; and finally, selecting a CIEDE2000 chromatic aberration formula as a loss function, and optimizing model parameters in combination with an Adam optimizer. According to the method, by means of fusion of multi-modal data, a unique fusion compensation strategy and a self-adaptive complementary fusion mechanism, the precision, accuracy and stability of yarn color prediction are remarkably improved, and a more reliable technical scheme is provided for color prediction of colored spun yarns in the textile printing and dyeing industry.

Description

Technical Field

[0001] The present invention relates to the technical field of textile printing and dyeing, and particularly to a yarn color prediction method and system based on multimodal learning. Background Art

[0002] Yarns or fabrics formed by mixing two or more different fiber dyed with original solution and then spun and processed are deeply favored by consumers due to their unique appearance characteristics. Their appearance is delicate and gentle, presenting a hazy three-dimensional feeling and unique texture, building a rich material selection foundation for various fashion designs and high-end clothing fields. In particular, the advanced process of "dying first and then spinning" used in color-spun yarns makes the color presentation mode more flexible and diverse on the one hand, meeting diverse design requirements; on the other hand, it shows excellent environmental friendliness advantages, conforming to the current trend of sustainable development.

[0003] However, the traditional color mixing theory is insufficient in analyzing the unique color mixing rules of colored fiber color mixing systems, which leads to a significant reduction in the color matching accuracy of color-spun products in the actual production process, with a deviation from the expected color. At present, in order to achieve the production requirement of color-spun products that are consistent with the color of the sample provided by the customer (or the established target sample), the vast majority of color-spun enterprises still follow the ancient manual color matching routine. However, this traditional process not only takes a long time, greatly reducing production efficiency, but also causes serious waste of materials during the repeated trial-and-error process. Most importantly, the accuracy of the final product in reproducing the target sample is unsatisfactory. Summary of the Invention

[0004] In view of the deficiencies in the applicability and accuracy of existing yarn color prediction methods, such as the traditional theoretical model being restricted by assumption conditions and the neural network method relying heavily on the data sample size, etc., the present invention makes effective improvements. The present invention provides a yarn color prediction method based on multimodal learning, which specifically includes the following steps: Step 1, collect the main modal data, auxiliary modal data, basic color fiber color data, and yarn color data during the yarn production process, where the main modal data includes mixed cotton fiber data and production process parameters, and the auxiliary modal data is the yarn appearance image, and preprocess the collected data; Step 2, construct a color prediction network, including a feature extraction module and a compensation fusion module; Construct different feature extraction modules for different input data, extract features from the main modal data to obtain the main modal features and the color prediction indicators in the first stage, and further transform and process to obtain the preliminary color prediction value, extract features from the auxiliary modal data to obtain the image color features, that is, the auxiliary modal features; calculate the correlation between the auxiliary modal features and the preliminary color prediction value in the compensation fusion module, and compensate the preliminary color prediction value to obtain the final prediction output; Step 3: Calculate the loss between the final predicted output and the true value of the yarn color using the loss function, optimize the network parameters, and use the optimized network to achieve yarn color prediction.

[0005] Furthermore, the blended cotton fiber data includes blended base color fiber attribute data and the cotton usage ratio during the blending process; the base color fiber attribute data includes length index, strength index, micronaire value, maturity coefficient index, short fiber content index, and base color fiber color index; the production process parameters include spindle speed, draft ratio, and twist coefficient; the base color fiber color data is spectral reflectance, and the yarn color data is in LAB value; for the yarn appearance image, the yarn appearance image during the ring spinning process is collected by an industrial camera at a speed of f frames per second. The yarn appearance image is a color image, and the sliding window method is used to sample the yarn appearance image to keep the sampling frequency consistent with the main modal data.

[0006] Furthermore, the data preprocessing process includes data alignment, data cleaning, and normalization; specifically, the sliding window method is used to process the blended cotton fiber data and the yarn appearance data to ensure that the data of the main modal and the auxiliary modal are aligned on the same time series segment; and the yarn appearance image is adjusted to a certain size. Isolation Forest is used to clean the collected data, and min-max normalization is used to process the data. The specific calculation formula is as follows:

[0007] In the above formula, represents the i-th data of the j-th feature; and represent the maximum and minimum values of the j-th feature respectively; represents the data after normalization processing.

[0008] Furthermore, 1D-CNN is used to extract features from the main modal data and make a preliminary prediction of the yarn color. The main modal data is defined as , to respectively refer to the length index, strength index, micronaire value, maturity coefficient index, short fiber content index, base color fiber color index, fiber usage ratio, spindle speed, draft ratio, and twist coefficient; the feature extraction module corresponding to the main modal data consists of three convolutional and pooling layers and two fully connected layers. The transfer operations are alternately performed through convolution, batch normalization, and non-linear activation functions. Each convolutional kernel moves along the time axis within the sliding window. The features extracted after passing through the three convolutional and pooling layers are the main modal features , and the convolution calculation formula is as follows:

[0009]

[0010]

[0011] In the above formula, represents the feature of the layer extracted from the main modal data, represents the convolution operation, and are the weight and bias of the th convolution layer respectively;

[0012] x is the input of ReLU; the main modal feature obtained after the last convolution and pooling layers is , and the color prediction index of the first stage is output through the connection of two fully connected layers , where the calculation formula of the fully connected layer is as follows:

[0013]

[0014] In the above formula, and are the weight and bias of the th layer respectively. The fourth and fifth layers are fully connected layers, represents the activation function.

[0015] Furthermore, the color prediction index of the first stage is converted into LAB values to obtain the preliminary color prediction value , which is used for the subsequent compensation fusion stage. The specific calculation formula is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] For the intermediate variable The calculation method is as follows: When is the case, ; When is the case, ; In the above formula, the CIE standard illuminant D65 is adopted, , k is a normalization constant, are respectively the spectral power distribution of the illumination light source and the standard observer color matching function, which can be obtained from CIE standard documents and relevant databases; is the wavelength interval; X, Y, and Z are the tristimulus values corresponding to the color prediction index in the first stage.

[0023] Furthermore, VGG16 is used to extract features from the auxiliary modal data to obtain auxiliary modal features, denoted as , which is specifically composed of three alternating convolutional layers and pooling layers. Among them, the convolutional kernel size of the convolutional layer is 3×3, the stride is 1, the pooling kernel size of the first two pooling layers is 2×2, the stride is 2, and the last pooling layer is a global maximum pooling layer. The finally extracted feature map is stored in the channel corresponding to the preliminary color prediction index.

[0024] Furthermore, the compensation fusion module is used to fuse the main modal features extracted from the main modal and the auxiliary modal to compensate the preliminary color prediction value. By calculating the correlation relationship between the two, an association evaluation matrix is formed, where m is the number of channels participating in the compensation fusion. The specific calculation formula is as follows:

[0025] In the above formula, and respectively represent the i-th element in the extracted main modal features and the j-th element in the auxiliary modal features, represents 's regularization, projecting the feature representation into a multi-dimensional space. In the multi-dimensional space, the closer its value is to 1, the higher the alignment degree; Multiply the extracted features by the association evaluation matrix to obtain the compensation amount. The specific calculation formula is as follows:

[0026] In the above formula, represents the transpose of , represents the modulus of the matrix, is the weight coefficient; Use the compensation amount to correct the preliminary prediction value to obtain the final prediction index , the specific calculation is as follows:

[0027] in, is the preliminary color prediction value.

[0028] Furthermore, the loss function adopts the CIE DE2000 color difference formula, and the Adam optimizer is used to optimize the network parameters. The specific calculation formula of the loss function is as follows:

[0029] In the above formula, For brightness difference, For color difference, is the hue difference, which is obtained by subtracting the predicted value of yarn color from the actual value of yarn color; They correspond to the weight factors of brightness difference, chroma difference and hue difference respectively; is the rotation factor; is the correction factor.

[0030] Furthermore, it also includes the use of the coefficient of determination , mean square error MSE and mean absolute error MAE are used as evaluation indicators to evaluate the effect of color prediction.

[0031] The present invention also provides a yarn color prediction system based on multimodal learning, comprising: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a yarn color prediction method based on multimodal learning as described in the above scheme.

[0032] Compared with the prior art, the present invention is beneficial in that: (1) Introducing complementary knowledge to enhance multimodal learning methods, breaking through the limitation of using only single information for yarn color prediction, integrating multi-source information such as fiber, process, and appearance images, and integrating multimodal data to improve prediction accuracy and fill in the gaps in the past.

[0033] (2) A unique compensation fusion strategy is adopted. The evaluation matrix is ​​first calculated, and the cosine similarity between the auxiliary feature map and the main modal prediction value is defined to distinguish the strength of the correlation between the main and auxiliary modalities. Then, based on the matrix, the fusion gate is used to assign weights according to the strength of the correlation, calculate the compensation amount, calibrate the initial prediction value, and enhance the accuracy and stability of the results.

[0034] (3) Adaptive complementary fusion mechanism: In the past, deep multimodal fusion was difficult to tap into the potential of information complementarity. The present invention achieves adaptive complementary fusion by means of a compensatory fusion mechanism. When fusion of features is performed, weights are dynamically adjusted according to correlation, the fusion process is automatically optimized, and the efficiency of fusion and complementary utilization is improved. Description of the Drawings

[0035] Figure 1 is the flowchart of the method provided by the present invention; Figure 2 is the flowchart of the main modal feature extraction provided by the present invention; Figure 3 is the flowchart of the auxiliary modal feature extraction provided by the present invention. Specific Implementation Method

[0036] Next, the technical solution of the present invention will be completely described in conjunction with the drawings in the present invention.

[0037] As Figure 1 shown, a yarn color prediction method based on multi-modal learning provided by an embodiment of the present invention includes the following steps: Step 1, collect the mixed cotton fiber data, production process parameters, fiber color data, yarn appearance images, and yarn color data during the yarn production process, and preprocess the collected data; The mixed cotton fiber data includes mixed base color fiber attribute data and the cotton usage ratio during the cotton mixing process; among them, the base color fiber attribute data includes length index, strength index, micronaire value, maturity coefficient index, and short fiber content index; the production process parameters include spindle speed, draft ratio, and twist coefficient; the base color fiber color data is spectral reflectance, and the yarn color value is the LAB value (as the true value); for the yarn appearance image, a CMOS industrial camera is used to collect the yarn appearance image during the ring spinning process at a speed of 23.5 frames per second. The image is a color image. To keep the sampling frequency consistent with the main mode (fiber data, process parameters), the sliding window method is used to process the mixed cotton fiber data and the yarn appearance data to ensure that the data of the main mode and the auxiliary mode (yarn appearance image) are aligned on the same time series segment; at the same time, the yarn appearance image is adjusted to 224×224 pixels; The data preprocessing process includes data alignment, data cleaning, and normalization; specifically, the sliding window method is used to process the mixed cotton fiber data and the yarn appearance data to ensure that the data of the main mode and the auxiliary mode are aligned on the same time series segment; the yarn appearance image is adjusted to 224×224 pixels; Isolation forest is used to clean the collected data, and the maximum-minimum normalization is used to process the model input data. The specific calculation formula is as follows:

[0038] In the above formula, represents the i-th data of the j-th feature; and represent the maximum and minimum values of the j-th feature respectively; Represents the data after normalization.

[0039] Step 2: Construct a color prediction network, including a feature extraction module and a compensation fusion module; For different input data, different feature extraction modules are constructed to respectively extract features from the main modalities (mixed cotton fiber data, production process parameters) and the auxiliary modality (yarn appearance image), complete the preliminary prediction of yarn color and the extraction of image color features; in the compensation fusion module, the correlation between the color features of the auxiliary modality and the preliminarily predicted yarn color is calculated, and the preliminarily predicted yarn color is compensated to obtain the final prediction output; In Step 2, the main modality extraction module uses 1D-CNN to extract features from the main modality and make a preliminary prediction. The input of the main modality is defined as , to respectively refer to the length index, strength index, micronaire value, maturity coefficient index, short fiber content index, base color fiber color index, fiber usage ratio, spindle speed, draft ratio, twist coefficient; the main modality extraction module consists of three convolutional and pooling layers and two fully connected layers, and the transfer operation is alternately executed through convolution, batch normalization, and non-linear activation functions. Among them, each convolutional kernel moves along the time axis within the sliding window, and the size of the convolutional kernel of each convolutional layer is 1×3 and the stride is 1. The features extracted through the three convolutional and pooling layers are the main modality features , and the convolution calculation formula is as follows:

[0040]

[0041]

[0042] In the above formula, represents the feature of the layer extracted from the main modality data, represents the convolution operation, and are the weights and biases of the th convolutional layer respectively, The calculation formula of the ReLU activation function is as follows:

[0043] x is the input of ReLU; the main modality feature obtained after the last convolutional and pooling layer is , and the first-stage color prediction index is output through the connection of two fully connected layers, where the calculation formula of the fully connected layer is as follows:

[0044]

[0045] In the above formula, and are the weight and bias of the th layer respectively. The fourth and fifth layers are fully connected layers, represents the activation function; Specifically, the color prediction index in the first stage is converted into LAB values to obtain the preliminary color prediction value , which is used for compensating the fusion stage later. The specific calculation formula is as follows:

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] For the intermediate variable , the calculation method is as follows: When , ; When , ; In the above formula, the CIE standard illuminant D65 is adopted, , k is the normalization constant, are the spectral power distribution of the illumination source and the standard observer color matching function respectively, which can be obtained from the CIE standard documents and relevant databases; is the wavelength interval; X, Y, and Z are the tristimulus values corresponding to the color prediction index in the first stage; Furthermore, VGG16 is used to extract features from the auxiliary modality to obtain the auxiliary modality features, denoted as , specifically composed of three alternating convolutional layers and pooling layers, where the convolutional kernel size of the convolutional layer is 3×3 and the stride is 1. The pooling kernel sizes of the first two pooling layers are 2×2 and the stride is 2. The last pooling layer is a global maximum pooling layer, and the finally extracted feature map is stored in the channel corresponding to the preliminary color prediction index, where the initial parameters of the feature extractor come from VGG16 trained on ImageNet; Furthermore, the compensation fusion module is used to fuse the complementary features extracted from the main modality and the auxiliary modality and to compensate for the preliminary predicted yarn color index. By calculating the correlation relationship between the two, promoting strong correlation fusion, eliminating weak correlation interference, calculating the correlation between the auxiliary feature map and the preliminary color prediction value, and forming an association evaluation matrix , where m is the number of channels participating in the compensation fusion, and the specific calculation formula is as follows:

[0053] In the above formula, and respectively represent the i-th element in the extracted main modality feature and the j-th element in the auxiliary modality feature, represents of regularization, projecting the feature representation into a multi-dimensional space. In the multi-dimensional space, the closer its value is to 1, the higher the alignment degree; To promote the positive interaction between complementary features and suppress the interference of irrelevant features between the main modality and the auxiliary modality, multiply the extracted features by the correlation evaluation matrix to obtain the compensation amount , and the specific calculation formula is as follows:

[0054] In the above formula, represents transpose of, represents the modulus of the matrix, is the weight coefficient; use the compensation amount to correct the preliminary prediction value to obtain the final prediction index , and the specific calculation is as follows:

[0055] Step 3, the loss function adopts the CIEDE2000 color difference formula, and selects the Adam optimizer to optimize the model parameters. The specific calculation formula of the loss function is as follows:

[0056] In the above formula, is the lightness difference, is the chroma difference, is the color difference, obtained by subtracting the predicted value of the yarn color from the true value of the yarn color; are the weight factors corresponding to the lightness difference, chroma difference, and hue difference respectively; is the rotation factor; all are known quantities. During the optimization process, the Adam optimizer is used to optimize the model parameters. is the correction coefficient, which is taken as in this embodiment.

[0057] Input the main mode and auxiliary mode parameters of the yarn color to be predicted into the trained model to obtain the corresponding predicted result of the yarn color.

[0058] To quantitatively evaluate the color prediction effect and model performance, the coefficient of determination , mean squared error MSE, and mean absolute error MAE are used as evaluation indicators, and the formulas are as follows:

[0059]

[0060]

[0061]

[0062] In the above three formulas, n is the number of test samples, is the true value of the color-spun yarn color, is the predicted value of the color-spun yarn color, is the average value of the true values of the color-spun yarn color data in the test set.

[0063] Furthermore, to verify the effectiveness of the model, this example selects the production data of a cotton textile enterprise from April to June 2024 for experiments. The prediction model is built using PyTorch in Python, and the Adam optimizer is used to optimize the network structure parameters. The learning rate in the pre-training stage of the model parameters is set to 0.01, the learning rate in the fusion training stage is 0.001, the number of fusion channels is 3, the batch size is 32, and the number of training rounds is 500; the model provided by this application can reach , MSE = 0.0022, MAE = 0.0216.

[0064] On the other hand, the embodiment of the present invention also provides a yarn color prediction system based on multi-modal learning, including: A processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a yarn color prediction method based on multi-modal learning as described in the above solution.

[0065] It should be understood that the above description of the preferred embodiments is relatively detailed, and thus it should not be considered as a limitation on the protection scope of the present invention. Under the inspiration of the present invention, those of ordinary skill in the art can also make substitutions and modifications without departing from the scope protected by the claims of the present invention, and all of them fall within the protection scope of the present invention. The scope of protection claimed by the present invention shall be subject to the appended claims.

Claims

1. A yarn color prediction method based on multimodal learning, characterized in that, It includes the following steps: Step 1: Collect the main modal data, auxiliary modal data, base color fiber color data, and yarn color data in the yarn production process. The main modal data includes mixed cotton fiber data and production process parameters, and the auxiliary modal data is the yarn appearance image. Preprocess the collected data. Step 2: Construct a color prediction network, including a feature extraction module and a compensation fusion module. Construct different feature extraction modules for different input data. Extract features from the main modal data to obtain the main modal features and the color prediction index in the first stage, and further transform and process to obtain the preliminary color prediction value. Extract features from the auxiliary modal data to obtain the image color features, i.e., the auxiliary modal features. Calculate the correlation between the auxiliary modal features and the preliminary color prediction value in the compensation fusion module, and compensate the preliminary color prediction value to obtain the final prediction output. Step 3: Use the loss function to calculate the loss between the final prediction output and the true value of the yarn color, optimize the network parameters, and use the optimized network to realize yarn color prediction.

2. The method for predicting yarn color based on multimodal learning according to claim 1, characterized in that: The mixed cotton fiber data includes mixed base color fiber attribute data and the cotton usage ratio in the cotton mixing process. The base color fiber attribute data includes length index, strength index, micronaire value, maturity coefficient index, short fiber content index, and base color fiber color index. The production process parameters include spindle speed, draft ratio, and twist coefficient. The base color fiber color data is the spectral reflectance, and the yarn color data is the LAB value. For the yarn appearance image, use an industrial camera to collect the yarn appearance image in the ring spinning process at a speed of f frames per second. The yarn appearance image is a color image, and the sliding window method is used to sample the yarn appearance image to keep the sampling frequency consistent with the main modal data.

3. The method for predicting yarn color based on multimodal learning according to claim 1, wherein: The data preprocessing process includes data alignment, data cleaning, and normalization. Specifically, use the sliding window method to process the mixed cotton fiber data and the yarn appearance data to ensure that the data of the main modal and the auxiliary modal are aligned on the same time series segment. And adjust the yarn appearance image to a certain size. Use the isolation forest to clean the collected data, and use the maximum-minimum normalization to process the data. The specific calculation formula is as follows: ； In the above formula, represents the i-th data of the j-th feature; and represent the maximum and minimum values of the j-th feature respectively; represents the data after normalization.

4. The method for predicting yarn color based on multimodal learning according to claim 1, wherein: Use 1D-CNN to extract features from the main modal data and make a preliminary prediction of the yarn color. The main modal data is defined as , to respectively refer to the length index, strength index, micronaire value, maturity coefficient index, short fiber content index, base color fiber color index, fiber usage ratio, spindle speed, draft ratio, and twist coefficient. The feature extraction module corresponding to the main modal data consists of three convolutional and pooling layers and two fully connected layers. Transmission operations are alternately performed through convolution, batch normalization, and non-linear activation functions. Each convolutional kernel moves along the time axis within the sliding window. The features extracted after passing through the three convolutional and pooling layers are the main modal features . The convolution calculation formula is as follows: ； ； ； x is the input of ReLU; the main modal feature obtained after the last convolutional and pooling layers is ; Output the color prediction index of the first stage through the connection of two fully connected layers , where the calculation formula of the fully connected layer is as follows: ； ； In the above formula, and are the weights and biases of the th layer respectively. The fourth and fifth layers are fully connected layers, represents the activation function.

5. The method for predicting yarn color based on multi-modal learning according to claim 4, wherein: Convert the color prediction index in the first stage into LAB values to obtain the preliminary color prediction value , which is used to compensate the fusion stage later. The specific calculation formula is as follows: ; ; ； ； ； ； ； For intermediate variables The calculation method is as follows: When then ; When then ; In the above formula, the CIE standard illuminant D65 is adopted, , where k is a normalization constant, are the spectral power distribution of the illumination light source and the color matching function of the standard observer, respectively, which can be obtained from CIE standard documents and relevant databases; is the wavelength interval; X, Y, and Z are the tristimulus values corresponding to the color prediction index in the first stage.

6. The method for predicting yarn color based on multimodal learning according to claim 1, wherein: The VGG16 is used to extract features from the auxiliary modality data to obtain auxiliary modality features, denoted as , which is specifically composed of three alternating convolutional layers and pooling layers. The convolutional kernel size of the convolutional layer is 3×3, the stride is 1, the pooling kernel size of the first two pooling layers is 2×2, the stride is 2, and the last pooling layer is a global max pooling layer. The finally extracted feature map is stored in the channel corresponding to the preliminary color prediction index.

7. The method for predicting yarn color based on multimodal learning according to claim 1, wherein: The compensation fusion module is used to fuse the main modal features extracted from the main modality and the auxiliary modality and the auxiliary modal features to compensate the preliminary color prediction value. By calculating the correlation between the two, an association evaluation matrix is formed , where m is the number of channels participating in the compensation fusion, and the specific calculation formula is as follows: ； In the above formula, and respectively represent the i-th element in the extracted main modal features and the j-th element in the auxiliary modal features. represents of regularization, which projects the feature representation into a multi-dimensional space. In the multi-dimensional space, the closer its value is to 1, the higher the alignment degree. Multiply the extracted features by the relevant evaluation matrix to obtain the compensation amount. The specific calculation formula is as follows: ； In the above formula, represents transpose of, represents the modulus of the matrix, is the weight coefficient; the preliminary predicted value is corrected using the compensation amount to obtain the final prediction index , and the specific calculation is as follows: ； Among them, is the preliminary color prediction value.

8. The method for predicting yarn color based on multimodal learning according to claim 1, wherein: The loss function uses the CIE DE2000 color difference formula, and the Adam optimizer is selected to optimize the network parameters. The specific calculation formula of the loss function is as follows: ； In the above formula, is the lightness difference, is the chroma difference, is the hue difference, which is obtained by subtracting the predicted value of the yarn color from the true value of the yarn color; correspond to the weight factors of the lightness difference, chroma difference and hue difference respectively; is the rotation factor; is the correction coefficient.

9. The method for predicting yarn color based on multimodal learning according to claim 1, wherein: It also includes using the coefficient of determination , the mean squared error MSE, and the mean absolute error MAE as evaluation indicators to evaluate the effect of color prediction.

10. A yarn color prediction system based on multimodal learning, characterized in that, It includes: A processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a yarn color prediction method based on multi-modal learning as described in any one of claims 1-9.

Images