Textile fabric recognition method and device based on multi-modal model, equipment and storage medium

Abstract

The application discloses a textile fabric recognition method and device based on a multi-modal model, equipment and a storage medium, relates to the field of textile industry and computer vision technology, and comprises the following steps: acquiring a multi-view fabric image, a fabric language text corresponding to the multi-view fabric image, a fabric color and a fabric database; inputting the multi-view fabric image, the fabric language text and the fabric color into a target multi-modal model to obtain fusion features; calculating the similarity between the fusion features and the fusion features in the fabric database; reordering the fusion features in the fabric database according to the similarity to obtain a reordering result; obtaining a fabric category according to the reordering result to complete fabric recognition. The application can quickly and accurately identify the specific category of the input fabric from the fabric database, and improves the accuracy and efficiency of textile fabric recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of textile industry and computer vision technology, and particularly relates to a textile fabric recognition method and device based on a multi-modal model, equipment and a storage medium. BACKGROUND

[0002] With the development of automation and intelligence, textile fabric recognition technology has become increasingly important in the textile industry. This technology not only relates to production efficiency, but also directly affects product quality and market competitiveness. Traditional fabric recognition methods mainly rely on manual visual inspection, which is inefficient and easily affected by subjective factors, resulting in inconsistency of recognition results. Therefore, developing efficient and accurate fabric recognition methods has become an urgent need in the industry.

[0003] In recent years, fabric recognition methods based on computer vision and image processing technology have gradually emerged. These methods collect images of fabrics and use image processing algorithms to extract feature information, achieving automatic classification and recognition. For example, certain deep learning networks are used to extract fabric recognition features and perform retrieval. However, these existing methods often rely on single modal information, usually using only single-view images for recognition.

[0004] Existing single modal recognition methods perform poorly in complex environments, especially when fabric material similarity is high or detailed features are difficult to capture, recognition accuracy is limited. In addition, existing approaches lack comprehensive consideration of fabric front and back features, and cannot fully utilize multi-view image information, thereby reducing recognition accuracy and robustness. Therefore, how to improve the accuracy and efficiency of textile fabric recognition has become a problem to be solved.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] The present application aims to provide a textile fabric recognition method, device, equipment and storage medium based on a multi-modal model, aiming to solve the technical problem of how to improve the accuracy and efficiency of textile fabric recognition.

[0007] To achieve the above-mentioned purpose, the present application provides a textile fabric recognition method based on a multi-modal model, which comprises:

[0008] Obtaining multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors, and a fabric database;

[0009] Inputting the multi-view fabric images, the fabric language texts and the fabric colors into a target multi-modal model to obtain fusion features;

[0010] calculating a similarity between the fusion feature and fusion features in the fabric database;

[0011] reordering the fusion features in the fabric database according to the similarity, to obtain a reordering result;

[0012] obtaining a fabric category according to the reordering result, to complete fabric recognition.

[0013] In an embodiment, the textile fabric recognition method based on the multi-modal model is applied to a fabric recognition device, and the fabric recognition device comprises at least two detection cameras, and the focal lengths of the detection cameras are different.

[0014] The step of obtaining the multi-view fabric image comprises:

[0015] A plurality of groups of fabric images are obtained, and each fabric image is collected by a detection camera, and each group of fabric images corresponds to each detection camera.

[0016] A multi-view fabric image is determined according to each fabric image.

[0017] In an embodiment, the textile fabric recognition method based on the multi-modal model is applied to a fabric recognition device, and the fabric recognition device comprises a first camera, a second camera, a third camera, and a fourth camera, the focal length of the first camera is greater than the focal length of the second camera, the focal length of the second camera is greater than the focal length of the third camera, and the focal length of the third camera is greater than the focal length of the fourth camera.

[0018] The step of obtaining the multi-view fabric image comprises:

[0019] A multi-view fabric image is obtained according to a first-view fabric image, a second-view fabric image, a third-view fabric image, and a fourth-view fabric image collected by the first camera, the second camera, the third camera, and the fourth camera, the first view is smaller than the second view, the second view is smaller than the third view, and the third view is smaller than the fourth view.

[0020] In an embodiment, the multi-view fabric image comprises a front multi-view fabric image and a back multi-view fabric image, the target multi-modal model comprises a multi-modal feature encoder and a multi-modal feature fusioner, the multi-modal feature encoder comprises a feature embedding layer, an attention layer, and a full connection layer, and the step of inputting the multi-view fabric image, the fabric language text, and the fabric color into the target multi-modal model to obtain a fusion feature comprises:

[0021] input the front multi-view fabric image, the back multi-view fabric image, the fabric language text and the fabric color into the multi-modal feature encoder in sequence, and extract the bottom layer features of the multi-view fabric image and the fabric language text through the feature embedding layer;

[0022] down-sample the bottom layer features through the attention layer, and reduce the feature dimension to obtain down-sampled features;

[0023] extract high-level semantic features in the down-sampled features through the fully connected layer to obtain deep features;

[0024] fuse the deep features through the multi-modal feature fusioner to obtain fused features.

[0025] In an embodiment, before the step of inputting the multi-view fabric image, the fabric language text and the fabric color into the target multi-modal model to obtain fused features, the method further comprises:

[0026] obtaining a target fabric dataset, and dividing the target fabric dataset into a training set and a test set;

[0027] training and adjusting an initial multi-modal model using the training set to obtain a reference multi-modal model;

[0028] testing and verifying the target detection model through the test set to obtain a test result;

[0029] when the test result does not meet the preset requirement, returning to the step of training and adjusting the initial multi-modal model using the training set;

[0030] when the test result meets the preset requirement, taking the reference multi-modal model as the target multi-modal model.

[0031] In an embodiment, the step of obtaining a target fabric dataset comprises:

[0032] obtaining an initial fabric dataset and historical fabric data;

[0033] evaluating and screening the initial fabric dataset to obtain to-be-labeled data;

[0034] comparing the to-be-labeled data with the historical fabric data to obtain a similarity;

[0035] sorting the to-be-labeled data according to the similarity, and filtering the to-be-labeled data with a similarity lower than a preset similarity threshold to obtain a target fabric dataset.

[0036] In an embodiment, the training set includes image data, category data, fabric description language text data, and fabric color data, the image data includes front-and-back multi-view image data, front multi-view image data, back multi-view image data, multi-view image data missing second and third views, and first-view front image data, the initial multi-modal model includes a multi-modal feature encoder, a multi-modal feature fusioner, and a multi-modal feature classifier, the step of training and adjusting the initial multi-modal model using the training set to obtain a reference multi-modal model includes:

[0037] inputting the fabric description language text data, the fabric color data, the front-and-back multi-view image data, the front multi-view image data, the back multi-view image data, the multi-view image data missing second and third views, or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features;

[0038] fusing the fabric multi-modal deep features by the multi-modal feature fusioner to obtain fabric multi-modal fusion features;

[0039] learning and distinguishing the fabric multi-modal fusion features by the multi-modal feature classifier to obtain a fabric category;

[0040] when the fabric category does not match the category data, adjusting the multi-modal feature encoder, the multi-modal feature fusioner, and the multi-modal feature classifier, and returning to the step of inputting the fabric description language text data, the fabric color data, the front-and-back multi-view image data, the front multi-view image data, the back multi-view image data, the multi-view image data missing second and third views, or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features;

[0041] when the fabric category matches the category data, obtaining a reference multi-modal model according to the multi-modal feature encoder, the multi-modal feature fusioner, and the multi-modal feature classifier.

[0042] In addition, to achieve the above object, the application further provides a textile fabric recognition device based on a multi-modal model, which comprises:

[0043] a data acquisition module configured to acquire multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors, and a fabric database;

[0044] a model processing module configured to input the multi-view fabric images, the fabric language texts, and the fabric colors into a target multi-modal model to obtain fusion features;

[0045] a similarity calculation module configured to calculate a similarity between the fusion feature and fusion features in the fabric database;

[0046] a reordering module configured to reorder the fusion features in the fabric database according to the similarity, to obtain a reordering result;

[0047] a fabric recognition module configured to obtain a fabric category according to the reordering result, to complete fabric recognition.

[0048] In addition, to achieve the above object, the present application further provides a textile fabric recognition device based on a multi-modal model, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the textile fabric recognition method based on the multi-modal model as described above.

[0049] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the textile fabric recognition method based on the multi-modal model as described above.

[0050] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the textile fabric recognition method based on the multi-modal model as described above.

[0051] The one or more technical solutions provided by the present application have at least the following technical effects:

[0052] The multi-view fabric image, the fabric language text corresponding to the multi-view fabric image, the fabric color, and the fabric database are obtained; the multi-view fabric image, the fabric language text, and the fabric color are input into a target multi-modal model to obtain fusion features; the similarity of the fusion features and the fusion features in the fabric database is calculated; the fusion features in the fabric database are reordered according to the similarity to obtain a reordered result; and a fabric category is obtained according to the reordered result to complete fabric recognition. The multi-view images of the front and back of the fabric are obtained by multiple cameras to comprehensively capture the features, which are input into a multi-modal model, processed by convolution, pooling, and fully connected layers to generate a fusion feature vector, and converted into deep features to improve the recognition accuracy and efficiency. The system then calculates the similarity of the input fabric features and the database sample features, reorders the samples accordingly, places the most matching fabric in the front row, and quickly determines the category of the input fabric to complete the recognition process. This process can quickly and accurately identify the specific category of the input fabric from the fabric database, even in complex environments or when facing similar fabrics, while maintaining high accuracy. The entire fabric recognition process not only improves the accuracy and efficiency of textile fabric recognition, but also reduces errors caused by human factors, providing technical support for the automated production process of the textile industry. In addition, it also has good flexibility and generalization ability, providing strong technical support for fabric quality control, inventory management, and personalized customization. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0054] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0055] Figure 1 A flowchart is provided for the textile fabric recognition method based on the multi-modal model of the present application embodiment one;

[0056] Figure 2 A front and back textile fabric recognition device schematic diagram is provided for the textile fabric recognition method based on the multi-modal model of the present application embodiment one;

[0057] Figure 3 A multi-view fabric image schematic diagram is provided for the textile fabric recognition method based on the multi-modal model of the present application embodiment one;

[0058] Figure 4The front and back fabric image schematic diagram provided by the textile fabric recognition method based on a multi-modal model of the present application embodiment one;

[0059] Figure 5 The flowchart provided by the textile fabric recognition method based on a multi-modal model of the present application embodiment two;

[0060] Figure 6 The fabric color recognition schematic diagram provided by the textile fabric recognition method based on a multi-modal model of the present application embodiment two;

[0061] Figure 7 The initial multi-modal model structure schematic diagram provided by the textile fabric recognition method based on a multi-modal model of the present application embodiment two;

[0062] Figure 8 The initial multi-modal model training schematic diagram provided by the textile fabric recognition method based on a multi-modal model of the present application embodiment two;

[0063] Figure 9 The module structure schematic diagram of the textile fabric recognition device based on a multi-modal model of the present application embodiment;

[0064] Figure 10 The device structure schematic diagram of the hardware running environment involved in the textile fabric recognition method based on a multi-modal model of the present application embodiment.

[0065] The purpose of the present application, the function characteristics and the advantages will be further explained in combination with the embodiments, with reference to the drawings. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0067] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail in combination with the drawings and the specific embodiments of the present application.

[0068] With the progress of automation and intelligence, textile fabric recognition technology is crucial to improve production efficiency, product quality and market competitiveness. Traditional manual inspection method is low in efficiency and easy to be affected by subjective factors, so the industry urgently needs efficient and accurate recognition method. At present, the fabric recognition technology based on computer vision is developing, which can automatically classify fabrics through image processing, but mostly relies on single view image, and the accuracy is limited in complex situations. In addition, these technologies do not fully consider the front and back features and multi-view information of the fabric, resulting in unsatisfactory recognition effect.

[0069] The main solution of the embodiment of the present application is: through multiple cameras to obtain multi-view images of the front and back of the fabric, to comprehensively capture its features, and input them into a multi-modal model, and through convolution, pooling and fully connected layer processing, to generate a fusion feature vector, to convert visual information into deep features, so as to improve the recognition accuracy and efficiency. The system then calculates the similarity of the input fabric features and the database sample features, reorders the samples accordingly, and places the most matching fabric in the front row, so as to quickly determine the category of the input fabric and complete the recognition process.

[0070] It should be noted that the execution subject of the embodiment of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a fabric recognition device, a fabric recognition system, etc. that can realize the above functions. The following takes the fabric recognition system as an example to describe the embodiment and the following embodiments.

[0071] Based on this, the embodiment of the present application provides a textile fabric recognition method based on a multi-modal model, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the textile fabric recognition method based on a multi-modal model of the present application is shown in the figure.

[0072] In the embodiment, the textile fabric recognition method based on a multi-modal model includes steps S10-S50:

[0073] Step S10, obtaining multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors and a fabric database;

[0074] It should be noted that the multi-view fabric image refers to fabric pictures taken from different angles and distances (focal length). Specifically, it includes using cameras with different focal lengths to capture large, medium, and small angle images of the fabric. These images correspond to different actual physical areas of the fabric and contain different ranges and details of content information: large angle fabric image: corresponding to a larger actual physical area, containing a larger range of fabric image information, suitable for capturing the overall pattern and structure of the fabric; medium angle fabric image: corresponding to a medium actual physical area, containing a medium range of fabric image information, used to supplement the information between large and small angle images; small angle fabric image: corresponding to a smaller actual physical area, containing detailed image information of the fabric, suitable for capturing the microscopic texture and detail features of the fabric; In addition, the front and back images of the fabric are also considered, as the front and back features of different fabrics may differ greatly, and collecting front and back images of the fabric can more accurately determine the category of the fabric. Fabric language text refers to fabric attribute information described in natural language, which includes fabric composition, pattern, yarn state, finishing process, and other detailed parameters. These information is usually provided by industry experts or system users based on the characteristics and classification standards of the fabric, to describe the physical and chemical properties of the fabric in detail, helping to understand and identify the type and characteristics of the fabric. Different fabrics are divided according to the parameters of textile fabrics, including the following parameters: fabric composition category, fabric composition classification, fabric pattern, yarn state, finishing process, and fabric composition. Fabric parameter division requires strict industry and professional knowledge. The following are some examples of parameters: fabric composition category, such as cotton, chemical fiber, hemp, and artificial leather; fabric composition classification, such as single silk cotton, golden cotton, and washed cotton; fabric pattern, such as plain weave, mixed weave, knit, and mesh; yarn state, such as single yarn, double yarn, and spliced yarn; finishing process, such as face milling, grabbing, gold stamping, printing, and washing; fabric composition, such as cotton, polyester, and cotton + modal. A fabric can be described using language text, for example: a chemical fiber satin double-sided silk fabric with plain weave, satin pattern, high elasticity, dyeing, and polyester fabric picture. When users search, they usually only focus on part of the fabric parameters, such as cotton and washing. Only the part of the language text of interest needs to be input. Fabric color refers to the specific numerical value of fabric color obtained using a colorimeter, usually represented by Lab color space. Lab color space is a color space where L represents lightness (brightness), A represents the component from green to red, and B represents the component from blue to yellow. These color values can accurately describe the color characteristics of the fabric, providing a quantitative data basis for fabric color matching and identification.The fabric database refers to a collection of data that gathers and organizes a large number of fabric samples, including images and other related parameters of different fabrics. The establishment of the fabric database involves the following aspects: fabric sample collection: collecting images of various fabrics, including images from different angles and front and back images; fabric parameter recording: recording various parameters of the fabric, such as composition category, subcategory, pattern, yarn condition, yarn count, spinning process, finishing process, fiber morphology, glossiness, smoothness, fiber length, etc.; fabric classification: dividing fabrics into different categories based on their parameters, providing a foundation for subsequent fabric recognition and search matching; feature vector extraction: using a multi-modal model to extract feature vectors from fabric images in the database and saving these feature vectors in the database for efficient search and matching.

[0075] It can be understood that first, the same kind of fabric needs to be photographed using cameras with different focal lengths to obtain images with large, medium, and small angles. This is done to capture the overall pattern, medium-range details, and microscopic texture of the fabric, ensuring that the characteristics of the fabric can be recorded comprehensively. At the same time, the front and back of the fabric are also photographed, as the characteristics of the front and back of the fabric may differ, which is crucial for accurately identifying the fabric category. In addition, each view of the fabric image corresponds to a detailed language text description, which covers the composition, pattern, yarn condition, and other key attributes of the fabric, providing rich semantic information for the fabric. Furthermore, the color of the fabric is obtained through a professional colorimeter, accurately recorded in the form of Lab color domain values, ensuring the accuracy and consistency of the color. All of these data, including multi-view images, language text descriptions, and color values, as well as the existing fabric database, together form a comprehensive dataset. This dataset not only contains visual information of the fabric but also includes detailed attributes and historical data of the fabric, providing a solid foundation for subsequent model training and fabric recognition, making the entire recognition process more accurate and efficient.

[0076] As an example, the multi-modal model-based textile fabric recognition method is applied to a fabric recognition device, which includes at least two detection cameras with different focal lengths. The step of obtaining multi-view fabric images includes obtaining multiple sets of fabric images, each of which is collected by a detection camera, and each set of fabric images corresponds to each detection camera. According to each fabric image, a multi-view fabric image is determined.

[0077] Fabric recognition equipment refers to an automated machine or system specifically designed to identify different fabrics. This equipment typically includes multiple components such as cameras, lighting systems, image processing units, and possibly robotic arms or other operational components, which work together to capture fabric images and analyze them for identification purposes. Detection cameras refer to the cameras installed on the fabric recognition equipment that are used to capture images of the fabric. These cameras are mounted on the equipment to obtain visual data of the fabric for subsequent image processing and analysis. Focal length refers to the optical parameter of a camera lens that determines how far the camera can "see" and the magnification of the image. Different focal lengths affect the range of view and the level of detail that a camera can capture in an image. For example, a wide-angle lens (short focal length) can capture a wider field of view, while a telephoto lens (long focal length) can capture more distant or detailed images. Multiple sets of fabric images refer to multiple images of the same fabric taken by detection cameras with different focal lengths. These images capture the fabric from different angles and distances to ensure that the fabric's features, including macro patterns, textures, and micro details, are fully recorded. Each set of fabric images corresponds to a specific detection camera, ensuring that visual information is obtained from different angles and focal lengths to provide rich data input for the multi-modal model.

[0078] First, each detection camera captures images of the same fabric, forming multiple sets of fabric images. Each set of images corresponds to a specific camera, ensuring that different angles from wide-angle to telephoto are captured. Then, the images captured by different focal length cameras are combined to form multi-angle fabric images. These images contain both macro features and micro details of the fabric, providing rich visual data for the subsequent multi-modal model, allowing the model to more accurately identify and classify different textile fabrics. In this way, the fabric recognition equipment can fully capture the characteristics of the fabric, improving the accuracy and robustness of identification.

[0079] As an example, the multi-modal model-based textile fabric identification method is applied to fabric recognition equipment, which includes a first camera, a second camera, a third camera, and a fourth camera. The focal length of the first camera is greater than that of the second camera, the focal length of the second camera is greater than that of the third camera, and the focal length of the third camera is greater than that of the fourth camera. The step of obtaining multi-angle fabric images includes obtaining multi-angle fabric images from the first, second, third, and fourth camera images captured by the first, second, third, and fourth cameras. The first angle is smaller than the second angle, the second angle is smaller than the third angle, and the third angle is smaller than the fourth angle.

[0080] The first camera, the second camera, the third camera, and the fourth camera refer to four cameras fixedly configured on the fabric recognition device. They all maintain the same distance from the fabric, but have different focal lengths. The first camera has the longest focal length, suitable for capturing the macro features of the fabric; the second camera has a slightly shorter focal length, used to capture the medium-range features of the fabric; the third camera has a shorter focal length, used to capture more detailed features of the fabric; the fourth camera has the shortest focal length, used to capture the smallest angle range of the fabric, but the details are the most abundant. The first perspective fabric image, the second perspective fabric image, the third perspective fabric image, and the fourth perspective fabric image refer to the fabric images collected by the above four cameras respectively, and each camera corresponds to an image of a certain perspective. Since all cameras are at the same distance from the fabric but have different focal lengths, the perspective range and detail level of the fabric images they capture are also different. The first perspective fabric image shows the largest perspective range, which may contain the overall pattern and structure of the fabric; the second perspective fabric image shows a slightly smaller perspective range, which may contain medium-scale patterns and some details of the fabric; the third perspective fabric image shows a smaller perspective range, which may contain local patterns and more details of the fabric; the fourth perspective fabric image shows the smallest perspective range, which may contain the micro-texture and the most abundant details of the fabric. By combining these images of different perspectives, a comprehensive multi-perspective fabric image can be constructed, providing rich visual data for the multi-modal model, thereby improving the accuracy and robustness of fabric recognition.

[0081] Please refer to Figure 2 , Figure 2 The front and back view of the textile fabric recognition device provided by the first embodiment of the textile fabric recognition method based on the multi-modal model of the present application is shown. This figure shows the configuration and operation interface of the textile fabric recognition device when shooting the front and back images of the fabric. The device is equipped with a double-sided shooting function (not simultaneous shooting of the front and back), which can combine the front and back of the fabric for recognition, ensuring that comprehensive information of the fabric is obtained from two directions. In addition, the device also provides a single-sided shooting option for shooting the front or back of the fabric separately for recognition to adapt to different recognition needs. Users can accurately control the shooting process through the manual shooting mode to ensure image quality. The device also has 2x and 5x image zoom functions, allowing users to zoom in on images as needed to observe the detailed features of the fabric more closely. The integration of these functions enables the fabric recognition device to flexibly adapt to various fabric recognition scenarios, improving the accuracy and efficiency of recognition. In the multi-modal model training mentioned in the disclosure, these front and back images will be used as important input data to help the model learn and recognize the features of the fabric, thereby achieving efficient and accurate fabric recognition in practical applications.

[0082] Please refer to Figure 3 , Figure 3The multi-view fabric image schematic diagram provided by the first embodiment of the textile fabric recognition method based on a multi-modal model of the present application shows multi-modal images of the same fabric under different viewing angles, which recognize the fabric category. The large-angle image, i.e., the first viewing angle, provides the overall appearance of the fabric, which helps the model understand the macro texture and pattern of the fabric; the medium-angle image, corresponding to the second and third viewing angles, reveals the medium-scale features of the fabric, such as the details of the texture and the structure of the yarn; and the small-angle image, i.e., the fourth viewing angle, highlights the microscopic details of the fabric, such as the interweaving of fibers and the surface texture. The collection of these multi-view images enables the multi-modal model to learn the features of the fabric from multiple perspectives, thereby achieving higher accuracy in the fabric recognition task. By analyzing these images, the model can distinguish between different fabric categories, even when the fabrics have similar appearances or textures. This multi-view approach enhances the model's ability to capture fabric details, improving the accuracy and efficiency of textile fabric recognition.

[0083] The fabric recognition device is equipped with four cameras with different focal lengths, namely the first, second, third, and fourth cameras, among which the first camera has the longest focal length and the fourth camera has the shortest focal length. These cameras are installed at equal distances from the fabric to capture images of the fabric. The steps to obtain multi-view fabric images are as follows: first, use the four cameras to simultaneously capture fabric images, and each camera captures fabric images with different viewing angle ranges according to its focal length. The first camera, with the longest focal length, captures the smallest-angle fabric image, mainly containing the detailed features of the fabric; the second camera captures the slightly larger-angle fabric image, containing more medium-range features; the third camera captures the larger-angle fabric image, containing more extensive features of the fabric; finally, the fourth camera, with the shortest focal length, captures the largest-angle fabric image, mainly showing the overall pattern and structure of the fabric. By combining these four fabric images with different viewing angles, a comprehensive multi-view fabric image can be obtained, which contains rich information from macro to micro, providing detailed visual data for the subsequent multi-modal model, thereby enabling more accurate identification and classification of different textile fabrics.

[0084] Step S20, inputting the multi-view fabric image, the fabric language text, and the fabric color into the target multi-modal model to obtain fused features;

[0085] It should be noted that the target multi-modal model refers to a deep learning model designed to process and analyze multi-view fabric images. This model can receive fabric images captured by different cameras as input, including different viewing angles and possible different lighting conditions or detail levels. The core function of the target multi-modal model is to understand and analyze the rich information contained in these multi-view images and extract the key features of the fabric. The target multi-modal model has strong zero-shot learning ability, multi-modal understanding ability and generalization performance. By constructing fabric language and image data pairs, training the multi-modal model with fabric language-image data pairs, the multi-modal model can learn the relationship between language text and images. The multi-modal model includes a fabric language-image contrast learning model and a multi-modal fabric feature extraction model. The fabric language-image contrast learning model only uses a text feature extractor to convert language text into text features. The text features and fabric images are input into the multi-modal fabric feature extraction model to obtain fusion features, which are used for search. The fabric language-image contrast learning model includes an image feature extractor and a text feature extractor, both of which are built using transformer neural networks based on attention mechanisms. The fabric language-image contrast learning model learns the relationship between fabric images and description texts, combining visual information with language information. When a new fabric appears, it has better generalization ability. During fabric retrieval, the text feature extractor in the fabric language-image contrast learning model is used to obtain the deep features of the fabric description language text, which are used as input for the fabric feature extraction model. The text can guide the learning of the fabric feature extractor, resulting in more accurate search results in the context of fabric language text. Typically, users only focus on certain parameters of the fabric, such as cotton, water washing, etc. During model training, partial text is shielded to enable the fabric language-image contrast learning model to learn the relationship between fabric parameters and images.

[0086] Fusion features refer to the comprehensive feature representation obtained by analyzing multi-view fabric images through the target multi-modal model. These features are not simply linearly stacked from different viewing angles, but are effectively integrated through complex calculations and learning processes within the model to obtain a feature vector that represents the essential properties of the fabric. This feature vector captures multi-dimensional information of the fabric, including color, texture, structure, and details, providing strong data support for accurate identification and classification of fabrics. The generation of fusion features is a process of feature extraction, fusion, and classifier prediction by the multi-modal model learning different fabric sample features. The final output fusion feature vector can be used for fabric similarity comparison and identification.

[0087] It can be understood that first, the collected multi-view fabric image, corresponding fabric language text and fabric color data are taken as input data set. These images capture the appearance characteristics of the fabric from different angles, the language text provides detailed attribute description, and the color data is obtained by a colorimeter to represent the Lab color gamut, ensuring accurate recording of the color. Second, these multi-modal data are input into the target multi-modal model. The model first processes the image using a multi-modal feature encoder to extract the visual features of the fabric, and at the same time, a text feature extractor processes the language text to extract the semantic features of the fabric. Then, a feature fusioner fuses these visual features and semantic features to form a comprehensive fusion feature, which not only contains the visual appearance information of the fabric, but also fuses the detailed attributes and color information of the fabric. Finally, this fusion feature is used for fabric recognition and classification, so that the model can more accurately understand and recognize the fabric, improving the accuracy and efficiency of fabric recognition. The effect of this method is that even in the case of similar fabric appearance or similar texture, different fabrics can be accurately distinguished, providing strong technical support for automatic recognition and retrieval of textile fabrics.

[0088] As an example, the multi-view fabric image includes front multi-view fabric image and back multi-view fabric image, the target multi-modal model includes multi-modal feature encoder and multi-modal feature fusioner, the multi-modal feature encoder includes feature embedding layer, attention layer and fully connected layer, and the step of inputting the multi-view fabric image, the fabric language text and the fabric color into the target multi-modal model to obtain the fusion feature includes: inputting the front multi-view fabric image, the back multi-view fabric image, the fabric language text and the fabric color in sequence into the multi-modal feature encoder, and extracting the bottom layer features of the multi-view fabric image and the fabric language text through the feature embedding layer; down-sampling the bottom layer features through the attention layer and reducing the feature dimension to obtain down-sampled features; extracting high-level semantic features in the down-sampled features through the fully connected layer to obtain deep features; and fusing the deep features through the multi-modal feature fusioner to obtain the fusion feature.

[0089] The front multi-view fabric image and the back multi-view fabric image refer to multi-view images collected from the front and back of the fabric, respectively. Each view image is taken by a camera with different focal lengths to capture the overall characteristics of the fabric. The front and back images are both used for identification because the front and back of the fabric may have different texture and structural characteristics. Please refer to Figure 4 , Figure 4The front and back fabric image schematic diagram provided by the first embodiment of the textile fabric recognition method based on the multi-modal model of the present application shows the front and back images of the same fabric. These images serve as input data for the multi-modal model's fabric category recognition. The front image presents the main texture and color of the fabric, while the back image reveals different surface features and subtle color variations, which are crucial for accurate classification of the fabric. In the application of the multi-modal model, these front and back images enable the model to analyze fabric features from multiple angles, improving the accuracy of recognition. By learning the features of these images, the model can distinguish the type and attributes of the fabric, even when the fabric appearance is similar or the texture is similar. In this way, the model can utilize the multi-dimensional information of the fabric to enhance the ability to capture fabric details, improving the accuracy and efficiency of textile fabric recognition.

[0090] The multi-modal feature encoder is part of the target multi-modal model, responsible for converting the input fabric image into a series of feature representations. It usually contains multiple layers such as feature embedding layers, attention layers, and fully connected layers. The multi-modal feature fusioner is another part of the model, which further fuses the deep features extracted by the encoder to obtain a comprehensive feature vector representing the multi-dimensional information of the fabric. Feature embedding layers, attention layers, and fully connected layers are different types of neural network layers that make up the multi-modal feature encoder. The feature embedding layer is responsible for extracting local features in the image, such as edges, textures, etc. The attention layer then down-samples the features output by the feature embedding layer, i.e., reduces the spatial dimension of the features while preserving the most important information, which helps to reduce computational complexity and improve the generalization ability of the model. The fully connected layer maps the features output by the attention layer to a higher-dimensional space to extract more abstract high-level semantic features. Bottom-level features refer to the local features of the fabric image initially extracted by the feature embedding layer. Down-sampling is the operation of the attention layer, which reduces the dimension of the features by reducing the size of the feature map. Feature dimension refers to the spatial size of the feature, for example, width and height in an image. Down-sampled features refer to features processed by the attention layer, whose dimension is reduced. High-level semantic features refer to features extracted by the fully connected layer, which represent more complex and abstract concepts or patterns in the image. Deep features refer to features processed by the fully connected layer, which are high-level, abstract representations of the fabric image, containing rich fabric information.

[0091] First, the front and back multi-angle fabric image and the corresponding fabric language text and color data are input into the multi-modal feature encoder in turn. These input data are first passed through a feature embedding layer, which is responsible for extracting the visual features of the image and the semantic features of the text, providing the underlying features for subsequent processing. Then, these underlying features are passed to the attention layer, which reduces the spatial dimension of the features through a downsampling process while retaining the most critical information, which helps to reduce computational complexity and prevent overfitting. Then, the fully connected layer further processes these downsampled features, extracting higher-level semantic features to form deep features that contain rich attribute information of the fabric. Finally, the multi-modal feature fusioner integrates these deep features to form a comprehensive fusion feature, which comprehensively represents the multi-dimensional information of the fabric, providing a solid foundation for accurate identification and classification of the fabric. The effect of the whole process is to improve the accuracy and efficiency of fabric recognition, ensuring that different fabrics can be accurately distinguished even in the case of similar appearance or similar texture of fabrics.

[0092] Step S30, calculating the similarity of the fusion feature and the fusion feature in the fabric database;

[0093] It should be noted that the fusion feature is generated by the target multi-modal model and represents the key features of the input fabric image, including deep information extracted and fused from the front and back multi-angle fabric images. The fusion feature in the fabric database is the feature vector of the fabric sample previously processed by the same multi-modal model and saved in the database, representing the key features of each fabric sample in the database.

[0094] It can be understood that this process uses specific algorithms such as cosine similarity or Euclidean distance to quantify the similarity of two feature vectors in high-dimensional space. Specifically, the system will compare the fusion feature vector of the input fabric with the fusion feature vector of each fabric sample in the database, calculating a similarity score between 0 and 1, where 1 means exactly the same and 0 means completely different. This score reflects the similarity of the input fabric to each sample in the database. Then, the system sorts the fabric samples in the database according to the similarity score to find the sample closest to the input fabric feature. This process allows accurate identification and classification of fabrics even in the case of a wide variety of fabrics with subtle differences in features, providing an efficient method for rapid retrieval and matching of fabrics.

[0095] Step S40, reordering the fusion features in the fabric database according to the similarity to obtain a reordered result;

[0096] It is noted that the reordering refers to the system reordering the fabric samples in the fabric database according to the similarity scores. The samples with the highest similarity scores will be placed at the front, while the samples with lower similarity scores will be placed at the back. The reordering result refers to the result list of reordering the fabric samples in the fabric database according to the calculated similarity scores.

[0097] It can be understood that by calculating the similarity scores between the input fabric and each sample in the fabric database through the multi-modal model, the system will reorder the samples according to these scores. This process involves sorting all fabric samples in the database from high to low according to their similarity to the input fabric, generating a reordering result list. In this list, the sample with the highest similarity to the input fabric will be placed at the front, while the samples with lower similarity will be placed in turn at the back. The reordering result is equivalent to a sorted fabric sample index, which guides the user or automated system to quickly locate the most likely matching fabric option, thereby achieving efficient retrieval and identification. This method is particularly suitable for cases where the fabric library is large and the fabric features are complex, and can significantly improve the speed and accuracy of fabric searching, bringing convenience to fabric selection, procurement, design and production, etc.

[0098] Step S50, obtaining the fabric category according to the reordering result, and completing the fabric identification.

[0099] It is noted that the fabric category is the result of detailed classification based on different parameters of the fabric, including fabric composition category, fabric composition subcategory, fabric pattern, yarn state, yarn count, spinning process, finishing process, fiber form, glossiness, smoothness, fiber length, etc. These detailed parameters help to divide the textile fabrics on the market into different categories, each category containing fabrics with the same parameters, while different parameters define new categories. Each fabric sample in the fabric database is labeled with these detailed parameters, forming a detailed fabric category system. When the similarity between the input fabric and the fabric samples in the database is compared, the system will determine the most matching fabric samples according to the reordering result of the similarity, and the category to which these most matching samples belong is the predicted fabric category of the input fabric.

[0100] It can be understood that the reordering result refers to a list in which the fabric samples in the database are sorted from high to low in similarity after the multi-modal model compares the similarity of the fusion features of the input fabric with the fusion features of each sample in the fabric database. In this list, the fabric sample ranked first is most similar to the input fabric. According to this reordering result, the system can directly read the class label of the fabric sample ranked first (or a few most similar samples), which comprehensively reflects the detailed parameters of the fabric such as composition, pattern, yarn state, spinning process, color, etc., and represents the specific classification of the fabric. The system takes these class labels as the predicted categories of the input fabric, thereby completing the automatic identification of the fabric. This process not only improves the accuracy and efficiency of identification, but also provides strong technical support for the intelligent upgrading of the textile industry, making the classification and management of fabrics more accurate and efficient.

[0101] The embodiment provides a textile fabric recognition method based on a multi-modal model, acquires multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors and a fabric database; inputs the multi-view fabric images, the fabric language texts and the fabric colors into a target multi-modal model to obtain fusion features; calculates the similarity of the fusion features and fusion features in the fabric database; reorders the fusion features in the fabric database according to the similarity to obtain a reordering result; and obtains a fabric category according to the reordering result to complete fabric recognition. First, multi-view images of the fabric are captured through multiple cameras, including the front and back of the fabric, to ensure that the characteristics of the fabric are captured comprehensively from a macroscopic to a microscopic view, and detailed visual data is provided for recognition. Then, the multi-view images and corresponding fabric language texts and fabric colors are input into a multi-modal model, the model extracts features through a feature embedding layer, performs down-sampling through an attention layer, and generates a fusion feature vector containing multi-dimensional information of the fabric through a full connection layer, which helps to convert the visual information of the fabric into machine-understandable deep features, thereby improving the accuracy and efficiency of textile fabric recognition. Subsequently, the system calculates the similarity scores between the fusion features of the input fabric and the fusion features of each sample in the fabric database, and this step quantitatively evaluates the similarity between the fabrics to provide a scientific basis for fabric matching and recognition. Based on these similarity scores, the system reorders the fabric samples in the database, with the most similar samples to the input fabric ranked first, which helps to quickly filter out the most likely matching fabric samples and significantly improves the efficiency of the recognition process. Finally, the system determines the predicted category of the input fabric according to the category label of the sample ranked first in the reordering result, and completes the recognition of the fabric. This process can quickly and accurately identify the specific category of the input fabric from the fabric database, even in complex environments or when facing similar fabrics, while maintaining a high accuracy. The entire fabric recognition process not only improves the accuracy and efficiency of textile fabric recognition, but also reduces errors caused by human factors, providing technical support for the automated production process of the textile industry. In addition, it also has good flexibility and generalization ability, providing strong technical support for the quality control, inventory management and personalized customization of fabrics.

[0102] Based on the first embodiment of the present application, the same or similar contents as the above-mentioned embodiment one can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 5 , Figure 5 The flowchart of the second embodiment of the textile fabric recognition method based on the multi-modal model of the present application, before step S20 of the textile fabric recognition method based on the multi-modal model, further comprises steps S01-S05:

[0103] Step S01, obtain a target fabric dataset, and divide the target fabric dataset into a training set and a test set;

[0104] It should be noted that the target fabric dataset refers to a collection of a large number of fabric images and their related feature data collected for training and evaluating a multi-modal fabric recognition model. This dataset contains images of various fabrics, each image is taken from different angles and different focal lengths, and contains the front and back of the fabric to ensure that the dataset can fully cover various features of the fabric. These images and their labels (such as fabric composition, pattern, etc. classification information) are used to train the model to enable it to recognize and classify different fabrics. The training set is a subset of data used to train the model, containing a series of fabric images with known labels. During the training process, the model learns how to recognize and classify different fabric features by analyzing these images and their corresponding labels. The role of the training set is to enable the model to "learn" the characteristics of the fabric and the corresponding classification relationship. The test set is a subset of data used to evaluate the performance of the model, which contains fabric images that have not been used in the model training process. The purpose of the test set is to verify the recognition ability and accuracy of the model when processing unseen fabric images. Through the performance on the test set, the generalization ability of the model, i.e. the recognition ability of the model on new fabric data, can be evaluated.

[0105] It can be understood that first, the target fabric dataset is collected, which includes using multiple cameras to take the front and back of the fabric at different angles and focal lengths to capture the overall characteristics of the fabric. Then, these images are labeled and their detailed attributes such as composition, pattern, etc. are recorded. Next, in order to train and evaluate the model, the dataset needs to be divided into a training set and a test set. The division is usually done in a certain proportion, such as 80% of the data used as the training set and the remaining 20% as the test set. This division can be achieved by random selection or using specific data segmentation techniques, the purpose is to ensure that the samples in the training set and the test set can represent the diversity of the entire dataset. The training set is used for the model's learning and parameter adjustment, enabling the model to recognize fabric features and classification. The test set is used to evaluate the model's performance on unseen data after the model training is completed, thereby verifying the accuracy and generalization ability of the model. In this way, a model that performs well on training data and can also accurately recognize new, unseen fabric data is trained, improving the accuracy and efficiency of textile fabric recognition.

[0106] As an example, the step of obtaining the target fabric dataset includes: obtaining an initial fabric dataset and historical fabric data; performing evaluation screening on the initial fabric dataset to obtain to-be-labeled data; comparing the to-be-labeled data with the historical fabric data to obtain a similarity; sorting the to-be-labeled data according to the similarity, and filtering the to-be-labeled data with a similarity lower than a preset similarity threshold to obtain the target fabric dataset.

[0107] The initial fabric dataset refers to a newly collected set of fabric images that have not been labeled or recognized. The historical fabric data refers to previously collected and labeled fabric data that already contains detailed information about the fabric, such as composition, texture, etc., and has been used to train or verify the model. Evaluation screening refers to the process of quality control and relevance evaluation of images in the initial fabric dataset. The purpose of this step is to determine which images are valuable and worthy of further processing and labeling. Evaluation may be based on factors such as image clarity, whether it contains sufficient fabric details, whether the image is representative, etc. To-be-labeled data refers to the set of images that have been selected for further labeling after evaluation screening. These images are selected because they provide sufficient information to describe the characteristics of the fabric, but currently do not have specific labels or classification information. Similarity refers to the use of specific algorithms (such as feature matching, pattern recognition, etc.) to quantify the similarity between two sets of data. In fabric recognition, similarity usually refers to the closeness of visual features between to-be-labeled fabric images and existing images in the historical fabric data. The preset similarity threshold is a previously set numerical standard used to determine the acceptable degree of similarity between fabric images. When the similarity between to-be-labeled fabric images and images in the historical fabric database is calculated, only those images with a similarity higher than the threshold will be retained as part of the target fabric dataset. The purpose of setting this threshold is to filter out fabric images with low similarity to existing datasets, so as to ensure that the images in the dataset have high uniqueness and reduce data redundancy, thereby making subsequent fabric category recognition more accurate. For example, if the preset similarity threshold is set to 70%, only to-be-labeled fabric images with a similarity of more than 70% to existing images in the database will be retained for subsequent labeling and model training, which can ensure that the model can learn more different fabric features and improve the accuracy of recognition.

[0108] First, an initial fabric dataset needs to be collected, which usually involves taking or scanning a series of new fabric sample images, while also utilizing existing historical fabric data, i.e., a previously labeled fabric image collection. Next, the images in the initial fabric dataset are evaluated and filtered for quality, removing those that are unclear or not representative, leaving high-quality images to be labeled. Then, using image processing and pattern recognition techniques, the images to be labeled are compared with the images in the historical fabric dataset, calculating their similarity. This step usually involves extracting image features and using algorithms such as cosine similarity to quantify the similarity between images. According to the calculated similarity, the data to be labeled is sorted, and a preset similarity threshold is set, such as 80%, only retaining images with a similarity higher than this threshold, and filtering out images with a lower similarity. In this way, the final target fabric dataset only contains fabric images with high similarity to existing data, ensuring the diversity and novelty of the dataset, and laying a solid foundation for training an accurate and efficient fabric recognition model. Through these steps, the amount of data processed can be reduced, and the accuracy of fabric recognition can be improved.

[0109] Step S02, training and adjusting the initial multi-modal model using the training set to obtain a reference multi-modal model;

[0110] It should be noted that the initial multi-modal model refers to the basic multi-modal model before training begins. This model usually contains multiple modules that can handle images of different perspectives and front and back fabrics. The initial model may be pre-designed based on previous research or existing model architecture, and it has the basic ability to extract and fuse features from multi-perspective fabric images, but has not been trained and parameter-adjusted on a specific dataset. The reference multi-modal model is a multi-modal model that has been trained and adjusted using the training set. During the training process, the parameters of the initial model are optimized based on the fabric images and their labels in the training set, so that the model can more accurately capture and fuse the features of the fabric and improve the recognition accuracy. After this process, the model becomes more suitable for handling specific fabric recognition tasks and can be used as a reference or benchmark for subsequent testing, comparison, or further optimization. The reference multi-modal model can be used to evaluate performance on the test set and can serve as a basis for further research and development.

[0111] It can be understood that the collected large number of fabric images and their corresponding detailed labels are first input into the initial model. This initial model has the ability to handle multi-view and front-back fabric images, but still needs to be further optimized through actual data. The training set is composed of screened fabric images, including various features and classification information of the fabric. During the training process, the parameters of the model are continuously adjusted to minimize the difference between the predicted label and the actual label. This process involves updating the weights and biases of the model, with the aim of improving the accuracy of the model in identifying fabric features and classification. Through repeated iterations of data on the training set, the initial model gradually learns how to extract key information from images and make accurate classifications. Finally, the trained and adjusted model becomes a reference multi-modal model that can more accurately reflect the characteristics of the fabric and provide an optimized benchmark for fabric recognition tasks. This reference model not only improves the accuracy of recognition, but also lays the foundation for subsequent testing and application.

[0112] As an example, the training set includes image data, category data, fabric description language text data, and fabric color data, the image data includes front-back multi-view image data, front multi-view image data, back multi-view image data, multi-view image data missing second and third views, and first-view front image data, the initial multi-modal model includes a multi-modal feature encoder, a multi-modal feature fusioner, and a multi-modal feature classifier, and the step of training and adjusting the initial multi-modal model using the training set to obtain a reference multi-modal model includes: inputting the fabric description language text data, the fabric color data, the front-back multi-view image data, the front multi-view image data, the back multi-view image data, the multi-view image data missing second and third views, or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features; fusing the fabric multi-modal deep features through the multi-modal feature fusioner to obtain fabric multi-modal fusion features; learning and distinguishing the fabric multi-modal fusion features through the multi-modal feature classifier to obtain fabric categories; when the fabric categories do not match the category data, adjusting the multi-modal feature encoder, the multi-modal feature fusioner, and the multi-modal feature classifier, and returning to the step of inputting the fabric description language text data, the fabric color data, the front-back multi-view image data, the front multi-view image data, the back multi-view image data, the multi-view image data missing second and third views, or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features; when the fabric categories match the category data, obtaining a reference multi-modal model according to the multi-modal feature encoder, the multi-modal feature fusioner, and the multi-modal feature classifier.

[0113] Fabric description language text data refers to text information that describes fabric attributes using natural language, including fabric composition, pattern, yarn condition, finishing process, etc. For example, "a synthetic satin double-sided silk fabric, woven-satin, high elasticity, dyed, polyester fabric picture." This description provides key features of the fabric, which can be used to train the model to identify and retrieve fabrics. Fabric color refers to the color presented on the surface of the fabric, and fabric color data is the specific numerical value of the fabric color obtained by a colorimeter. In the textile industry, it is usually represented by Lab color space, where L represents lightness, A represents the component from green to red, and B represents the component from blue to yellow. These color data help the model consider color matching when retrieving fabrics. Positive-negative multi-view image data includes multi-view images of the front and back of the fabric, and each view is captured by a camera with different focal lengths to capture the features of the fabric at large, medium and small angles. This data enables the model to understand the appearance and structure of the fabric from different angles. Positive multi-view image data (missing back modal picture): when only the multi-view images of the front of the fabric are available, the model needs to be able to process single-sided data, which requires the model to be able to identify and retrieve fabrics based only on the features of the front image. Negative multi-view image data (missing positive modal picture): corresponding to positive multi-view image data, this case involves only multi-view images of the back of the fabric, and the model needs to be able to process these data and identify the features of the fabric. Missing second and third view multi-view image data (missing medium angle modal picture): in some cases, the medium angle image may be missing, and the model needs to be able to adapt to this situation and perform effective fabric identification and retrieval even when some view data is missing. First view positive image data (missing back modal picture, medium and small angle modal pictures): this refers to the availability of only the first view image of the front of the fabric, with missing back and medium and small angle images, and the model needs to be able to handle this incomplete data situation and identify the fabric as accurately as possible. Image data refers to fabric images captured from multiple angles, including the front and back of the fabric, which constitute the main part of the training set. Class data refers to label information corresponding to these images, such as fabric category, composition, texture, etc. classification identification, used to tell the model the correct category of each image. Multi-modal feature encoder, multi-modal feature fusioner and multi-modal feature classifier are key components of the initial multi-modal model, multi-modal feature encoder is responsible for extracting deep features from input fabric images, multi-modal feature fusioner further integrates these deep features to form a more comprehensive fabric feature representation, and multi-modal feature classifier uses these integrated features to predict the category of the fabric. Fabric multi-modal deep features refer to the fabric features obtained after the multi-modal feature encoder processes the image data, which contain deep information of the image such as texture, color, shape, etc., and are the basis for fabric recognition.Fabric multi-modal fusion features refer to the features obtained after the multi-modal feature fusioner further processes the deep features. These fusion features integrate information from different perspectives and aspects, providing a more comprehensive basis for classification. Fabric categories refer to specific categories that fabrics are classified into based on their physical properties and visual characteristics, such as "cotton plain" or "silk jacquard". During the training process, the goal of the model is to accurately predict the category label of the input fabric image.

[0114] First, a training set containing a large number of fabric images and their corresponding category labels is used. The image data of the training set captures the multi-perspective and front-back features of the fabric, while the category data provides information such as the type of fabric to which each image belongs. The initial multi-modal model consists of three main parts: multi-modal feature encoder, multi-modal feature fusioner, and multi-modal feature classifier. At the beginning of training, image data is input into the multi-modal feature encoder, which extracts multi-modal deep features of the fabric through a series of deep learning operations. These features contain key visual information of the fabric. Subsequently, these deep features are sent to the multi-modal feature fusioner, which fuses information from different perspectives and feature layers to form more comprehensive fabric multi-modal fusion features. Next, the multi-modal feature classifier learns and classifies these fusion features to predict the category of the fabric. If the classifier's predicted category does not match the actual category data in the training set, the parameters of the encoder, fusioner, and classifier will be adjusted, and the process will be repeated to optimize the model's performance. When the model's predicted category matches the actual category data, it means that the model has been able to accurately learn and distinguish the features of different fabrics, and a well-trained and adjusted reference multi-modal model has been obtained, which can accurately identify and classify fabrics and provide strong support for fabric identification tasks in practical applications.

[0115] Please refer to Figure 6 , Figure 6The fabric color recognition diagram provided by Embodiment Two of the textile fabric recognition method based on a multi-modal model of the present application shows the process of fabric color recognition, which contains specific numerical values of color measurement data representing different dimensional characteristics of fabric color. In the diagram, three key data points can be seen: 81.48, 8.11, and 0.11, which may correspond to the parameters of color lightness, saturation, and hue, respectively. These parameters are key features in the multi-modal model for identifying and distinguishing different fabric colors. By inputting fabric images into the multi-modal feature encoder, the model can extract these color features and combine them with other visual features (such as texture, pattern) through the multi-modal feature fusioner, and finally classified by the multi-modal feature classifier to achieve accurate identification of fabric color. This color recognition method helps improve the accuracy of fabric classification, especially in automated textile production and quality control, ensuring the consistency and accuracy of fabric color to meet specific design and production requirements.

[0116] Please refer to Figure 7 , Figure 7 The initial multi-modal model structure diagram provided by Embodiment Two of the textile fabric recognition method based on a multi-modal model of the present application shows the structure of the initial multi-modal model, which is designed to process and analyze multi-modal data of textile fabrics. The model includes three main parts: multi-modal feature encoder, multi-modal feature fusioner, and multi-modal feature classifier. The multi-modal feature encoder is responsible for extracting deep features from the multi-view images of the fabric, including the front and back of the fabric, as well as large, medium, and small angle images of different focal lengths. Then, the multi-modal feature fusioner integrates these deep features with the features extracted from the language text description of the fabric to form a comprehensive fabric feature representation. Finally, the multi-modal feature classifier uses these fused features to identify and classify the fabric, outputting the category of the fabric. This structure enables the model to capture fabric features from multiple dimensions, including visual information and language description, thereby improving the accuracy and robustness of retrieval. Through this structure, the model can effectively process and compare fabric images, text descriptions, and color data, achieving efficient and accurate fabric retrieval to meet the needs of the textile industry in intelligent upgrading, quality control, and personalized customization.

[0117] Please refer to Figure 8 , Figure 8 The initial multi-modal model training diagram provided by Embodiment Two of the textile fabric recognition method based on a multi-modal model of the present application corresponds to Figure 7Fabric image and image feature encoder, which shows a schematic diagram of the initial multimodal model training process, where the front and back images of the fabric are captured at large, medium, and small angles to comprehensively obtain the fabric's feature information. These images are first passed through a multimodal feature encoder, which is responsible for extracting deep features from the images at each angle. Subsequently, these features are sent to a multimodal feature fusioner, where the features from different angles are integrated to form a comprehensive fabric feature representation. Finally, the fused features are sent to a multimodal feature classifier, where the model learns to distinguish between different fabric categories. The entire process is automated, aiming to optimize the model's parameters through deep learning techniques to ensure that it can accurately identify and classify various fabrics, even when the fabrics have similar appearances or textures. In this way, the model can learn the fabric's features at different scales and angles, making it more accurate in fabric classification and identification in practical applications.

[0118] Step S03, testing and verifying the target detection model through the test set to obtain test results;

[0119] It should be noted that testing and verification refers to using an independent test set to evaluate the performance of the target detection model, and the test set contains fabric images and their category labels that have not been used in the model training process. During testing and verification, the model will make predictions on the images in the test set, trying to identify and classify the fabric without using the actual label information of these images. This step is to test the accuracy and generalization ability of the model when dealing with new data, ensuring that it is not just "remembering" the labels of the training set, but truly learning how to identify the features of the fabric. The test results are the model's performance on the test set, including various performance indicators such as accuracy, precision, recall, and F1 score, etc. These indicators show the matching degree between the model's predicted categories and the actual category labels in the test set. The test results can tell us how the model may perform in practical applications, whether it can accurately identify the category of the fabric, and its stability and reliability under different circumstances. If the test results meet the predetermined performance standards, the model can be considered effective and can be used for actual fabric identification tasks. If the test results are not satisfactory, further adjustments to the model or retraining may be needed.

[0120] It can be understood that the model processes the images in the test set, using the rules learned in the training stage to predict the fabric category of each image. After the prediction is completed, the prediction results of the model are compared with the true labels in the test set, and the performance of the model is measured by calculating performance indicators such as accuracy, recall rate and F1 score. These indicators reflect the accuracy and consistency of the model in identifying fabric categories. The final test results can indicate whether the model can reliably identify fabrics and whether it is suitable for deployment in practical applications. If the test results do not meet the expected standards, further adjustments and optimizations of the model may be needed. This process ensures the generalization ability of the model on new data and provides an important reference for actual deployment.

[0121] Step S04, when the test results do not meet the preset requirements, returning to the step of training and adjusting the initial multi-modal model using the training set.

[0122] It should be noted that the preset requirements refer to a series of standards or thresholds set before model testing and validation to evaluate model performance. These requirements usually include the minimum accuracy, recall rate, precision rate or other related indicators that the model should achieve on the test set. For example, if the preset requirement is that the accuracy of the model needs to be above 95%, then the test results need to meet this condition for the model to be considered effective.

[0123] It can be understood that when the test results show that the performance of the model does not meet these preset requirements, such as accuracy below 95%, it indicates that the model may need further training or adjustment to improve its recognition ability. In this case, it is necessary to return to the step of training and adjusting the initial multi-modal model using the training set, which may include adjusting model parameters, increasing training data, performing data enhancement, adjusting network structure or optimizing algorithm, etc. to improve the performance of the model to meet the preset requirements. This process may need to be iterated several times until the performance of the model on the test set reaches a satisfactory level.

[0124] Step S05, when the test results meet the preset requirements, the reference multi-modal model is used as the target multi-modal model.

[0125] It can be understood that when the test result meets the preset requirement, for example, the accuracy rate reaches more than 95%, which means that the model can also accurately perform the identification task on the unseen fabric image, showing good generalization ability. In this case, the reference multi-modal model, that is, the model after training and adjustment, is confirmed to be effective and is used as the target multi-modal model for the actual fabric identification task. This model has proven its reliability and accuracy in handling diverse fabric images, so it can be deployed in textile production, quality control or fabric recommendation systems to automate and optimize the classification and identification process of fabrics.

[0126] The embodiment obtains a target fabric dataset and divides the target fabric dataset into a training set and a test set; trains and adjusts an initial multi-modal model using the training set to obtain a reference multi-modal model; tests and verifies the target detection model through the test set to obtain a test result; when the test result does not meet the preset requirement, returns to the step of training and adjusting the initial multi-modal model using the training set; when the test result meets the preset requirement, uses the reference multi-modal model as a target multi-modal model. First, a target fabric dataset is obtained by collecting multi-view images of different fabrics and their attribute labels, and the dataset is carefully divided into a training set and a test set. This is done to ensure that the model can learn the characteristics of the fabric during the training phase, and at the same time verify the model's recognition ability for new data during the test phase. Then, the training set is used to train and adjust the initial multi-modal model, the purpose of which is to enable the model to learn how to accurately extract the characteristics of the fabric and classify it by learning a large number of fabric images and corresponding labels. Then, the trained model is tested and verified through the test set, and the test result is obtained. This step is to evaluate the generalization ability of the model to ensure that it can accurately identify unseen fabric images. If the test result does not meet the preset performance requirement, such as accuracy rate not exceeding the set threshold, the model will return to the training phase for further optimization and adjustment, which helps to improve the robustness of the model and ensure its stable work under various conditions. Finally, when the test result meets the preset requirement, the reference multi-modal model is confirmed to be effective and is used as the target multi-modal model for the actual fabric identification task. Through this process, a high-performance fabric identification model is developed, which can improve the accuracy and efficiency of textile fabric identification, reduce human error, improve production efficiency, and provide strong technical support for automated and intelligent textile production processes. In addition, this method also enhances the adaptability and flexibility of the model, enabling it to cope with various fabrics and environmental conditions, providing technical support for the innovation and development of the textile industry.

[0127] The application also provides a textile fabric identification device based on a multi-modal model, which is described in detail in the followingFigure 9 The textile fabric recognition device based on a multi-modal model comprises:

[0128] A data acquisition module 10 is configured to acquire multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors, and a fabric database.

[0129] A model processing module 20 is configured to input the multi-view fabric images, the fabric language texts, and the fabric colors into a target multi-modal model to obtain fusion features.

[0130] A similarity calculation module 30 is configured to calculate similarities between the fusion features and fusion features in the fabric database.

[0131] A reordering module 40 is configured to reorder the fusion features in the fabric database according to the similarities to obtain a reordering result.

[0132] A fabric recognition module 50 is configured to obtain a fabric category according to the reordering result to complete fabric recognition.

[0133] In an embodiment, the data acquisition module 10 is further configured to acquire a plurality of groups of fabric images, each of the fabric images is acquired by the detection camera, each of the groups of fabric images corresponds to one of the detection cameras, and the multi-view fabric images are determined according to each of the fabric images.

[0134] In an embodiment, the data acquisition module 10 is further configured to obtain multi-view fabric images according to first-view fabric images, second-view fabric images, third-view fabric images, and fourth-view fabric images acquired by the first camera, the second camera, the third camera, and the fourth camera, the first view is smaller than the second view, the second view is smaller than the third view, and the third view is smaller than the fourth view.

[0135] In an embodiment, the model processing module 20 is further configured to input the front multi-view fabric images, the back multi-view fabric images, the fabric language texts, and the fabric colors into the multi-modal feature encoder in sequence, extract low-level features of the multi-view fabric images and the fabric language texts through the feature embedding layer, down-sample the low-level features through the attention layer, reduce the feature dimension, obtain down-sampled features, extract high-level semantic features in the down-sampled features through the full connection layer, obtain deep features, and fuse the deep features through the multi-modal feature fusioner to obtain fusion features.

[0136] In an embodiment, the model processing module 20 is further configured to obtain a target fabric data set, and divide the target fabric data set into a training set and a test set; train and adjust an initial multi-modal model using the training set to obtain a reference multi-modal model; test and verify the target detection model through the test set to obtain a test result; when the test result does not meet a preset requirement, return to the step of training and adjusting the initial multi-modal model using the training set; and when the test result meets the preset requirement, use the reference multi-modal model as a target multi-modal model.

[0137] In an embodiment, the model processing module 20 is further configured to obtain an initial fabric data set and historical fabric data; evaluate and screen the initial fabric data set to obtain to-be-labeled data; compare the to-be-labeled data with the historical fabric data to obtain a similarity; sort the to-be-labeled data according to the similarity, and filter the to-be-labeled data with a similarity lower than a preset similarity threshold to obtain a target fabric data set.

[0138] In an embodiment, the model processing module 20 is further configured to input the fabric description language text data, the fabric color data, the front-reverse multi-view image data, the front multi-view image data, the reverse multi-view image data, the multi-view image data missing the second and third views, or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features; fuse the fabric multi-modal deep features through the multi-modal feature fusioner to obtain fabric multi-modal fusion features; learn and distinguish the fabric multi-modal fusion features through the multi-modal feature classifier to obtain a fabric category; when the fabric category does not match the category data, adjust the multi-modal feature encoder, the multi-modal feature fusioner, and the multi-modal feature classifier, and return to the step of inputting the fabric description language text data, the fabric color data, the front-reverse multi-view image data, the front multi-view image data, the reverse multi-view image data, the multi-view image data missing the second and third views, or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features; and when the fabric category matches the category data, obtain a reference multi-modal model according to the multi-modal feature encoder, the multi-modal feature fusioner, and the multi-modal feature classifier.

[0139] The textile fabric recognition device based on the multi-modal model provided in the application adopts the textile fabric recognition method based on the multi-modal model in the above embodiment, and can solve the technical problem of how to improve the accuracy and efficiency of textile fabric recognition. Compared with the prior art, the beneficial effects of the textile fabric recognition device based on the multi-modal model provided in the application are the same as those of the textile fabric recognition method based on the multi-modal model provided in the above embodiment, and other technical features of the textile fabric recognition device based on the multi-modal model are the same as those disclosed in the above embodiment method, which will not be repeated here.

[0140] The application provides a textile fabric recognition device based on a multi-modal model. The textile fabric recognition device based on the multi-modal model comprises at least one processor and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the textile fabric recognition method based on the multi-modal model in the above embodiment one.

[0141] Reference will be made to the following description Figure 10 which shows a structural diagram of a textile fabric recognition device based on a multi-modal model suitable for implementing the embodiments of the application. The textile fabric recognition device based on the multi-modal model in the embodiments of the application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The textile fabric recognition device based on the multi-modal model shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the application.

[0142] As Figure 10As shown, the textile fabric recognition device based on multi-modal model can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the textile fabric recognition device based on multi-modal model are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the textile fabric recognition device based on multi-modal model to communicate wirelessly or wired with other devices to exchange data. Although the textile fabric recognition device based on multi-modal model with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0143] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0144] The textile fabric recognition device based on the multi-modal model provided in the application adopts the textile fabric recognition method based on the multi-modal model in the above embodiment, and can solve the technical problem of how to improve the accuracy and efficiency of textile fabric recognition. Compared with the prior art, the beneficial effects of the textile fabric recognition device based on the multi-modal model provided in the application are the same as those of the textile fabric recognition method based on the multi-modal model provided in the above embodiment, and other technical features of the textile fabric recognition device based on the multi-modal model are the same as those disclosed in the previous embodiment method, which will not be repeated here.

[0145] The application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon for executing the textile fabric recognition method based on the multi-modal model in the above embodiment.

[0146] The computer readable storage medium provided in the application can be a U disk, an electric, magnetic, optical, electromagnetic, infrared or semiconductor system, device or component, or any combination thereof. More specific examples include, but are not limited to: an electric connection (one or more wires), a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer readable storage medium can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, device or component. Program code can be transmitted through electric wires, optical cables, radio frequency (RF) or any suitable combination thereof.

[0147] The above computer readable storage medium can be included in the textile fabric recognition device based on the multi-modal model; or it can exist separately and not be assembled into the textile fabric recognition device based on the multi-modal model.

[0148] The above computer readable storage medium carries one or more programs, which, when executed by the textile fabric recognition device based on the multi-modal model, cause the textile fabric recognition device based on the multi-modal model to: acquire a multi-view fabric image, a fabric language text corresponding to the multi-view fabric image, a fabric color, and a fabric database; input the multi-view fabric image, the fabric language text and the fabric color into a target multi-modal model to obtain a fusion feature; calculate the similarity of the fusion feature and the fusion feature in the fabric database; reorder the fusion features in the fabric database according to the similarity to obtain a reordering result; obtain a fabric category according to the reordering result to complete fabric recognition.

[0149] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or a procedural programming language, such as the C programming language. The code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through a local area network or a wide area network, or the connection can be made to an external computer, for example, through the Internet using an Internet service provider.

[0150] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams represents a module, or a portion of code that comprises specific logical functions. The order in which the blocks are presented is not necessarily the order in which the functions are performed. The functions can be performed in any order, or in parallel, depending on the circumstances. The blocks and combinations thereof can be implemented by special purpose hardware-based systems that perform the specified functions or by combinations of software and hardware.

[0151] The modules involved in the embodiments of the present application can be implemented by software or by hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0152] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above-mentioned textile fabric identification method based on a multi-modal model, and can solve the technical problem of how to improve the accuracy and efficiency of textile fabric identification. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the multi-modal model based textile fabric identification method provided by the above-mentioned embodiments, and will not be described here.

[0153] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the multi-modal model based textile fabric identification method as described above.

[0154] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy and efficiency of textile fabric identification. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the multi-modal model based textile fabric identification method provided by the above-mentioned embodiments, and will not be described here.

Claims

1. A method for textile fabric recognition based on a multi-modal model, characterized in that, The method is applied to a cloth recognition device, and the method comprises: obtaining multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors, and a fabric database, wherein the multi-view fabric images are fabric pictures taken from the same fabric distance, different angles, and different focal lengths, the multi-view fabric images comprise front multi-view fabric images and back multi-view fabric images, and the fabric colors are specific numerical values of fabric colors obtained by a colorimeter; inputting the multi-view fabric images, the fabric language texts, and the fabric colors into a target multi-modal model to obtain fusion features; calculating similarities between the fusion features and fusion features in the fabric database; reordering the fusion features in the fabric database according to the similarities to obtain a reordering result; obtaining a fabric category according to the reordering result to complete fabric recognition; wherein the cloth recognition device comprises a first camera, a second camera, a third camera, and a fourth camera, the focal length of the first camera is greater than the focal length of the second camera, the focal length of the second camera is greater than the focal length of the third camera, and the focal length of the third camera is greater than the focal length of the fourth camera; the step of obtaining multi-view fabric images comprises: obtaining multi-view fabric images from first-view fabric images, second-view fabric images, third-view fabric images, and fourth-view fabric images collected by the first camera, the second camera, the third camera, and the fourth camera, wherein the first view is smaller than the second view, the second view is smaller than the third view, the third view is smaller than the fourth view, the first-view fabric images are used to provide the overall appearance of the fabric, the second-view fabric images and the third-view fabric images are used to provide medium-scale features of the fabric, and the fourth-view fabric images are used to provide microscopic details of the fabric.

2. The method of claim 1, wherein, The cloth recognition device comprises at least two detection cameras, and the focal lengths of the detection cameras are different; the step of obtaining multi-view fabric images comprises: obtaining multiple groups of fabric images, each of the fabric images is collected by a detection camera, and each of the fabric images corresponds to a detection camera; determining multi-view fabric images according to the fabric images.

3. The method of claim 1, wherein, The target multi-modal model comprises a multi-modal feature encoder and a multi-modal feature fusioner, the multi-modal feature encoder comprises a feature embedding layer, an attention layer, and a fully connected layer, and the step of inputting the multi-view fabric images, the fabric language texts, and the fabric colors into the target multi-modal model to obtain fusion features comprises: inputting the front multi-view fabric images, the back multi-view fabric images, the fabric language texts, and the fabric colors into the multi-modal feature encoder in sequence, and extracting bottom features of the multi-view fabric images and the fabric language texts through the feature embedding layer; down-sampling the bottom features and reducing the feature dimension through the attention layer to obtain down-sampled features; extracting high-level semantic features in the down-sampled features through the fully connected layer to obtain deep features; The deep features are fused by the multi-modal feature fusioner to obtain fused features.

4. The method of claim 1, wherein, Before the step of inputting the multi-view fabric image, the fabric language text and the fabric color into a target multi-modal model to obtain fused features, the method further comprises: obtaining a target fabric dataset and dividing the target fabric dataset into a training set and a test set; training and adjusting an initial multi-modal model using the training set to obtain a reference multi-modal model; testing and verifying the reference multi-modal model through the test set to obtain a test result; when the test result does not meet the preset requirement, returning to the step of training and adjusting the initial multi-modal model using the training set; when the test result meets the preset requirement, taking the reference multi-modal model as the target multi-modal model.

5. The method of claim 4, wherein, The step of obtaining the target fabric dataset comprises: obtaining an initial fabric dataset and historical fabric data; evaluating and screening the initial fabric dataset to obtain data to be labeled; comparing the data to be labeled with the historical fabric data to obtain a similarity; sorting the data to be labeled according to the similarity and filtering the data to be labeled with a similarity lower than a preset similarity threshold to obtain the target fabric dataset.

6. The method of claim 4, wherein, The training set comprises image data, category data, fabric description language text data and fabric color data, the image data comprises front-reverse multi-view image data, front multi-view image data, reverse multi-view image data, multi-view image data missing second and third views and first-view front image data, the initial multi-modal model comprises a multi-modal feature encoder, a multi-modal feature fusioner and a multi-modal feature classifier, and the step of training and adjusting the initial multi-modal model using the training set to obtain a reference multi-modal model comprises: inputting the fabric description language text data, the fabric color data, the front-reverse multi-view image data, the front multi-view image data, the reverse multi-view image data, the multi-view image data missing second and third views or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features; fusing the fabric multi-modal deep features through the multi-modal feature fusioner to obtain fabric multi-modal fused features; learning and distinguishing the fabric multi-modal fused features through the multi-modal feature classifier to obtain a fabric category; when the fabric category does not match the category data, adjusting the multi-modal feature encoder, the multi-modal feature fusioner and the multi-modal feature classifier and returning to the step of inputting the fabric description language text data, the fabric color data, the front-reverse multi-view image data, the front multi-view image data, the reverse multi-view image data, the multi-view image data missing second and third views or the first-view front image data into the multi-modal feature encoder to obtain fabric multi-modal deep features; When the fabric category matches the category data, a reference multi-modal model is obtained according to the multi-modal feature encoder, the multi-modal feature fusioner and the multi-modal feature classifier.

7. A textile fabric recognition device based on a multi-modal model, characterized in that, The device comprises: The data acquisition module is configured to acquire multi-view fabric images, fabric language texts corresponding to the multi-view fabric images, fabric colors and a fabric database, wherein the multi-view fabric images are fabric pictures taken from the same fabric distance, different angles and different focal lengths, the multi-view fabric images include front multi-view fabric images and back multi-view fabric images, and the fabric colors are specific values of fabric colors obtained by a colorimeter. The model processing module is configured to input the multi-view fabric images, the fabric language texts and the fabric colors into a target multi-modal model to obtain fusion features. The similarity calculation module is configured to calculate similarities between the fusion features and fusion features in the fabric database. The reordering module is configured to reorder the fusion features in the fabric database according to the similarities to obtain a reordering result. The fabric recognition module is configured to obtain a fabric category according to the reordering result to complete fabric recognition. The data acquisition module is configured to obtain multi-view fabric images from first-view fabric images, second-view fabric images, third-view fabric images and fourth-view fabric images collected by a first camera, a second camera, a third camera and a fourth camera, wherein the first view is smaller than the second view, the second view is smaller than the third view, the third view is smaller than the fourth view, the first-view fabric images are used to provide overall appearances of the fabric, the second-view fabric images and the third-view fabric images are used to provide medium-scale features of the fabric, and the fourth-view fabric images are used to provide microscopic details of the fabric. The focal length of the first camera is greater than the focal length of the second camera, the focal length of the second camera is greater than the focal length of the third camera, and the focal length of the third camera is greater than the focal length of the fourth camera. 8.A textile fabric recognition device based on a multi-modal model, characterized in that, The device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the multi-modal model-based textile fabric recognition method according to any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the multi-modal model-based textile fabric recognition method according to any one of claims 1 to 6.

Images