Knitted pattern retrieval method and system based on sketch and data driving, computer readable storage medium and computer program product
By combining generative adversarial networks and sketch recognition networks, the problem of low efficiency in knitted pattern recognition and retrieval is solved, enabling fast and accurate conversion and efficient retrieval of knitted patterns, which is suitable for the automation needs of the knitting industry.
Patent Information
- Application Number
- CN202511062507.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-14
AI Technical Summary
Existing knitted pattern recognition and retrieval methods are inefficient and inaccurate, making it difficult to meet the needs of industrial automation, especially in the retrieval of specified images based on user input, where there is a lack of stable and reliable data-driven solutions.
Generative adversarial networks (such as pix2pix or CycleGAN models) are used to convert knitted patterns into sketches, and an index database is built through a sketch recognition network. GALIF features and a bag-of-words visual model are used for efficient retrieval, including the sketch acquisition, training, generation and retrieval process.
It enables rapid and accurate automatic conversion and retrieval of knitted patterns, improves the speed and accuracy of knitted pattern recognition and retrieval, solves the problem of low efficiency in traditional methods, and adapts to the characteristics of high repetition and lack of specific objects in the knitting industry.
Smart Images
Figure CN120950713A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a sketch- and data-driven method, system, computer-readable storage medium, and computer program product for retrieving knitted patterns. Background Technology
[0002] In the knitting industry, accurate and rapid identification and retrieval of knitted patterns is crucial for industrial automation. Because knitted patterns are characterized by rich texture information, lack of object representation, high repetition and regularity, and difficulty in description, traditional retrieval methods rely on labels. This approach is neither intuitive nor efficient. In recent years, some manufacturers have used images, particularly user-input images, for matching and retrieval. However, relying solely on images for retrieval remains inefficient and lacks accuracy.
[0003] In recent years, with the rapid development of data-driven methods such as deep learning, they have been extensively studied and applied in the field of image processing. Compared with traditional methods, data-driven methods have significant advantages in processing speed and accuracy. Therefore, if these emerging data-driven methods can be applied, the speed and accuracy of knitted pattern recognition and retrieval should be further improved.
[0004] However, for the knitting industry, no stable and reliable research results have yet been found to enable retrieval applications based on user-input specified images. Against this backdrop, for those skilled in the art, designing and developing a data-driven method for knitted pattern recognition and retrieval that suits the knitting industry has become a pressing technical problem to be solved. Summary of the Invention
[0005] The purpose of this invention is to provide a sketch- and data-driven method, system, computer-readable storage medium, and computer program product for retrieving knitted patterns, in order to solve or at least partially solve the technical problems mentioned in the background section.
[0006] To achieve this objective, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a sketch- and data-driven method for retrieving knitted patterns, comprising:
[0008] M collectors collect sketches of each knitted pattern in the knitted pattern sample set to obtain a sketch sample set; wherein, the knitted patterns in the knitted pattern sample set are knitted patterns extracted from the knitted pattern database, M is a natural number greater than 1, and the sketch sample set includes all sketch samples collected by all collectors.
[0009] The knitted pattern sample set and the sketch sample set are used as training sets and input into a predetermined generative adversarial network for training.
[0010] Using the generator of the trained generative adversarial network, a corresponding sketch is generated for each knitted pattern in the knitted pattern database to obtain a sketch dataset;
[0011] The sketch dataset is input into a predetermined sketch recognition network for training, and an index database for retrieval is constructed.
[0012] Using the index database, the sketch to be queried is retrieved, and the most matching search result and its corresponding knitting pattern are output.
[0013] Optionally, the generative adversarial network adopts a pix2pix model or a CycleGAN model;
[0014] The generative adversarial network includes a generator for generating a sketch that mimics a human drawing based on an input knitted pattern, and a discriminator for determining whether the sketch is human-drawn or generated by the generator.
[0015] The step of using the multiple knitted pattern samples and the sketch sample set as a training set and inputting them into a predetermined generative adversarial network for training includes:
[0016] The generator's capabilities are continuously improved through adversarial training, making its generated sketches, which mimic human drawing styles, more consistent with those of human drawing.
[0017] Optionally, before generating a corresponding sketch for each knitted pattern in the knitted pattern database using the trained generative adversarial network generator, and obtaining the sketch dataset, the method further includes:
[0018] Several knitted patterns are randomly selected from the knitted pattern database and input into the generator to obtain several sketches that mimic human drawing.
[0019] Determine whether the sketch generated by the generator achieves the expected effect;
[0020] If so, the following steps are performed: using the generator of the trained generative adversarial network, a corresponding sketch is generated for each knitted pattern in the knitted pattern database to obtain a sketch dataset;
[0021] If not, increase the number of collectors and / or knitted pattern samples to further expand the number of sketch samples in the sketch sample set; and repeat the step: input the knitted pattern sample set and the sketch sample set as training sets into a predetermined generative adversarial network for training.
[0022] Optionally, determining whether the sketch generated by the generator achieves the expected effect specifically includes:
[0023] Determine whether the texture sharpness of the sketch generated by the generator meets predetermined requirements; and,
[0024] Humans judge whether the sketches generated by the generator are natural or whether the drawing style is similar to that of human drawing.
[0025] Optionally, the step of inputting the sketch dataset into a predetermined sketch recognition network for training to construct an index database for retrieval specifically includes:
[0026] Gabor filtering is performed on each sketch in the sketch dataset.
[0027] Dense sampling and feature description are performed on each sketch after Gabor filtering to obtain the GALIF local feature vectors corresponding to each sketch, forming a feature vector set.
[0028] A certain number of GALIF local feature vectors are randomly selected from the total set of feature vectors, and the selected GALIF local feature vectors are clustered using the k-means clustering algorithm to construct a visual dictionary; wherein, the visual words in the visual dictionary are the centers of each cluster.
[0029] For each sketch in the sketch dataset, extract all its GALIF local feature vectors;
[0030] Each GALIF local feature vector of each sketch is quantized into the index of the nearest visual word in the visual dictionary;
[0031] Count the frequency of each visual word in each sketch to form a word frequency histogram for each sketch;
[0032] The term frequency histograms are weighted using term frequency-inverse document frequency to obtain the global feature vector of each sketch;
[0033] The index database is constructed based on the global feature vectors of all sketches in the sketch dataset.
[0034] Optionally, the step of weighting each of the term frequency histograms using term frequency-inverse document frequency to obtain the global feature vector of each sketch is as follows:
[0035] h′ j =(h j / ∑h i )*log(N / f j );
[0036] Among them, h' j h is the global eigenvector of the sketch. j For the frequency of visual word occurrence at index j, ∑h i f is the total number of visual words in the sketch, N is the total number of sketches, and f j It is the number of sketches containing the visual word at index j.
[0037] Optionally, the step of using the index database to search for the sketch to be queried and outputting the most matching search result and its corresponding knitting pattern specifically includes:
[0038] Extract all GALIF local feature vectors of the sketch to be queried;
[0039] Each GALIF local feature vector of the sketch to be queried is quantized into the index of the nearest visual word in the visual dictionary;
[0040] The frequency of each visual word in the sketch to be queried is counted to form a word frequency histogram of the sketch to be queried;
[0041] We use term frequency-inverse document frequency to weight the term frequency histogram of the sketch to be queried, and obtain the global feature vector of the sketch to be queried.
[0042] The global feature vector of the sketch to be queried is compared with the global feature vectors of all sketches in the knitted pattern database. The sketches are then sorted in descending order of similarity.
[0043] Output the K knitting patterns that have the highest similarity to the sketch to be queried; where K is a natural number greater than or equal to 1.
[0044] Optionally, in the step of calculating the similarity between the global feature vector of the sketch to be queried and the global feature vectors of all sketches in the knitted pattern database, the similarity calculation method is as follows:
[0045] Using cosine similarity as the metric, the calculation formula is as follows:
[0046] Similarity(query,db)=(V_query·V_db) / (||V_query||*||V_db||);
[0047] Where V_query is the global feature vector of the sketch to be queried, and V_db is the global feature vector of the knitted pattern in the knitted pattern database; “·” represents the dot product or inner product, and “||||” represents the Euclidean norm or modulus of the vector; Similarity(query, db) is the cosine similarity between V_query and V_db, and the value of the cosine similarity is in the range of [-1, 1].
[0048] Optionally, K is 5.
[0049] Optionally, M is 2, and the knitted pattern sample set includes 100-300 knitted patterns.
[0050] Secondly, the present invention provides a sketch- and data-driven knitted pattern retrieval system, comprising:
[0051] A storage module is used to store a knitted pattern database, a knitted pattern sample set, and a sketch sample set; wherein, the knitted patterns in the knitted pattern sample set are knitted patterns extracted from the knitted pattern database, and the sketch sample set includes all sketch samples collected by all collectors, and the sketch samples are obtained by the collectors from sketching each knitted pattern in the knitted pattern sample set.
[0052] A data-driven module, electrically connected to the storage module, is used to train a predetermined generative adversarial network using the knitted pattern sample set and the sketch sample set as training sets. The data-driven module includes a generator, which is used to generate a corresponding sketch for each knitted pattern in the knitted pattern database to obtain a sketch dataset.
[0053] The data-driven module is also used to use the sketch dataset as a training set, a predetermined sketch recognition network, and to construct an index database for retrieval.
[0054] The retrieval module, electrically connected to the data-driven module, is used to retrieve the sketch to be queried using the index database and output the most matching retrieval result and its corresponding knitting pattern.
[0055] Optionally, the generative adversarial network adopts a pix2pix model or a CycleGAN model;
[0056] The generator is used to generate a sketch that mimics a hand-drawn sketch based on an input knitted pattern; the generative adversarial network also includes a discriminator for determining whether the sketch is hand-drawn or generated by the generator.
[0057] Training a generative adversarial network specifically includes:
[0058] The generator's capabilities are continuously improved through adversarial training, making its generated sketches, which mimic human drawing styles, more consistent with those of human drawing.
[0059] Optionally, the knitted pattern retrieval system further includes an effect verification module;
[0060] The generator is also used to randomly select several knitted patterns from the knitted pattern database based on the input, and generate several sketches that imitate human drawing.
[0061] The effect verification module is electrically connected to the data-driven module and is used to determine whether the sketches generated by the generator achieve the expected effect. If so, the data-driven module is notified to generate a sketch dataset. If not, the user is prompted that the current training effect is not up to standard, and the data-driven module is notified to retrain the generative adversarial network.
[0062] The data-driven module is also configured to further expand the number of samples in the sketch sample set and retrain the generative adversarial network after receiving a notification to retrain the generative adversarial network.
[0063] The method for expanding the number of samples in the sketch sample set is as follows:
[0064] Increase the number of collectors and / or knitted pattern samples to further expand the number of sketch samples in the sketch sample set.
[0065] Optionally, determining whether the sketch generated by the generator achieves the expected effect specifically includes:
[0066] Determine whether the texture sharpness of the sketch generated by the generator meets predetermined requirements; and,
[0067] Humans judge whether the sketches generated by the generator are natural or whether the drawing style is similar to that of human drawing.
[0068] Optionally, the step of using the sketch dataset as a training set, a predetermined sketch recognition network, and constructing an index database for retrieval specifically includes:
[0069] Gabor filtering is performed on each sketch in the sketch dataset.
[0070] Dense sampling and feature description are performed on each sketch after Gabor filtering to obtain the GALIF local feature vectors corresponding to each sketch, forming a feature vector set.
[0071] A certain number of GALIF local feature vectors are randomly selected from the total set of feature vectors, and the selected GALIF local feature vectors are clustered using the k-means clustering algorithm to construct a visual dictionary; wherein, the visual words in the visual dictionary are the centers of each cluster.
[0072] For each sketch in the sketch dataset, extract all its GALIF local feature vectors;
[0073] Each GALIF local feature vector of each sketch is quantized into the index of the nearest visual word in the visual dictionary;
[0074] Count the frequency of each visual word in each sketch to form a word frequency histogram for each sketch;
[0075] The term frequency histograms are weighted using term frequency-inverse document frequency to obtain the global feature vector of each sketch;
[0076] The index database is constructed based on the global feature vectors of all sketches in the sketch dataset.
[0077] Optionally, the step of weighting each of the term frequency histograms using term frequency-inverse document frequency to obtain the global feature vector of each sketch is as follows:
[0078] h′ j =(h j / ∑h i )*log(N / f j );
[0079] Among them, h' j h is the global eigenvector of the sketch. j For the frequency of visual word occurrence at index j, ∑h i f is the total number of visual words in the sketch, N is the total number of sketches, and f j It is the number of sketches containing the visual word at index j.
[0080] Optionally, the retrieval module is specifically used for:
[0081] Extract all GALIF local feature vectors of the sketch to be queried;
[0082] Each GALIF local feature vector of the sketch to be queried is quantized into the index of the nearest visual word in the visual dictionary;
[0083] The frequency of each visual word in the sketch to be queried is counted to form a word frequency histogram of the sketch to be queried;
[0084] We use term frequency-inverse document frequency to weight the term frequency histogram of the sketch to be queried, and obtain the global feature vector of the sketch to be queried.
[0085] The global feature vector of the sketch to be queried is compared with the global feature vectors of all sketches in the knitted pattern database. The sketches are then sorted in descending order of similarity.
[0086] Output the K knitting patterns that have the highest similarity to the sketch to be queried; where K is a natural number greater than or equal to 1.
[0087] Optionally, in the step of calculating the similarity between the global feature vector of the sketch to be queried and the global feature vectors of all sketches in the knitted pattern database, the similarity calculation method is as follows:
[0088] Using cosine similarity as the metric, the calculation formula is as follows:
[0089] Similarity(query,db)=(V_query·V_db) / (||V_query||*||V_db||);
[0090] Where V_query is the global feature vector of the sketch to be queried, and V_db is the global feature vector of the knitted pattern in the knitted pattern database; “·” represents the dot product or inner product, and “||||” represents the Euclidean norm or modulus of the vector; Similarity(query, db) is the cosine similarity between V_query and V_db, and the value of the cosine similarity is in the range of [-1, 1].
[0091] Optionally, K is 5.
[0092] Optionally, M is 2, and the knitted pattern sample set includes 100-300 knitted patterns.
[0093] Thirdly, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement a sketch- and data-driven knitted pattern retrieval method as described above.
[0094] Fourthly, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the sketch- and data-driven knitted pattern retrieval method described above.
[0095] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0096] The sketch- and data-driven knitted pattern retrieval method provided in this application can quickly and accurately convert complex knitted patterns into easily analyzable sketches through a trained generative adversarial network, solving the problem of low efficiency in manual processing. By training a sketch recognition network and then constructing an index database for retrieval, robust and efficient retrieval of sketches is achieved. It can well handle the characteristics of high repetition and lack of specific objects in knitted patterns, and improves retrieval speed and accuracy compared with traditional algorithms. Attached Figure Description
[0097] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0098] Figure 1 A flowchart illustrating a sketch- and data-driven knitted pattern retrieval method provided in this embodiment of the invention.
[0099] Figure 2 This is a schematic diagram of two knitted pattern sketches drawn by different collectors, provided for embodiments of the present invention.
[0100] Figure 3 This is a schematic diagram of a sketch automatically generated by a generator, provided as an embodiment of the present invention.
[0101] Figure 4 This is a schematic diagram of the search results output after searching a sketch to be queried, provided as an embodiment of the present invention.
[0102] Figure 5 This is a schematic diagram of the architecture of a sketch- and data-driven knitted pattern retrieval system provided in an embodiment of the present invention. Detailed Implementation
[0103] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0104] Example 1:
[0105] For details, please refer to Figure 1 , Figure 1 A flowchart of a sketch- and data-driven knitted pattern retrieval method provided in this embodiment of the invention is shown. The method specifically includes:
[0106] Preset step 110: M collectors collect sketches of each knitted pattern in the knitted pattern sample set to obtain the sketch sample set.
[0107] It should be noted that in step 110, all samples in the knitted pattern sample set are extracted from the knitted pattern database. In other words, the knitted patterns in the knitted pattern sample set are knitted patterns extracted from the knitted pattern database.
[0108] In this embodiment, M is a natural number greater than 1; for each knitted pattern in the knitted pattern sample set, each collector draws a sketch, that is, if the sum of the knitted patterns in the knitted pattern sample set is X, then the total number of sketch samples in the sketch sample set is M×X.
[0109] Step 120: Use the knitted pattern sample set and sketch sample set as training sets and input them into the predetermined generative adversarial network for training.
[0110] After preparing the sketch sample set for training, the training of the generative adversarial network can begin. In this embodiment, the generative adversarial network adopts the pix2pix model or the CycleGAN model. Since the pix2pix model or CycleGAN model mentioned above are publicly available models, their model structure and implementation principle will not be described in detail in this embodiment.
[0111] Specifically, generative adversarial networks consist of a generator (netG) and a discriminator (netD);
[0112] The generator's task is to take a knitted pattern as input and output a sketch that is as realistic as possible, mimicking the style of human drawing.
[0113] The discriminator's task is to determine whether the input sketch is a real human drawing or generated by a generator.
[0114] Therefore, the "training in the adversarial network" mentioned in this step specifically includes:
[0115] Through adversarial training, the generator's capabilities can be continuously improved, eventually enabling it to generate sketches that are highly consistent with human drawing styles.
[0116] In a preferred embodiment, step 120 is followed by:
[0117] Step 121: Randomly select several knitted patterns from the knitted pattern database, input them into the generator, and obtain several sketches that imitate human drawing.
[0118] Step 122: Determine whether the sketch generated by the generator achieves the expected effect; if yes, proceed to step 130; if no, proceed to step 123.
[0119] Step 130: Using the generator of the trained generative adversarial network, generate a corresponding sketch for each knitted pattern in the knitted pattern database to obtain the sketch dataset.
[0120] Step 123: Increase the number of collectors and / or knitted pattern samples to further expand the number of sketch samples in the sketch sample set; and repeat step 120.
[0121] Specifically, the implementation method of step 122 includes:
[0122] Determine whether the texture sharpness of the sketch generated by the generator meets the predetermined requirements; and,
[0123] Humans judge whether the sketches generated by the generator are natural or whether the drawing style is similar to that of human drawing.
[0124] Step 140: Input the sketch dataset into the predetermined sketch recognition network for training and build an index database for retrieval.
[0125] For ease of description, the knitted pattern database is defined as G1, and the sketch dataset is defined as G2;
[0126] This step mainly involves inputting the sketch dataset G2 into the SketchNet network for training, in order to learn a feature representation model that can effectively distinguish different sketches;
[0127] It should be noted that SketchNet in this step is not a simple classification network, but a feature extraction and retrieval framework based on GALIF (Gabor Local Line-based Feature). Its core function is to transform sketches into feature vectors that can be efficiently retrieved. The specific training and construction process is as follows:
[0128] 1) Local feature extraction from GALIF
[0129] (a) For each input sketch, a convolution is performed using a Gabor filter bank containing multiple directions (e.g., 4 or 8 directions). Gabor filters have a strong response to lines in specific directions and frequencies, thus effectively capturing the local texture direction of the sketch.
[0130] (b) Dense Sampling and Feature Description:
[0131] Perform dense mesh sampling on the sketch, defining a local image patch around each sampling point. Divide this local image patch into smaller cells (e.g., 4x4) and calculate the average response of each Gabor direction within each cell. Concatenate all these averages to form a high-dimensional GALIF local feature vector, which describes the texture information of this local image patch. Finally, obtain the GALIF local feature vectors corresponding to each sketch, forming a feature vector set.
[0132] 2) Constructing a visual dictionary
[0133] A large number of GALIF local feature vectors were randomly extracted from the sketch dataset G2. These vectors were then clustered using the k-means clustering algorithm, and the set of cluster centroids constituted the visual lexicon. The size of the lexicon (i.e., the k value, e.g., 2500) is an important hyperparameter that determines the discriminative power of the features. The visual words in the visual lexicon serve as the centers of each cluster.
[0134] 3) Global eigenvector representation of the sketch
[0135] For each sketch in the sketch dataset, extract all its GALIF local feature vectors;
[0136] Each GALIF local feature vector of each sketch is quantized into the index of the nearest visual word in the visual dictionary;
[0137] Count the frequency of each visual word in each sketch to form a word frequency histogram for each sketch;
[0138] The global feature vector of each sketch is obtained by weighting each term frequency histogram using term frequency-inverse document frequency.
[0139] An index database is constructed based on the global feature vectors of all sketches in the sketch dataset.
[0140] Specifically, the term frequency-inverse document frequency (IF-IVF) histograms are weighted to obtain the global feature vectors of each sketch. The specific method is as follows:
[0141] h′ j =(h j / ∑h i )*log(N / f j );
[0142] Among them, h' j h is the global eigenvector of the sketch. j For the frequency of visual word occurrence at index j, ∑h i f is the total number of visual words in the sketch, N is the total number of sketches, and f jIt is the number of sketches containing the visual word at index j.
[0143] The goal of step 140 is to obtain a stable visual dictionary and global feature vectors of all database sketches, and to build them into an efficient inverted index database for fast retrieval.
[0144] Step 150: Using the index database, retrieve the sketch to be queried, and output the most matching search results and their corresponding knitting patterns.
[0145] For ease of understanding, when a user inputs a sketch to be queried, step 150 specifically performs the following steps for retrieval and identification:
[0146] Step 151: Extract all GALIF local feature vectors of the sketch to be queried;
[0147] Step 152: Quantize each GALIF local feature vector of the sketch to be queried into the index of the nearest visual word in the visual dictionary;
[0148] Step 153: Count the frequency of each visual word in the sketch to be queried, and form a word frequency histogram of the sketch to be queried;
[0149] Step 154: Use term frequency-inverse document frequency to weight the term frequency histogram of the sketch to be queried, and obtain the global feature vector of the sketch to be queried;
[0150] Step 155: Calculate the similarity between the global feature vector of the sketch to be queried and the global feature vectors of all sketches in the knitted pattern database, and sort them in descending order of similarity.
[0151] Step 156: Output the K knitted patterns that have the highest similarity to the sketch to be queried.
[0152] In step 155, the similarity is calculated as follows:
[0153] Using cosine similarity as the metric, the calculation formula is as follows:
[0154] Similarity(query,db)=(V_query·V_db) / (||V_query||*||V_db||);
[0155] Where V_query is the global feature vector of the sketch to be queried, and V_db is the global feature vector of the knitted pattern in the knitted pattern database; "·" represents the dot product or inner product, and "||||" represents the Euclidean norm or modulus of the vector; Similarity(query, db) is the cosine similarity between V_query and V_db.
[0156] The cosine similarity ranges from [-1, 1]; where 1 indicates that the two vectors have the same direction (0 degrees between them), which is the maximum similarity; and -1 indicates that the two vectors have the opposite direction (180 degrees between them), which is the maximum dissimilarity (or maximum negative correlation).
[0157] In step 156, K is a natural number greater than or equal to 1.
[0158] For ease of understanding, the following provides a specific example of the sketch- and data-driven knitted pattern retrieval method described above:
[0159] In this example, two users collected a total of 2*150 images (sketch outline width set to 5) from the knitted pattern database G1, with serial numbers 1-150. Figure 2 As shown, it displays the results of two user sketches of two knitted patterns.
[0160] 300 collected sketch samples and 150 knitted patterns were used as training data to train the CycleGAN (or pix2pix) model, and the parameters of the discriminator and generator were obtained. The training time of this example on the GPU in the PyTorch environment is about 1.5 hours.
[0161] After training, the generator automatically generates corresponding sketches for all knitted patterns in the knitted pattern database G1, resulting in the sketch dataset G2. The generated sketches are shown below. Figure 3 As shown; this example uses GPU computation in the PyTorch environment, and the time to generate each sketch is approximately 100ms.
[0162] The sketch dataset G2 is then input into the SketchNet network for training to obtain the index database.
[0163] Ideally, the test sketch should be searched and identified to obtain the K search results with the highest probability, such as... Figure 4 As shown (here K=5); in this example, the single retrieval time for the test sketch is approximately 1ms.
[0164] In summary, the sketch- and data-driven knitted pattern retrieval method provided in this embodiment has at least the following effects:
[0165] Using cycleGAN (or pix2pix), complex knitted patterns can be automatically converted into easily analyzable sketches, solving the problem of low efficiency in manual processing;
[0166] The GALIF feature was introduced and specified. It is designed specifically for lines and textures. Combined with the visual bag-of-words model, it achieves robust and efficient retrieval of sketches. It can handle the high repetition and lack of specific objects in knitted patterns very well. Compared with traditional algorithms, it improves retrieval speed and accuracy.
[0167] It provides a paradigm for extracting features from complex patterns through sketches, which facilitates rapid retrieval.
[0168] Example 2:
[0169] Please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the architecture of a sketch- and data-driven knitted pattern retrieval system provided in an embodiment of the present invention. The system specifically includes:
[0170] The storage module 10 is used to store a knitted pattern database, a knitted pattern sample set, and a sketch sample set. The knitted patterns in the knitted pattern sample set are knitted patterns extracted from the knitted pattern database. The sketch sample set includes all sketch samples collected by all collectors. The sketch samples are obtained by the collectors from sketching each knitted pattern in the knitted pattern sample set.
[0171] The data-driven module 20 and the electrically connected storage module 10 are used to train a predetermined generative adversarial network using the knitted pattern sample set and the sketch sample set as training sets. The data-driven module 20 includes a generator used to generate a corresponding sketch for each knitted pattern in the knitted pattern database to obtain a sketch dataset.
[0172] The data-driven module 20 is also used to use the sketch dataset as a training set, a pre-defined sketch recognition network, and to build an index database for retrieval.
[0173] The retrieval module 40 and the electrically connected data drive module 20 are used to retrieve the sketch to be queried using the index database and output the most matching retrieval result and its corresponding knitting pattern.
[0174] Specifically, generative adversarial networks employ either the pix2pix model or the CycleGAN model;
[0175] The generator is used to generate a sketch that mimics a human drawing based on the input knitted pattern; the generative adversarial network also includes a discriminator to determine whether the sketch is human-drawn or generated by the generator.
[0176] Training a generative adversarial network specifically includes:
[0177] By continuously improving the generator's capabilities through adversarial training, the generated sketches that mimic human drawing tend to be consistent with the style of human drawing.
[0178] Specifically, the knitted pattern retrieval system also includes an effect verification module 30;
[0179] The generator is also used to randomly select several knitted patterns from the knitted pattern database based on the input and generate several sketches that mimic human drawing.
[0180] The effect verification module 30 is electrically connected to the data-driven module 20 and is used to determine whether the sketch generated by the generator achieves the expected effect. If yes, it notifies the data-driven module 20 to generate the sketch dataset. If no, it prompts the user that the current training effect is not up to standard and notifies the data-driven module 20 to retrain the generative adversarial network.
[0181] The data-driven module 20 is also used to further expand the number of samples in the sketch sample set and retrain the generative adversarial network after receiving a notification to retrain the generative adversarial network.
[0182] The method for expanding the number of samples in the sketch sample set is as follows:
[0183] Increase the number of collectors and / or knitted pattern samples to further expand the number of sketch samples in the sketch sample set.
[0184] Specifically, determining whether the sketch generated by the generator achieves the expected result includes:
[0185] Determine whether the texture sharpness of the sketch generated by the generator meets the predetermined requirements; and,
[0186] Humans judge whether the sketches generated by the generator are natural or whether the drawing style is similar to that of human drawing.
[0187] Specifically, the sketch dataset is used as the training set, a pre-defined sketch recognition network is used, and an index database for retrieval is constructed, including:
[0188] Gabor filtering is applied to the sketches in each sketch dataset;
[0189] Dense sampling and feature description are performed on each sketch after Gabor filtering to obtain the GALIF local feature vectors corresponding to each sketch, forming a feature vector set.
[0190] A certain number of GALIF local feature vectors are randomly selected from the total feature vector set. The k-means clustering algorithm is used to cluster the selected GALIF local feature vectors to construct a visual dictionary. The visual words in the visual dictionary are the centers of each cluster.
[0191] For each sketch in the sketch dataset, extract all its GALIF local feature vectors;
[0192] Each GALIF local feature vector of each sketch is quantized into the index of the nearest visual word in the visual dictionary;
[0193] Count the frequency of each visual word in each sketch to form a word frequency histogram for each sketch;
[0194] The global feature vector of each sketch is obtained by weighting each term frequency histogram using term frequency-inverse document frequency.
[0195] An index database is constructed based on the global feature vectors of all sketches in the sketch dataset.
[0196] Specifically, the term frequency-inverse document frequency (IF-Document Frequency) method is used to weight each term frequency histogram to obtain the global feature vector of each sketch. The specific method is as follows:
[0197] h′ j =(h j / ∑h i )*log(N / f j );
[0198] Among them, h' j h is the global eigenvector of the sketch. j For the frequency of visual word occurrence at index j, ∑h i f is the total number of visual words in the sketch, N is the total number of sketches, and f j It is the number of sketches containing the visual word at index j.
[0199] Specifically, the retrieval module 40 is used for:
[0200] Extract all GALIF local feature vectors of the sketch to be queried;
[0201] The local feature vector of each GALIF feature vector of the sketch to be queried is to be quantized into the index of the nearest visual word in the visual dictionary;
[0202] The frequency of each visual word in the sketch to be queried is counted to form a word frequency histogram of the sketch to be queried;
[0203] We use term frequency-inverse document frequency to weight the term frequency histogram of the sketch to be queried, and obtain the global feature vector of the sketch to be queried.
[0204] The global feature vector of the sketch to be queried is compared with the global feature vectors of all sketches in the knitted pattern database. The sketches are then sorted in descending order of similarity.
[0205] Output the K knitting patterns that have the highest similarity to the sketch to be queried; where K is a natural number greater than or equal to 1.
[0206] Specifically, in calculating the similarity between the global feature vector of the sketch to be queried and the global feature vectors of all sketches in the knitted pattern database, the similarity calculation method is as follows:
[0207] Using cosine similarity as the metric, the calculation formula is as follows:
[0208] Similarity(query,db)=(V_query·V_db) / (||V_query||*||V_db||);
[0209] Where V_query is the global feature vector of the sketch to be queried, and V_db is the global feature vector of the knitted pattern in the knitted pattern database; “·” represents the dot product or inner product, and “|||||” represents the Euclidean norm or modulus of the vector; Similarity(query, db) is the cosine similarity between V_query and V_db, and the value of the cosine similarity ranges from [-1, 1].
[0210] In this embodiment, the knitted pattern sample set includes 100-300 knitted patterns;
[0211] For example, M is 2. Using this knitted pattern retrieval system, the test sketch is retrieved and identified, yielding the K highest probability retrieval results, such as... Figure 4 As shown (here K=5).
[0212] The sketch- and data-driven knitted pattern retrieval system provided in this application can quickly and accurately convert complex knitted patterns into easily analyzable sketches, solving the problem of low efficiency in manual processing. It constructs an index database for retrieval, achieving robust and efficient retrieval of sketches, and can well handle the characteristics of high repetition and lack of specific objects in knitted patterns. Compared with traditional algorithms, it improves retrieval speed and accuracy.
[0213] Example 3:
[0214] This embodiment also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement a sketch- and data-driven knitted pattern retrieval method as described in Embodiment 1.
[0215] Since the sketch- and data-driven knitted pattern retrieval method has been described in detail in Example 1, it will not be repeated in this example.
[0216] Example 4:
[0217] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements a sketch- and data-driven knitted pattern retrieval method as described in Embodiment 1.
[0218] Since the sketch- and data-driven knitted pattern retrieval method has been described in detail in Example 1, it will not be repeated in this example.
[0219] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0220] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sketch- and data-driven method for retrieving knitted patterns, characterized in that, include: M collectors collect sketches of each knitted pattern in the knitted pattern sample set to obtain a sketch sample set; wherein, the knitted patterns in the knitted pattern sample set are knitted patterns extracted from the knitted pattern database, M is a natural number greater than 1, and the sketch sample set includes all sketch samples collected by all collectors. The knitted pattern sample set and the sketch sample set are used as training sets and input into a predetermined generative adversarial network for training. Using the generator of the trained generative adversarial network, a corresponding sketch is generated for each knitted pattern in the knitted pattern database to obtain a sketch dataset; The sketch dataset is input into a predetermined sketch recognition network for training, and an index database for retrieval is constructed. Using the index database, the sketch to be queried is retrieved, and the most matching search result and its corresponding knitting pattern are output.
2. The sketch- and data-driven knitted pattern retrieval method according to claim 1, characterized in that, The generative adversarial network adopts either the pix2pix model or the CycleGAN model. The generative adversarial network includes a generator for generating a sketch that mimics a human drawing based on an input knitted pattern, and a discriminator for determining whether the sketch is human-drawn or generated by the generator. The step of using the multiple knitted pattern samples and the sketch sample set as a training set and inputting them into a predetermined generative adversarial network for training includes: The generator's capabilities are continuously improved through adversarial training, making its generated sketches, which mimic human drawing styles, more consistent with those of human drawing.
3. The sketch- and data-driven knitted pattern retrieval method according to claim 2, characterized in that, Before obtaining the sketch dataset, the generator, which utilizes a trained generative adversarial network, generates a corresponding sketch for each knitted pattern in the knitted pattern database. Several knitted patterns are randomly selected from the knitted pattern database and input into the generator to obtain several sketches that mimic human drawing. Determine whether the sketch generated by the generator achieves the expected effect; If so, the following steps are performed: using the generator of the trained generative adversarial network, a corresponding sketch is generated for each knitted pattern in the knitted pattern database to obtain a sketch dataset; If not, increase the number of collectors and / or knitted pattern samples to further expand the number of sketch samples in the sketch sample set; and repeat the step: input the knitted pattern sample set and the sketch sample set as training sets into a predetermined generative adversarial network for training.
4. The sketch- and data-driven knitted pattern retrieval method according to claim 3, characterized in that, The determination of whether the sketch generated by the generator achieves the expected effect specifically includes: Determine whether the texture sharpness of the sketch generated by the generator meets predetermined requirements; and, Humans judge whether the sketches generated by the generator are natural or whether the drawing style is similar to that of human drawing.
5. The sketch- and data-driven knitted pattern retrieval method according to claim 1, characterized in that, The step of inputting the sketch dataset into a predetermined sketch recognition network for training and constructing an index database for retrieval specifically includes: Gabor filtering is performed on each sketch in the sketch dataset. Dense sampling and feature description are performed on each sketch after Gabor filtering to obtain the GALIF local feature vectors corresponding to each sketch, forming a feature vector set. A certain number of GALIF local feature vectors are randomly selected from the total set of feature vectors, and the selected GALIF local feature vectors are clustered using the k-means clustering algorithm to construct a visual dictionary; wherein, the visual words in the visual dictionary are the centers of each cluster. For each sketch in the sketch dataset, extract all its GALIF local feature vectors; Each GALIF local feature vector of each sketch is quantized into the index of the nearest visual word in the visual dictionary; Count the frequency of each visual word in each sketch to form a word frequency histogram for each sketch; The term frequency histograms are weighted using term frequency-inverse document frequency to obtain the global feature vector of each sketch; The index database is constructed based on the global feature vectors of all sketches in the sketch dataset.
6. The sketch- and data-driven knitted pattern retrieval method according to claim 5, characterized in that, The method of weighting each term frequency histogram using term frequency-inverse document frequency to obtain the global feature vector of each sketch is as follows: h′ j =(h j / ∑h i )*log(N / f j ); Among them, h' j h is the global eigenvector of the sketch. j For the frequency of visual word occurrence at index j, ∑h i f is the total number of visual words in the sketch, N is the total number of sketches, and f j It is the number of sketches containing the visual word at index j.
7. The sketch- and data-driven knitted pattern retrieval method according to claim 5, characterized in that, The process of using the index database to search for the sketch to be queried and outputting the most matching search result and its corresponding knitting pattern specifically includes: Extract all GALIF local feature vectors of the sketch to be queried; Each GALIF local feature vector of the sketch to be queried is quantized into the index of the nearest visual word in the visual dictionary; The frequency of each visual word in the sketch to be queried is counted to form a word frequency histogram of the sketch to be queried; We use term frequency-inverse document frequency to weight the term frequency histogram of the sketch to be queried, and obtain the global feature vector of the sketch to be queried. The global feature vector of the sketch to be queried is compared with the global feature vectors of all sketches in the knitted pattern database. The sketches are then sorted in descending order of similarity. Output the K knitting patterns that have the highest similarity to the sketch to be queried; where K is a natural number greater than or equal to 1.
8. The sketch- and data-driven knitted pattern retrieval method according to claim 7, characterized in that, In the step of calculating the similarity between the global feature vector of the sketch to be queried and the global feature vectors of all sketches in the knitted pattern database, the similarity calculation method is as follows: Using cosine similarity as the metric, the calculation formula is as follows: Similarity(query,db)=(V_query·V_db) / (||V_query||*||V_db||); Where V_query is the global feature vector of the sketch to be queried, and V_db is the global feature vector of the knitted pattern in the knitted pattern database; "·" represents the dot product or inner product, and "||||" represents the Euclidean norm or modulus of the vector; Similarity(query, db) is the cosine similarity between V_query and V_db, and the value range of the cosine similarity is [-1, 1].
9. A sketch- and data-driven knitted pattern retrieval method according to claim 7, characterized in that, K is 5.
10. A sketch- and data-driven knitted pattern retrieval method according to claim 1, characterized in that, M is 2, and the knitted pattern sample set includes 100-300 knitted patterns.
11. A sketch- and data-driven knitted pattern retrieval system, characterized in that, include: A storage module is used to store a knitted pattern database, a knitted pattern sample set, and a sketch sample set; wherein, the knitted patterns in the knitted pattern sample set are knitted patterns extracted from the knitted pattern database, and the sketch sample set includes all sketch samples collected by all collectors, and the sketch samples are obtained by the collectors from sketching each knitted pattern in the knitted pattern sample set. A data-driven module, electrically connected to the storage module, is used to train a predetermined generative adversarial network using the knitted pattern sample set and the sketch sample set as training sets. The data-driven module includes a generator, which is used to generate a corresponding sketch for each knitted pattern in the knitted pattern database to obtain a sketch dataset. The data-driven module is also used to use the sketch dataset as a training set, a predetermined sketch recognition network, and to construct an index database for retrieval. The retrieval module, electrically connected to the data-driven module, is used to retrieve the sketch to be queried using the index database and output the most matching retrieval result and its corresponding knitting pattern.
12. A sketch- and data-driven knitted pattern retrieval system according to claim 11, characterized in that, The generative adversarial network adopts either the pix2pix model or the CycleGAN model. The generator is used to generate a sketch that mimics a hand-drawn sketch based on an input knitted pattern; the generative adversarial network also includes a discriminator for determining whether the sketch is hand-drawn or generated by the generator. Training a generative adversarial network specifically includes: The generator's capabilities are continuously improved through adversarial training, making its generated sketches, which mimic human drawing styles, more consistent with those of human drawing.
13. A sketch- and data-driven knitted pattern retrieval system according to claim 12, characterized in that, It also includes an effect verification module; The generator is also used to randomly select several knitted patterns from the knitted pattern database based on the input, and generate several sketches that imitate human drawing. The effect verification module is electrically connected to the data-driven module and is used to determine whether the sketches generated by the generator achieve the expected effect. If so, the data-driven module is notified to generate a sketch dataset. If not, the user is prompted that the current training effect is not up to standard, and the data-driven module is notified to retrain the generative adversarial network. The data-driven module is also configured to further expand the number of samples in the sketch sample set and retrain the generative adversarial network after receiving a notification to retrain the generative adversarial network. The method for expanding the number of samples in the sketch sample set is as follows: Increase the number of collectors and / or knitted pattern samples to further expand the number of sketch samples in the sketch sample set.
14. A sketch- and data-driven knitted pattern retrieval system according to claim 13, characterized in that, The determination of whether the sketch generated by the generator achieves the expected effect specifically includes: Determine whether the texture sharpness of the sketch generated by the generator meets predetermined requirements; and, Humans judge whether the sketches generated by the generator are natural or whether the drawing style is similar to that of human drawing.
15. A sketch- and data-driven knitted pattern retrieval system according to claim 11, characterized in that, The step of using the sketch dataset as a training set, a predetermined sketch recognition network, and constructing an index database for retrieval specifically includes: Gabor filtering is performed on each sketch in the sketch dataset. Dense sampling and feature description are performed on each sketch after Gabor filtering to obtain the GALIF local feature vectors corresponding to each sketch, forming a feature vector set. A certain number of GALIF local feature vectors are randomly selected from the total set of feature vectors, and the selected GALIF local feature vectors are clustered using the k-means clustering algorithm to construct a visual dictionary; wherein, the visual words in the visual dictionary are the centers of each cluster. For each sketch in the sketch dataset, extract all its GALIF local feature vectors; Each GALIF local feature vector of each sketch is quantized into the index of the nearest visual word in the visual dictionary; Count the frequency of each visual word in each sketch to form a word frequency histogram for each sketch; The term frequency histograms are weighted using term frequency-inverse document frequency to obtain the global feature vector of each sketch; The index database is constructed based on the global feature vectors of all sketches in the sketch dataset.
16. A sketch- and data-driven knitted pattern retrieval system according to claim 15, characterized in that, The method of weighting each term frequency histogram using term frequency-inverse document frequency to obtain the global feature vector of each sketch is as follows: h′ j =(h j / ∑h i )*log(N / f j ); Among them, h' j h is the global eigenvector of the sketch. j For the frequency of visual word occurrence at index j, ∑h i f is the total number of visual words in the sketch, N is the total number of sketches, and f j It is the number of sketches containing the visual word at index j.
17. A sketch- and data-driven knitted pattern retrieval system according to claim 15, characterized in that, The retrieval module is specifically used for: Extract all GALIF local feature vectors of the sketch to be queried; Each GALIF local feature vector of the sketch to be queried is quantized into the index of the nearest visual word in the visual dictionary; The frequency of each visual word in the sketch to be queried is counted to form a word frequency histogram of the sketch to be queried; We use term frequency-inverse document frequency to weight the term frequency histogram of the sketch to be queried, and obtain the global feature vector of the sketch to be queried. The global feature vector of the sketch to be queried is compared with the global feature vectors of all sketches in the knitted pattern database. The sketches are then sorted in descending order of similarity. Output the K knitting patterns that have the highest similarity to the sketch to be queried; where K is a natural number greater than or equal to 1.
18. A sketch- and data-driven knitted pattern retrieval system according to claim 17, characterized in that, In the step of calculating the similarity between the global feature vector of the sketch to be queried and the global feature vectors of all sketches in the knitted pattern database, the similarity calculation method is as follows: Using cosine similarity as the metric, the calculation formula is as follows: Similarity(query,db)=(V_query·V_db) / (||V_query||*||V_db||); Where V_query is the global feature vector of the sketch to be queried, and V_db is the global feature vector of the knitted pattern in the knitted pattern database; "·" represents the dot product or inner product, and "||||" represents the Euclidean norm or modulus of the vector; Similarity(query, db) is the cosine similarity between V_query and V_db, and the value range of the cosine similarity is [-1, 1].
19. A sketch- and data-driven knitted pattern retrieval system according to claim 17, characterized in that, K is 5.
20. A sketch- and data-driven knitted pattern retrieval system according to claim 11, characterized in that, M is 2, and the knitted pattern sample set includes 100-300 knitted patterns.
21. A computer-readable storage medium storing at least one instruction, characterized in that, The instructions are loaded and executed by the processor to implement a sketch- and data-driven knitted pattern retrieval method as described in any one of claims 1-10.
22. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, a sketch- and data-driven knitted pattern retrieval method according to any one of claims 1-10 is implemented.