A color film matching method based on image classification retrieval and related device

Abstract

A color film matching method based on image classification retrieval and related device, comprising: collecting color film images, performing image data enhancement and category division, and constructing a color film data set; dividing the color film data set into a training set and a test set, training using an improved VGG16 network to obtain a feature extraction network; extracting features of the color film images, performing feature enhancement and feature fusion when extracting the features, and establishing a color film feature library; performing image classification on a query image and extracting features of the query image; performing similarity measurement on the query image and the color film feature library to obtain the top k images with the highest similarity; and performing secondary retrieval on the k retrieved images to obtain the final retrieval result. The application can quickly find similar images and their numbers of the query image in the image library, providing convenience for commercial activities.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image classification and image retrieval, and in particular to a color film matching method based on image classification retrieval and related devices. BACKGROUND

[0002] Color film technology is a printing technology that prints a given pattern on a floor tile. In international trade, it is too inefficient for foreign businessmen to find the style of the intended goods from tens of thousands of floor tiles, which seriously restricts the sales of Chinese floor tiles abroad. It is of great practical value to automatically and quickly search for matching results from a large number of sample libraries using deep learning image retrieval technology.

[0003] The existing color film retrieval has the disadvantages of slow retrieval speed and low accuracy. When using the image retrieval interfaces of Baidu and Ali for color film matching query, the images obtained are often quite different from the actual images, because the built-in algorithms of Baidu and Ali are for feature extraction and matching of ordinary images, and are not adapted and fine-tuned for color film images. SUMMARY

[0004] The present application relates to the field of image classification and image retrieval, and in particular to a color film matching method based on image classification retrieval and related devices.

[0005] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0006] A color film matching method based on image classification retrieval, comprising:

[0007] Collecting color film images, performing image data enhancement and class division, and constructing a color film data set;

[0008] Dividing the color film data set into a training set and a test set, training using an improved VGG16 network, and obtaining a feature extraction network;

[0009] Extracting features of the color film images, performing feature enhancement and feature fusion when extracting the features, and establishing a color film feature library;

[0010] Performing image classification on the query image and extracting features of the query image;

[0011] Measuring the similarity between the query image and the color film feature library to obtain the top k images with the highest similarity;

[0012] Performing secondary retrieval on the k images retrieved to obtain the final retrieval result.

[0013] Further, the color film image is a high-resolution image simulating the color and texture of wood, rock and cloth, and each image has a unique number; the sliding window method is used to slide the window on the original color film image for cutting, and the data set is expanded by rotation and scaling data enhancement method, and the color film data is divided into n categories.

[0014] Further, the color film data set is divided into training set and test set, and input into the improved VGG16 deep learning network for training:

[0015] VGG16 has a total of 16 layers, including 13 convolutional layers and 3 fully connected layers, of which 13 convolutional layers and their corresponding pooling layers are used as feature extraction network, and the last 3 fully connected layers are responsible for completing the classification task; the feature extraction network of VGG16 is divided into 5 blocks according to the position of 5 pooling layers, each block includes two-dimensional convolutional layer, ReLU activation function layer, CBAM mixed attention layer and maximum pooling layer; the improved VGG16 adds a mixed domain attention layer CBAM in each Block, and the output dimensions of the three fully connected layers in VGG16 are replaced by 512, 512 and 6 dimensions, respectively, and a Dropout layer is added between the first and second fully connected layers.

[0016] Further, load ImageNet pre-training weights in the process of training the improved VGG16 network, and adjust the network parameters in the classification network through fine-tuning method; first, freeze the training of convolutional layers and CBAM layers, and train the fully connected layers used for classification with a larger learning rate; when the accuracy of the validation set no longer improves, train the convolutional layers, CBAM layers and fully connected layers with a smaller learning rate.

[0017] Further, call the feature extraction network, extract the color film features using the convolutional layers in the network, and enhance the feature weights of the texture and pattern regions on the feature map through channel weighting and spatial weighting operations:

[0018] Select the output features of Block3, Block4 and Block5 layers of the improved VGG16 network for fusion, splice the 1x256, 1x512 and 1x512 feature vectors output by the three Blocks into a 1x1280 feature vector, and then normalize each component of the feature vector by dividing it by the modulus of the feature vector; the feature vector of the image and the number of the image are indexed one by one to generate a feature code library, and n feature libraries are established for n color film categories.

[0019] Further, classify the query image:

[0020] The query image refers to a picture that a user uploads and that a system needs to search, is an actual effect picture of a color film image shot by the user and printed on a floor tile; an input query image is called a trained classification network to predict the category of the query image; a feature extraction network and a feature weighting enhancement and feature fusion method are used to obtain a 1x1280-dimensional feature vector of the query image; images in an i-th feature library are sorted according to the order from large to small similarity, and the first k images with the highest similarity are output as the preliminary search result.

[0021] Further, the feature vectors of the k images in the preliminary search are averaged with the feature vector of the query image to generate a new 1280-dimensional feature vector; the new feature vector is used to perform similarity measurement with the feature vectors in the i-th feature library, and the first k images with the highest similarity are output as the final search result after the secondary search.

[0022] Further, a color film matching system based on image classification search includes:

[0023] A collection module is configured to collect color film images, perform image data enhancement and category division, and construct a color film data set;

[0024] A feature extraction module is configured to divide the color film data set into a training set and a test set, and train the color film data set by using an improved VGG16 network to obtain a feature extraction network;

[0025] A color film feature library establishment module is configured to extract features of the color film images, perform feature enhancement and feature fusion when extracting the features, and establish a color film feature library;

[0026] An image feature extraction module is configured to perform image classification on a query image and extract features of the query image;

[0027] An output module is configured to perform similarity measurement on the query image and the color film feature library to obtain the first k images with the highest similarity, and perform secondary search on the k images searched to obtain a final search result.

[0028] Further, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of a color film matching method based on image classification search when executing the computer program.

[0029] Further, a computer readable storage medium stores a computer program, and the computer program implements the steps of a color film matching method based on image classification search when executed by a processor.

[0030] Compared with the prior art, the present application has the following technical effects:

[0031] The application discloses a color film matching method based on classification retrieval, and provides a feasible and efficient scheme for color film image retrieval. First, a color film data set is constructed through a data enhancement method, then a color film classification model and a feature extractor are trained by using an improved VGG16 network, and a feature vector of an image is effectively extracted by using dense connection and CBAM attention mechanism in the network. The improved network has the advantages of small network parameters and high network training efficiency. On the other hand, because the color film types are diversified in reality, the network is fine-tuned on the initial weights of ImageNet for training, so that cross-domain recognition classification can be realized, and the accuracy and specificity of feature extraction are improved. In addition, feature fusion and feature enhancement are performed during feature extraction. Global features and local features are fused, and features are weighted by using an attention mechanism, so that the accuracy of retrieval is improved. The application combines image classification and image retrieval, and finally obtains a classification retrieval model with high efficiency and high precision. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flowchart of the color film matching method of the classification retrieval of the application is shown.

[0033] Figure 2 A specific step diagram of the color film matching method of the classification retrieval of the application is shown.

[0034] Figure 3 A CBAM block network diagram is shown.

[0035] Figure 4 An improved VGG16 network structure is shown.

[0036] Figure 5 A color film matching result display diagram of the application is shown. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0038] In the description of the application, it should be understood that the terms "include" and "contain" indicate the existence of described features, whole, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0039] It should also be understood that the terms used in the specification and the following claims are for the purpose of describing particular embodiments and are not intended to be limiting, as the specific scope of the invention is disclosed in the appended claims. As used in this specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise.

[0040] It should also be further understood that the term "and / or" used in the specification and the following claims, means one or more of the associated listed items as well as all possible combinations of the items, and includes the combination of these items, for example, A and / or B, can mean the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents the relationship between the front and rear associated objects as "or".

[0041] It should be understood that, although the terms first, second, third, etc. can be used in embodiments of the present application to describe a certain range, etc., these ranges should not be limited to these terms. These terms are only used to distinguish the ranges from each other. For example, the first range can also be referred to as the second range, and similarly, the second range can also be referred to as the first range, without departing from the scope of the embodiments of the present application.

[0042] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting". Similarly, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0043] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These drawings are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and certain details can be omitted. The shapes of various regions, layers and their relative sizes and positional relationships shown in the drawings are only exemplary, and in actuality, there can be deviations due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes and relative positions according to actual needs.

[0044] The present application will be further described in detail with reference to specific embodiments, which are explanatory rather than limiting of the present application.

[0045] The present application focuses on the feature extraction of the color film floor tile image, uses the improved VGG16 to learn the image features of the color film, improves the speed of retrieval through the hierarchical processing steps of classification and retrieval first, and makes it meet the requirements of real-time retrieval. The standard color film image is collected, the original image is cropped to expand the data set, the image is artificially classified, the training set and the test set are divided, the improved VGG16 network is used to train the color film classification model, the trained classification model is used as a feature extractor, feature enhancement and feature fusion are performed when extracting features, and a color film feature library is established; the query image is classified, classified into a subclass, and the feature of the query image is extracted; the features of the query image and the color film feature library of the subclass are similarity measured and secondarily retrieved, and the k images with the highest similarity after secondary retrieval are the classification retrieval results of the query image. The present application can quickly find the similar images and their numbers of the query image in the image library, and provides convenience for commercial activities.

[0046] The present application provides a color film matching method based on classification retrieval, mainly including offline feature library construction and online query. The offline feature library construction part firstly collects color film images and artificially classifies and screens the images to establish a standard database of color film images; divides the training set and the test set, uses the improved VGG16 network to train the color film classification model; the trained classification model is used as a feature extractor, feature enhancement and feature fusion are performed when extracting features; the features of the color film images are extracted, and a color film feature library is established. The composition of the two parts can be seen in the accompanying Figure 2 .

[0047] The online query part is that the user inputs an image, the system pre-processes and classifies the image, and classifies it into a subclass of color film images; then the feature vector of the image is extracted, the features of the query image and the color film feature library of the subclass are similarity measured and secondarily retrieved; the k images with the highest similarity after secondary retrieval are the classification retrieval results of the query image.

[0048] Embodiment:

[0049] Referring to Figure 1 A color film matching method based on classification retrieval, comprising the following steps:

[0050] S1, 3000 standard color film images are collected, the size of the image is about 4800x4800 pixels, small block images with original size of 1 / 2, 1 / 4, 1 / 8 are cut from the original image through sliding window, 16 sub-images are cut from each image, forming 48000 color film image data sets. From the actual demand and the existing division standard, the color film images are divided into six categories: wood grain, stone grain, cloth grain, spliced grain, strip grain and other artistic patterns. For images with labels, they can be directly divided into a certain subclass, and for unlabeled color film images, they are classified according to the pattern and color of the color film. The proportion of images with labels is about 70% of the total.

[0051] S2, divide the training set and the test set, train using the improved VGG16 network, get the feature extraction network and the classification network; the training set and the test set are randomly divided according to the ratio of 8:2, the number of images in the training set is 38400, and the number of images in the test set is 9600. Load the training set into the improved VGG16 network for training, and uniformly scale to 224x224 pixel image size when loading. And perform data enhancement operations, including random rotation, random horizontal or vertical flip, and random scale. Data enhancement operation helps to improve the robustness of the model.

[0052] The improved VGG16 introduces an attention mechanism, which can improve the attention of the classification network to details such as texture and shape, which is beneficial to the fine-grained classification of color film images. The attention mechanism generates a mask in the neural network model, which describes the feature weight of different positions. The mask map and the feature map are multiplied to obtain the feature after adding the attention mechanism.

[0053] According to the direction of attention weighting, it can be divided into three categories: spatial attention mechanism (Spatial Attention Module, SAM), channel attention mechanism (Channel Attention Module, CAM) and mixed domain attention mechanism (Convolutional Block Attention Module, CBAM). For a feature map with size HxWxC, SAM is weighted in the HxW direction of the feature map, CAM is weighted in the channel direction, and CBAM is a module formed by combining CAM and SAM, which is weighted in the channel and HxW direction. The network structure of CBAM block is shown in the following figure: Figure 3 .

[0054] The specific process of the channel attention module CAM is as follows:

[0055] First, the input HxWxC dimensional feature map F inGlobal max pooling and global average pooling are performed in the direction of HxW respectively to obtain two 1x1xC feature maps.

[0056] Then, they are respectively sent into a two-layer multi-layer perception, the first layer has C / r (r is the reduction rate) neurons, and the second layer has C neurons. ReLU activation function is used, and the two layers of neural networks are shared weights.

[0057] Then, the features output by the multi-layer perception are summed and activated by sigmoid to generate the final 1x1xC channel attention feature (CAF) map.

[0058] Finally, the CAF and the input feature map F in are multiplied to generate the channel-weighted feature map F out .

[0059] The specific process of the spatial attention module SAM is as follows:

[0060] First, the feature map F out output by the CAM module (HxWxC) is taken as the input feature map of the module. Global max pooling and global average pooling are performed in the channel direction to obtain two HxWx1 feature maps.

[0061] Then, the two feature maps are spliced in the channel direction to obtain an HxWx2 feature map. After a 7x7 convolution operation, the channel number is reduced to 1, i.e., an HxWx1 feature map is obtained.

[0062] After sigmoid activation function, an HxWx1 channel attention feature map (Spatial Attention Feature, SAF) is generated.

[0063] Finally, the SAF and the input feature map F out of the module are multiplied to obtain the final feature map.

[0064] The improved VGG16 network includes a feature extraction module and a classification module, and the network structure is shown in FIG. 1. Figure 4 The feature extraction module is composed of five Blocks, and the classifier is realized by three fully connected layers and a Softmax function. The network structure is as follows:

[0065] a convolution layer with filter 3x3x3x64, a convolution layer with filter 3x3x64x64, a CBAM attention layer with 64 channels, and a global max pooling layer with a stride of 2;

[0066] a convolutional layer with filter 3x3x64x128, a convolutional layer with filter 3x3x128x128, a CBAM attention layer with 128 channels, a global max pooling layer with a stride of 2;

[0067] a convolutional layer with filter 3x3x128x256, a convolutional layer with filter 3x3x256x256, a CBAM attention layer with 256 channels, a global max pooling layer with a stride of 2;

[0068] a convolutional layer with filter 3x3x256x512, a convolutional layer with filter 3x3x512x512, a convolutional layer with filter 3x3x512x512, a CBAM attention layer with 512 channels, a global max pooling layer with a stride of 2;

[0069] a convolutional layer with filter 3x3x512x512, a convolutional layer with filter 3x3x512x512, a convolutional layer with filter 3x3x512x512, a CBAM attention layer with 512 channels, a global max pooling layer with a stride of 2;

[0070] a fully connected layer with 25088x256, a Dropout layer, a fully connected layer with 256x256, a fully connected layer with 256x6.

[0071] The CBAM attention layer uses Sigmod as the activation function, and the other layers use ReLU as the activation function.

[0072] The general VGG16 has a total of 16 layers, including 13 convolutional layers and 3 fully connected layers. According to the position of the pooling layer, the VGG16 can be divided into 5 blocks, and the position of each block is shown in FIG. 1. Figure 4 The improved VGG16 adds a mixed domain attention layer CBAM after each block to strengthen the attention to texture and improve the accuracy of fine-grained classification. The output dimensions of the three fully connected layers in the VGG16 are 4096, 4096 and 1000, which are replaced by 512, 512 and 6 dimensions in the present application, which greatly reduces the parameter amount of the network and improves the training speed. And a Dropout layer is added.

[0073] The improved VGG16 deep learning network is adopted, the initial learning rate is 0.001, the optimizer is RMSprop, the initial weight is ImageNet weight, and the loss function is cross-entropy loss. The convolutional layer region is mainly responsible for feature extraction, and the fully connected layer classifies the extracted features. Therefore, the network can be regarded as a feature extraction network plus a classification network. The training is carried out by using the transfer learning model fine-tuning method.

[0074] First, freeze all the convolutional layers, only update the weights of the fully connected layers. The initial learning rate is 0.001, set the adaptive learning rate decay strategy, such as after 10 Epoch training, the accuracy of the validation set does not improve, the learning rate will be reduced to 0.5 times the original. Train for 200 Epochs, keep the model with the highest accuracy on the validation set.

[0075] Then, unfreeze the 3 convolutional layer blocks close to the fully connected layer, train the parameters of Block3, Block4, Block5 and the fully connected layer based on the model saved in the last step, set the initial learning rate to 0.0001, train for 200 Epochs, and get the best fine-tuned model. After repeatedly adjusting the learning rate, batchsize and other parameters for training, the final accuracy on the validation set can reach 0.96.

[0076] S3, using the improved feature extraction network to extract the features of the color film image, and performing feature enhancement and feature fusion when extracting the features, and establishing a color film feature library;

[0077] S301, using the feature extraction network trained in the S2 step to extract image features. The feature extraction is a process of inputting an image, performing convolution, activation and pooling, and outputting an n x 1-dimensional feature vector. The Crow pooling method converts the H x W x C feature map output by the convolutional layer into a 1 x C-dimensional feature vector. The Crow pooling method increases the weight of the target region, and reduces the weight of the non-target region through a weight adaptive method. This method aggregates feature maps by constructing spatial weights and channel weights, which can increase the weight of the region of interest and suppress the weight of the non-target region to a certain extent.

[0078] For a feature map, the calculation formula of the spatial weight in the Crow method is:

[0079]

[0080] Where S ij is the spatial weight, S ′ is the sum of all feature values arranged by channel corresponding to the corresponding position, (i, j) and (m, n) are spatial positions on the feature map, and p and q are adjustment parameters. p and q are taken as 2 here.

[0081] The calculation formula of the channel weight is:

[0082]

[0083] Where C k represents the weight of the kth channel, x i is the value of the non-zero region in the ith feature map, and y iis the value of the zero region in the i-th feature map. The channel weight calculates the proportion of non-zero data of each channel, and then sums the proportions of all channels. The logarithm of the proportion of non-zero data of a certain channel divided by the total proportion of non-zero data is the weight of the channel.

[0084] S302, the feature maps extracted from different CNN layers pass image information on different levels, so the features extracted from different network levels are not the same for retrieval. The shallower layer learns general image features, and the deeper layer learns abstract image features. For images very similar to the query image, especially when there are a large number of images of the same type, the image features extracted from a single layer cannot ensure that important information is contained, and often subtle differences are distinguished. Therefore, the combination of features is more robust. Different levels of feature fusion can combine the advantages of global and local features. VGG16 includes 5 convolutional blocks and three fully connected layers. Experimental results show that selecting the features of Block3, Block4 and Block5 to splice into a new feature vector can achieve very good retrieval accuracy under the feature length of 1280. Under the condition that 30 retrieval images are returned for each query image, 97% of the images can retrieve the correct numbered images.

[0085] S303, the 1x256, 1x512, 1x512 feature vectors output by Block3, Block4 and Block5 are spliced into a 1x1280 feature vector. Then each component of the feature vector is divided by the modulus of the feature vector to normalize. The feature vector of the image and the number of the image are indexed one by one to generate a feature code library. For n film categories, n feature libraries are established respectively

[0086] S4, image classification is performed on the query image, which is divided into a certain subclass, and the features of the query image are extracted; the query image is input, and the trained classification network is called to judge the category of the query image. The feature vector of the query image is extracted in the same way as the film image in step S3.

[0087] S5, the features of the query image are compared with the film feature library of the subclass to output the top k images with the highest similarity as the preliminary retrieval result. Assuming that the predicted category of the query image is the i-th category, the similarity between the feature vector of the query image and the feature vector stored in the i-th feature library is calculated. The cosine distance is used as the similarity measurement algorithm. According to the order from large to small, the images in the i-th feature library are sorted, and the top k images with the highest similarity are output as the preliminary retrieval result. See the online processing module in the attached Figure 2

[0088] ​S501, image classification is performed on the search image, a trained classifier S2 is called to obtain probability values of the search image being classified into different film categories, and the category with the highest probability value is the category of the search image.

[0089] S502, similarity measurement is to calculate the similarity between the query feature vector and the image library feature vector, and output the top k images with the highest similarity as the search result. The methods of similarity measurement include Euclidean distance, Manhattan distance, cosine similarity, Hamming distance, Jaccard distance, etc. We select cosine similarity to calculate the similarity. For two vectors X=(x1,2,... n ), Y=(y1,2,... n ), the similarity calculation formula between them is as follows.

[0090] The calculation formula of Euclidean distance is:

[0091]

[0092] The calculation formula of Manhattan distance is:

[0093]

[0094] The formula of cosine similarity is:

[0095]

[0096] The calculation formula of Jaccard distance is:

[0097]

[0098] The cosine similarity between the feature vector of the query image and the feature vector in the i-th film feature library is calculated, and the top k images with the highest similarity are output as the first search result.

[0099] S6, the k images searched are re-searched to obtain the final search result.

[0100] S601, the first search result is corrected by using the extended query (QE, query expansion). The idea of extended query is similar to KNN. In image retrieval, the feature vectors of the query image and the top-k image feature vectors are averaged, and then a query is performed again. After the extended query, the images that are not present in the first search are found by re-searching in the neighborhood of the first search result.

[0101] S602、Another idea of secondary retrieval is to first filter out the top k images most similar to the query image through the retrieval algorithm, and then input the k images again into the retrieval algorithm to obtain new k retrieval results. Finally, the results of the two retrievals are sorted according to the similarity and output as the final retrieval results. The significance is to make the ranking of the retrieval results more accurate, but it will greatly increase the calculation amount and query time. For example, the first retrieval outputs the top 10 images with the highest similarity. Then, the 10 images are used as query images for retrieval again, and the top 10 retrieval results are returned for each image, a total of 100 query results, and the top 30 images with the highest similarity are selected as the final retrieval results.

[0102] The Figure 5 The effect of color film image classification retrieval is shown, where the picture in the first column of each row is a query image uploaded by a user to the retrieval system, and the images in columns 2 to 6 show the query results output by the retrieval system. Here, 5 images from the color film library are output as retrieval results for each query image, and the picture selected by the bold frame is the correct image of the retrieval.

[0103] The accuracy and query speed are used to evaluate the effectiveness of the method.

[0104] Retrieval accuracy Accuracy@Top-k (briefly Accuracy@k): The proportion of correct images in the total images in the top k retrieved images. The calculation formula is as follows:

[0105]

[0106]

[0107] Retrieval recognition speed fps: The number of query images that can be identified per second. The calculation formula is as follows:

[0108]

[0109] Where N is the number of images retrieved, T is the time for retrieving N images, and the reciprocal of fps is the time for retrieving one image.

[0110] If the top k images contain the number category of the query image, it can be determined that the retrieval result of the image is accurate. The accuracy of Top-1 is 0.81, the accuracy of Top-10 is 0.94, and the accuracy of Top-30 is 0.97, that is, the proportion of correct images in the output 30 images is 0.97, fps is 16, which meets the needs of actual production and life. Manual auxiliary classification query can be performed according to actual conditions to obtain the best results.

[0111] In still another embodiment of the present application, a color film matching system based on image classification retrieval is provided, which can be used to implement the above-mentioned color film matching method based on image classification retrieval.

[0112] The acquisition module is configured to acquire color film images, perform image data enhancement and category division, and construct a color film dataset.

[0113] The feature extraction module is configured to divide the color film dataset into a training set and a test set, and train the color film dataset by using the improved VGG16 network to obtain a feature extraction network.

[0114] The color film feature library establishment module is configured to extract features of the color film images, perform feature enhancement and feature fusion when extracting the features, and establish a color film feature library.

[0115] The image feature extraction module is configured to perform image classification on the query image and extract features of the query image.

[0116] The output module is configured to perform similarity measurement on the query image and the color film feature library to obtain the top k images with the highest similarity, and perform secondary retrieval on the k images retrieved to obtain a final retrieval result.

[0117] The division of the modules in the embodiments of the present application is illustrative, and is merely a logical function division, and another division mode can be used in actual implementation.

[0118] In still another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory being configured to store a computer program, the computer program comprising program instructions, and the processor being configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc., which are the core of computing and control of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the color film matching method based on image classification retrieval.

[0119] In still another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in the computer device, and is configured to store programs and data. It can be understood that the computer readable storage medium herein can include the built-in storage medium in the computer device, and of course can also include the expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the color film matching method based on image classification retrieval in the above embodiments.

[0120] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0121] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0122] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0124] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application is described in detail with reference to the above embodiments, those skilled in the field should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A color film matching method based on image classification search, characterized by, The application relates to a method for searching a color film image, and belongs to the technical field of image processing. The method comprises the following steps: Collecting a color film image, performing image data enhancement and category division, and constructing a color film data set; Dividing the color film data set into a training set and a test set, training the color film data set by using an improved VGG16 network, and obtaining a feature extraction network; Extracting features of the color film image, performing feature enhancement and feature fusion when the features are extracted, and establishing a color film feature library; Performing image classification on a query image, and extracting features of the query image; Carrying out similarity measurement on the query image and the color film feature library, and obtaining the top k images with the highest similarity; Performing secondary search on the k images searched out, and obtaining a final search result; Calling the feature extraction network, extracting color film features by using a convolutional layer in the network, and enhancing feature weights of texture and pattern regions on a feature map by means of channel weighting and spatial weighting operations: Selecting output features of Block3, Block4 and Block5 layers of the improved VGG16 network, fusing the features, splicing 1x256, 1x512 and 1x512 feature vectors output by the three blocks into a 1x1280 feature vector, and then unitizing each component of the feature vector by dividing the component by a module of the feature vector; indexing the feature vector of the image and the number of the image one by one, generating a feature code library, and establishing n feature libraries for n color film categories; Performing image classification on a query image: The query image refers to a picture uploaded by a user and required to be searched by the system, is an actual effect picture of a color film image printed on a floor tile by the user, the query image is input, and a classified network trained is called to predict a category of the query image; The feature extraction network and the feature weighting enhancement and feature fusion method are adopted to obtain a 1x1280-dimensional feature vector of the query image; the images in an i-th feature library are sorted according to a similarity from large to small, and the top k images with the highest similarity are output as a preliminary search result; 2. The color film matching method based on image classification retrieval according to claim 1, characterized in that, The feature vector of the k images searched out in the preliminary search is averaged with the feature vector of the query image to generate a new 1280-dimensional feature vector; the new feature vector is used to perform similarity measurement on the feature vectors in the i-th feature library again, and the top k images with the highest similarity are output as a final search result after secondary search.

3. The color film matching method based on image classification retrieval according to claim 1, characterized in that, The color film image is a high-resolution image simulating colors and textures of wood, rock and cloth, and each image has a unique number; a sliding window is slid on an original color film image to crop the color film image, and a data set is expanded by using a rotation and scaling data enhancement method; and the color film data is divided into n categories. The color film data set is divided into a training set and a test set, and is input into an improved VGG16 deep learning network for training. VGG16 has a total of 16 layers, including 13 convolutional layers and 3 fully connected layers, of which the 13 convolutional layers and their corresponding pooling layers serve as the feature extraction network, and the last 3 fully connected layers are responsible for completing the classification task; the feature extraction network of VGG16 is divided into 5 Blocks according to the positions of the 5 pooling layers, each Block including a two-dimensional convolutional layer, a ReLU activation function layer, a CBAM hybrid attention layer and a maximum pooling layer; the improved VGG16 adds a hybrid domain attention layer CBAM in each Block, and the output dimensions of the three fully connected layers in VGG16 are replaced by 512, 512 and 6 dimensions respectively, and a Dropout layer is added between the first and second fully connected layers.

4. The color film matching method based on image classification retrieval according to claim 3, characterized in that, In the process of training the improved VGG16 network, the ImageNet pre-training weight is loaded, and the network parameters in the classification network are adjusted through the fine-tuning method; first, freeze the training of the convolutional layer and the CBAM layer, and train the fully connected layer used for classification with a larger learning rate; when the accuracy of the validation set no longer improves, train the convolutional layer, CBAM layer and fully connected layer with a smaller learning rate.

5. A color film matching system based on image classification search, characterized by, It comprises: The acquisition module is used for acquiring the color film image, performing image data enhancement and class division, and constructing a color film data set; The feature extraction module is used for dividing the color film data set into training set and test set, and training by using the improved VGG16 network to obtain a feature extraction network; The color film feature library establishment module is used for extracting the features of the color film image, performing feature enhancement and feature fusion when extracting the features, and establishing a color film feature library; The image feature extraction module is used for image classification of the query image and extracting the features of the query image; The output module is used for similarity measurement between the query image and the color film feature library to obtain the top k images with the highest similarity; and secondary retrieval is performed on the k images retrieved to obtain the final retrieval result; The feature extraction network is called to extract the color film features by using the convolutional layer in the network, and the feature weights of the texture and pattern regions on the feature map are enhanced through channel weighting and spatial weighting operations: The output features of the Block3, Block4 and Block5 layers of the improved VGG16 network are selected for fusion, the 1x256, 1x512 and 1x512 feature vectors output by the three Blocks are spliced into a 1x1280 feature vector, and then each component of the feature vector is divided by the module of the feature vector for unitization; the feature vector of the image and the number of the image are indexed one by one to generate a feature code library, and n feature libraries are established for n color film categories; Image classification of the query image: The query image refers to the picture uploaded by the user for retrieval by the system, which is the actual effect picture of the color film image printed on the floor tile by the user; the query image is input, and the trained classification network is called to predict the category of the query image; The feature extraction network and the feature weighting enhancement and feature fusion method are used to obtain a 1*1280-dimensional feature vector of the query image; the images in the i-th feature library are sorted according to the order from large to small similarity, and the first k images with the highest similarity are output as the preliminary retrieval result; The feature vectors of the k images in the preliminary retrieval are averaged with the feature vector of the query image to generate a new 1280-dimensional feature vector; the new feature vector is used to perform similarity measurement with the feature vectors in the i-th feature library, and the first k images with the highest similarity are output as the final retrieval result after the secondary retrieval.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the color film matching method based on image classification retrieval according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program is executed by the processor to implement the steps of the color film matching method based on image classification retrieval according to any one of claims 1 to 4.

Images