A mural image retrieval method and system based on unsupervised feature engineering

Abstract

This invention proposes a method and system for retrieving mural images based on unsupervised feature engineering, belonging to the field of image retrieval technology. The method includes: acquiring mural images and constructing a mural image database; traversing each image in the mural image database and extracting features from each image based on feature engineering to construct a feature database; acquiring the query target and extracting the target features corresponding to the query target based on feature engineering; sequentially measuring the similarity between the features in the feature database and the target features, and returning the retrieval results in descending order of similarity. This invention can improve the retrieval accuracy of mural images while saving retrieval costs.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval technology, and in particular relates to a method and system for retrieving mural images based on unsupervised feature engineering. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Mural image retrieval, as a key technology for the digital preservation of cultural heritage, has broad development prospects. With the acceleration of the global digitization of cultural relics, mural retrieval technology will play an important role in the fields of cultural relic preservation, archaeological research, and cultural dissemination.

[0004] Currently, common image retrieval methods include supervised and unsupervised methods. Supervised image retrieval relies on manual annotation, using deep learning models to learn deep features and optimizing the feature space using classification or metric loss. These methods require professionals to meticulously annotate a large number of mural images, which is not only costly in terms of manpower and time, but also susceptible to subjective influences, making it difficult to meet the practical needs of a large number and diverse styles of mural images. Therefore, unsupervised mural image retrieval methods have been widely used due to their elimination of the need for manually annotated data.

[0005] However, existing unsupervised methods for retrieving mural images still face some intractable technical problems, such as:

[0006] (1) Existing technologies include unsupervised retrieval methods based on image features, which extract multi-layer semantic features of images for matching. However, such methods only extract features from the original image or images that have undergone simple enhancement processing. The extracted features are difficult to fully reflect the detailed information of the mural image, resulting in insufficient accuracy of feature description and thus affecting retrieval accuracy.

[0007] (2) There are also unsupervised retrieval methods based on text matching in the existing technology, but this method is difficult to control the retrieval cost for murals with abstract style and complex subject matter. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention provides a mural image retrieval method and system based on unsupervised feature engineering, which can improve the retrieval accuracy of mural images while saving retrieval costs.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] The first aspect of this invention provides a method for retrieving mural images based on unsupervised feature engineering.

[0011] A mural image retrieval method based on unsupervised feature engineering includes:

[0012] Acquire mural images and construct a mural image database;

[0013] The mural image database is traversed, and features of each image are extracted based on feature engineering to construct a feature database. The feature engineering process is as follows: the line drawing and color map of each image are obtained, the feature extractor is used to extract features from the original image, line drawing and color map corresponding to each image, and the feature fusioner is used to perform reverse feature fusion.

[0014] Obtain the query target and extract the target features corresponding to the query target based on feature engineering;

[0015] The similarity between the features in the feature database and the target features is measured sequentially, and the search results are returned in descending order of similarity.

[0016] Further, the line graph of the image is obtained, including: first, filtering the current image traversed in the mural image database using the Canny operator Gaussian kernel; then, calculating the horizontal and vertical gradients of the current image after filtering based on the Sobel operator; subsequently, determining the magnitude and direction of the gradients based on the obtained horizontal and vertical gradients; and performing pixel suppression and edge enhancement according to the magnitude and direction of the gradients.

[0017] Furthermore, pixel suppression and edge enhancement are performed based on the magnitude and direction of the gradient, including: first, quantizing the gradient direction into four main directions, and determining whether to suppress the pixel by comparing the gradient information of the current pixel's neighboring pixels; then, setting high and low thresholds for edge detection and connection, and determining whether to enhance the edge based on the comparison results between the current pixel and the set high and low thresholds.

[0018] Further, obtaining the color map of the image includes: performing Lab space transformation on the current image traversed in the mural image database, treating pixel colors as data points, and using the Kmeans algorithm to perform a five-class clustering operation on the data points.

[0019] Furthermore, the K-means algorithm is used to perform five-class clustering on the data points, including: randomly selecting multiple data points as initial cluster centers, and iteratively performing two steps: first, assigning all data points to the cluster containing the nearest cluster center; second, recalculating the mean of all data points in each cluster, and using this mean as the new cluster center.

[0020] Furthermore, a feature fusion processor is used to perform reverse feature fusion, including: calculating the relative entropy of the line graph features, original image features, and color map features corresponding to the current image, respectively, constructing a fusion matrix based on the obtained relative entropy calculation results, and flattening the constructed fusion matrix to obtain the fused features.

[0021] Furthermore, relative entropy calculations are performed on the line graph features, original image features, and color map features corresponding to the current image, including: first, calculating the relative entropy between the line graph features and the line graph features, original image features, and color map features respectively; then, calculating the relative entropy between the original image features and the line graph features, original image features, and color map features respectively; subsequently, calculating the relative entropy between the color map features and the line graph features, original image features, and color map features respectively.

[0022] A second aspect of the present invention provides a mural image retrieval system based on unsupervised feature engineering.

[0023] A mural image retrieval system based on unsupervised feature engineering includes:

[0024] The mural image database construction module is configured to: acquire mural images and construct a mural image database;

[0025] The feature engineering module is configured to: traverse each image in the mural image database and extract features from each image based on feature engineering to construct a feature database; wherein, the implementation process of the feature engineering is as follows: obtain the line drawing and color map of each image, use a feature extractor to extract features from the original image, line drawing, and color map corresponding to each image, and use a feature fusioner to perform reverse feature fusion; obtain the query target, and extract the target features corresponding to the query target based on feature engineering;

[0026] The similarity measurement module is configured to measure the similarity between features in the feature database and the target features in turn, and return the search results in descending order of similarity.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a mural image retrieval method based on unsupervised feature engineering as described in the first aspect of the present invention.

[0028] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a mural image retrieval method based on unsupervised feature engineering as described in the first aspect of the present invention.

[0029] The above one or more technical solutions have the following beneficial effects:

[0030] (1) This invention is based on feature engineering, obtaining line drawings and color maps for each image. A feature extractor is used to extract features from the original image, line drawing, and color map corresponding to each image, and a feature fusion tool is used for reverse feature fusion. The line drawing can accurately depict the outline structure and brushstroke details of the mural, while the color map can quantify the color distribution pattern. These two features complement the original image features, thus comprehensively covering the structural complexity, color specificity, and texture details of the mural. Therefore, compared with existing technologies, this invention has better accuracy in feature description and higher retrieval precision.

[0031] (2) The present invention employs a completely unsupervised end-to-end retrieval method, that is, from feature extraction of line graphs and color maps to feature fusion through inverse relative entropy calculation, no annotation information is required throughout the entire process; among which, line graphs are automatically generated through gradient operations, color maps are automatically divided into color spaces based on clustering algorithms, and feature fusion is achieved through matrix operations, without the need for supervised signals. Therefore, compared with the prior art, the present invention is more efficient and can better control retrieval costs.

[0032] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0034] Figure 1 This is a flowchart of a mural image retrieval method based on unsupervised feature engineering in Embodiment 1 of the present invention.

[0035] Figure 2 This is a flowchart illustrating the implementation of feature engineering in Embodiment 1 of the present invention.

[0036] Figure 3 This is a flowchart of obtaining a color map in Embodiment 1 of the present invention.

[0037] Figure 4 This is a flowchart of reverse feature fusion using a feature fusion processor in Embodiment 1 of the present invention. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] The overall approach proposed in this invention is as follows: This invention provides a mural image retrieval method based on unsupervised feature engineering. First, the original image is analyzed to obtain its line drawing and color map. Next, features are extracted from the original image, line drawing, and color map respectively. Then, the three types of features are fused to obtain a comprehensive descriptive feature. Following this, the similarity between the image and images in the database is queried based on the comprehensive descriptive feature technique to obtain a similarity measure. Finally, the retrieval results are returned based on the comprehensive similarity measure.

[0042] Example 1

[0043] This embodiment discloses a mural image retrieval method based on unsupervised feature engineering.

[0044] like Figure 1 As shown, a mural image retrieval method based on unsupervised feature engineering includes:

[0045] Step S1: Obtain mural images and construct a mural image database;

[0046] Step S2: Traverse each image in the mural image database and extract features from each image based on feature engineering to construct a feature database; wherein, the implementation process of the feature engineering is as follows: obtain the line drawing and color map of each image, use a feature extractor to extract features from the original image, line drawing and color map corresponding to each image, and use a feature fusioner to perform reverse feature fusion.

[0047] Step S3: Obtain the query target and extract the target features corresponding to the query target based on feature engineering;

[0048] Step S4: Measure the similarity between the features in the feature database and the target features in turn, and return the search results in descending order of similarity.

[0049] Based on the above process, this invention can improve the retrieval accuracy of mural images while saving retrieval costs. To facilitate understanding of the technical solution of this invention, the specific implementation methods of this invention will be further explained and described below.

[0050] In step S1, mural images are acquired, and a mural image database is constructed. Specifically, this involves collecting mural images to obtain the mural image database. Among them, the mural image database The capacity is and with Represents a database of mural images The first in Zhang Image, i.e. .

[0051] In step S2, each image in the mural image database is traversed, and features of each image are extracted based on feature engineering to construct a feature database.

[0052] By iterating through each image in the mural image database The features extracted using the feature engineering proposed in this invention are denoted as follows: To construct a feature database from all image features .in, Represents a database of mural images The first in Zhang Image; Represents a database of mural images The first in Zhang Image Features ;database Represents a set of features.

[0053] like Figure 2 As shown, the feature engineering implementation process is as follows: Obtain the line drawing and color map of each image; use a feature extractor to extract features from the original image, line drawing, and color map corresponding to each image; and use a feature fusion processor for reverse feature fusion. Specifically, this can be achieved through the following methods:

[0054] Step S2-1: Obtain the line drawing of the image.

[0055] First, the Canny Gaussian kernel is used to filter the current image traversed in the mural image database; that is, let the original image be... The Canny operator is used to obtain the line drawing, and the Gaussian kernel shown in the following formula is used to compare the original image. Perform a filtering operation to obtain the filtered image. ,Right now:

[0056] ;

[0057] Next, the horizontal gradient of the current image after filtering is calculated based on the Sobel operator. and vertical gradient ,Right now:

[0058] ;

[0059] ;

[0060] Subsequently, the magnitude of the gradient is determined based on the obtained horizontal and vertical gradients. and direction ,Right now:

[0061] ;

[0062] ;

[0063] Finally, pixel suppression and edge enhancement are performed based on the magnitude and direction of the gradient, specifically:

[0064] First, the direction of the gradient The gradient is quantized into four main directions (0°, 45°, 90°, 135°). By comparing the gradient magnitude of the current pixel with its two neighboring pixels along the gradient direction, if it is not the maximum value, it is suppressed (set to 0). Subsequently, high and low thresholds are set for edge detection and connection. Based on the comparison results between the current pixel and the set high and low thresholds, it is determined whether the edge is enhanced or not. That is, if the gradient magnitude of the current pixel is greater than the high threshold, it is a strong edge; if it is less than the low threshold, it is a non-edge; if it is between the low and high thresholds, it is a weak edge.

[0065] Step S2-2: Obtain the color map of the image.

[0066] like Figure 3 As shown, obtaining the color map of the image includes: converting the current image traversed in the mural image database from RGB space to Lab space, treating pixel colors as data points, and using the Kmeans algorithm to perform a five-class clustering operation on the data points.

[0067] The K-means algorithm is used to cluster data points into five classes. This involves randomly selecting five data points as initial cluster centers and iteratively performing two core operations: First, all data points are assigned to the cluster containing the nearest cluster center, calculated by euclidean distance between each data point and its corresponding center. Second, the mean of all data points in each cluster is recalculated, and this mean is used as the new cluster center. These two steps are repeated until the cluster center positions no longer change significantly. Finally, the cluster label for each data point and the converged cluster centers are output. After clustering, the color of each cluster pixel is assigned the color of the cluster center, resulting in a color map.

[0068] Step S2-3: Use a feature extractor to extract features from the original image, line graph, and color map corresponding to each image.

[0069] Using dimension The feature extractor extracts features from the original image, line graph, and color map respectively, as shown in the following formulas, and then performs feature fusion to obtain the fused features.

[0070] ;

[0071] ;

[0072] ;

[0073] in, Indicates the features of the original image. To represent the characteristics of a line drawing. Represents the characteristics of a color map; express The One portion, express The One portion, express The Each component.

[0074] Step S2-4: Perform reverse feature fusion using a feature fusion facilitator.

[0075] like Figure 4 As shown, a feature fusion facilitator is used for reverse feature fusion, including: calculating the relative entropy of the line graph features, original image features and color map features corresponding to the current image respectively; constructing a fusion matrix based on the obtained relative entropy calculation results; and flattening the constructed fusion matrix to obtain the fused features.

[0076] The relative entropy is calculated for the line graph features, original image features, and color map features corresponding to the current image, respectively, including:

[0077] 1) Calculate the features of the line drawing respectively. Features of line drawings Original image features Color map features Relative entropy:

[0078] ;

[0079] ;

[0080] ;

[0081] in, express and The relative entropy, express and The relative entropy, express and The relative entropy.

[0082] 2) Calculate the features of the original image respectively. Features of line drawings Original image features Color map features The relative entropy, that is:

[0083] ;

[0084] ;

[0085] ;

[0086] in, express and The relative entropy, express and The relative entropy, express and The relative entropy.

[0087] 3) Calculate the color map features separately. Features of line drawings Original image features Color map features The relative entropy, that is:

[0088] ;

[0089] ;

[0090] ;

[0091] in, express and The relative entropy, express and The relative entropy, express and The relative entropy.

[0092] Based on this, a fusion matrix can be constructed as shown in the following equation:

[0093] ;

[0094] By flattening the fusion matrix, the fusion features can be obtained. .

[0095] In step S3, the query target is obtained, and the target features corresponding to the query target are extracted based on feature engineering. The query target is then obtained. Then, the feature engineering proposed in this invention can be used to extract the target features. The implementation process of feature engineering is the same as step S2.

[0096] In step S4, the features in the feature database are sequentially compared with the target features for similarity measurement, and the search results are returned in descending order of similarity.

[0097] The similarity between features in the feature database and target features is measured, i.e.:

[0098] ;

[0099] in, Indicate the query target With the mural image database Zhang Image Similarity, Representation feature database The first in Features of the image Indicate the query target Its characteristics.

[0100] Therefore, a similarity metric can be constructed based on all the similarity calculation results. ,Right now:

[0101] ;

[0102] in, Indicate the query target Similarity to the first image in the mural image database, Indicate the query target Similarity to the second image in the mural image database, Indicate the query target With the mural image database Similarity of images.

[0103] Finally, the similarity measures are sorted, and the search results are returned in descending order of similarity.

[0104] Example 2

[0105] This embodiment discloses a mural image retrieval system based on unsupervised feature engineering.

[0106] A mural image retrieval system based on unsupervised feature engineering includes:

[0107] The mural image database construction module is configured to: acquire mural images and construct a mural image database;

[0108] The feature engineering module is configured to: traverse each image in the mural image database and extract features from each image based on feature engineering to construct a feature database; wherein, the implementation process of the feature engineering is as follows: obtain the line drawing and color map of each image, use a feature extractor to extract features from the original image, line drawing, and color map corresponding to each image, and use a feature fusioner to perform reverse feature fusion; obtain the query target, and extract the target features corresponding to the query target based on feature engineering;

[0109] The similarity measurement module is configured to measure the similarity between features in the feature database and the target features in turn, and return the search results in descending order of similarity.

[0110] Example 3

[0111] The purpose of this embodiment is to provide a computer-readable storage medium.

[0112] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a mural image retrieval method based on unsupervised feature engineering as described in Embodiment 1 of this disclosure.

[0113] Example 4

[0114] The purpose of this embodiment is to provide an electronic device.

[0115] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a mural image retrieval method based on unsupervised feature engineering as described in Embodiment 1 of this disclosure.

[0116] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0117] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0118] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for retrieving mural images based on unsupervised feature engineering, characterized in that, include: Acquire mural images and construct a mural image database; The mural image database is traversed, and features of each image are extracted based on feature engineering to construct a feature database. The feature engineering process is as follows: the line drawing and color map of each image are obtained, the feature extractor is used to extract features from the original image, line drawing and color map corresponding to each image, and the feature fusioner is used to perform reverse feature fusion. Obtaining the color map of the image includes: performing Lab space transformation on the current image traversed in the mural image database, treating pixel colors as data points, and using the Kmeans algorithm to perform a five-class clustering operation on the data points; The K-means algorithm is used to cluster the data points into five classes, which includes: randomly selecting multiple data points as initial cluster centers, and iteratively performing two steps: first, assigning all data points to the cluster containing the nearest cluster center; second, recalculating the mean of all data points in each cluster and using this mean as the new cluster center. Inverse feature fusion is performed using a feature fusion processor, which includes: calculating the relative entropy of the line graph features, original image features, and color map features corresponding to the current image, respectively; constructing a fusion matrix based on the calculated relative entropy; and flattening the constructed fusion matrix to obtain the fused features. Obtain the query target and extract the target features corresponding to the query target based on feature engineering; The similarity between the features in the feature database and the target features is measured sequentially, and the search results are returned in descending order of similarity.

2. The mural image retrieval method based on unsupervised feature engineering as described in claim 1, characterized in that, The process of obtaining the line graph of the image includes: first, filtering the current image traversed in the mural image database using the Canny operator Gaussian kernel; then, calculating the horizontal and vertical gradients of the current image after filtering based on the Sobel operator; subsequently, determining the magnitude and direction of the gradients based on the obtained horizontal and vertical gradients; and performing pixel suppression and edge enhancement based on the magnitude and direction of the gradients.

3. The mural image retrieval method based on unsupervised feature engineering as described in claim 2, characterized in that, Pixel suppression and edge enhancement are performed based on the magnitude and direction of the gradient, including: First, the gradient direction is quantized into four main directions, and the pixel suppression is determined by comparing the gradient information of the current pixel's neighboring pixels; then, high and low thresholds are set for edge detection and connection, and the edge enhancement is determined based on the comparison results between the current pixel and the set high and low thresholds.

4. The mural image retrieval method based on unsupervised feature engineering as described in claim 1, characterized in that, The relative entropy of the line graph features, original image features, and color map features corresponding to the current image is calculated separately, including: first, calculating the relative entropy of the line graph features with the line graph features, original image features, and color map features respectively; then, calculating the relative entropy of the original image features with the line graph features, original image features, and color map features respectively; and finally, calculating the relative entropy of the color map features with the line graph features, original image features, and color map features respectively.

5. A mural image retrieval system based on unsupervised feature engineering, employing the mural image retrieval method as described in any one of claims 1-4, characterized in that, include: The mural image database construction module is configured to: acquire mural images and construct a mural image database; The feature engineering module is configured to: traverse each image in the mural image database and extract features from each image based on feature engineering to construct a feature database; wherein, the implementation process of the feature engineering is as follows: obtain the line drawing and color map of each image, use a feature extractor to extract features from the original image, line drawing, and color map corresponding to each image, and use a feature fusioner to perform reverse feature fusion; obtain the query target, and extract the target features corresponding to the query target based on feature engineering; The similarity measurement module is configured to measure the similarity between features in the feature database and the target features in turn, and return the search results in descending order of similarity.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the mural image retrieval method based on unsupervised feature engineering as described in any one of claims 1-4.

7. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the mural image retrieval method based on unsupervised feature engineering as described in any one of claims 1-4.

Images