Mural image retrieval method and system based on unsupervised feature engineering

Abstract

The invention provides a mural image retrieval method and system based on unsupervised feature engineering, and belongs to the technical field of image retrieval. Comprising the steps of obtaining a mural image, and constructing a mural image database; traversing each image in the mural image database, extracting features of each image based on feature engineering, and constructing a feature database; obtaining a query target, and extracting a target feature corresponding to the query target based on the feature engineering; and performing similarity measurement on the features in the feature database and the target features in sequence, and returning retrieval results according to the similarity from large to small. According to the method, the retrieval precision of the mural image can be improved on the basis of saving the retrieval cost.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval technology, and in particular relates to a mural image retrieval method and system based on unsupervised feature engineering. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As a key technology for the digital preservation of cultural heritage, mural image retrieval has broad development prospects. With the acceleration of the digitization of cultural relics worldwide, mural retrieval technology will play an important role in cultural relic preservation, archaeological research, and cultural communication.

[0004] Currently, common image retrieval methods include supervised and unsupervised methods. Supervised image retrieval relies on manual annotation, using deep learning models to learn deep features and optimize the feature space using classification or metric losses. These methods require professionals to meticulously annotate large numbers of mural images, which is not only labor-intensive and time-consuming, but also susceptible to subjective factors, making them difficult to adapt to the large number of mural images and the diverse styles they represent. Consequently, unsupervised mural image retrieval methods have gained widespread application due to their lack of manual data annotation.

[0005] However, existing mural image retrieval methods based on unsupervised methods still have some difficult-to-solve technical problems, such as: (1) In the existing technology, there are unsupervised retrieval methods based on image features, which extract multi-layer semantic features of images for matching. However, such methods only extract features from original images or images that have undergone simple enhancement processing. The extracted features are difficult to fully reflect the detailed information of the mural images, resulting in insufficient accuracy of feature description, which in turn affects the retrieval accuracy.

[0006] (2) There are also unsupervised retrieval methods based on text matching in the existing technology, but this method is difficult to control the retrieval cost for murals with abstract styles and complex themes. Summary of the Invention

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a mural image retrieval method and system based on unsupervised feature engineering, which can improve the retrieval accuracy of mural images while saving retrieval costs.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: A first aspect of the present invention provides a mural image retrieval method based on unsupervised feature engineering.

[0009] A mural image retrieval method based on unsupervised feature engineering, comprising: Acquire mural images and build a mural image database; Traverse each image in the mural image database and extract features from each image based on feature engineering to build a feature database. The feature engineering implementation process is as follows: obtain the line drawing and color map of each image, use a feature extractor to extract features from the original image, line drawing, and color map corresponding to each image, and use a feature fusion device to perform reverse feature fusion. Obtain the query target and extract the target features corresponding to the query target based on feature engineering; The similarity between the features in the feature database and the target features is measured in turn, and the search results are returned in descending order of similarity.

[0010] Furthermore, a line drawing of the image is obtained, including: first, using the Canny operator Gaussian kernel to filter the current image traversed in the mural image database; then, calculating the horizontal gradient and vertical gradient of the filtered current image based on the Sobel operator; then, determining the amplitude and direction of the gradient based on the obtained horizontal gradient and vertical gradient; and performing pixel suppression and edge enhancement according to the amplitude and direction of the gradient.

[0011] Furthermore, pixel suppression and edge enhancement are performed according to the amplitude and direction of the gradient, including: first, quantizing the direction of the gradient into four main directions, and determining whether the pixel is suppressed by comparing the gradient information of the pixels in the neighborhood of the current pixel; then, setting high and low thresholds for edge detection and connection, and determining whether the edge is enhanced based on the comparison results of the current pixel with the set high and low thresholds.

[0012] Furthermore, obtaining a color map of the image includes: performing Lab space conversion on the current image traversed in the mural image database, treating pixel colors as data points, and performing a five-category clustering operation on the data points using a Kmeans algorithm.

[0013] Furthermore, the Kmeans algorithm is used to cluster the data points into five categories, including: randomly selecting multiple data points as initial cluster centers, and iteratively performing two steps, namely: in the first step, all data points are assigned to the cluster where the nearest cluster center is located; in the second step, the mean of all data points in each cluster is recalculated and the mean is used as the new cluster center.

[0014] Furthermore, a feature fuser is used to perform reverse feature fusion, including: performing relative entropy calculations on the line drawing features, original image features, and color map features corresponding to the current image, respectively, and constructing a fusion matrix based on the relative entropy calculation results; flattening the constructed fusion matrix to obtain fusion features.

[0015] Furthermore, relative entropies are calculated for the line drawing features, original image features and color map features corresponding to the current image, including: first, calculating the relative entropies of the line drawing features, the line drawing features, the original image features and the color map features, respectively; then, calculating the relative entropies of the original image features, the line drawing features, the original image features and the color map features, respectively; and then, calculating the relative entropies of the color map features, the line drawing features, the original image features and the color map features, respectively.

[0016] A second aspect of the present invention provides a mural image retrieval system based on unsupervised feature engineering.

[0017] A mural image retrieval system based on unsupervised feature engineering, comprising: The mural image database construction module is configured to: obtain mural images and construct a mural image database; The feature engineering module is configured to: traverse each image in the mural image database and extract features of each image based on feature engineering to construct a feature database; wherein the feature engineering implementation process is as follows: obtain the line drawing and color map of each image, use the feature extractor to extract features from the original image, line drawing and color map corresponding to each image, and use the feature fusion device to perform reverse feature fusion; obtain the query target, and extract the target features corresponding to the query target based on feature engineering; The similarity measurement module is configured to measure the similarity between the features in the feature database and the target features in sequence, and return the search results in descending order of similarity. The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the mural image retrieval method based on unsupervised feature engineering as described in the first aspect of the present invention.

[0018] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, the steps in the mural image retrieval method based on unsupervised feature engineering as described in the first aspect of the present invention are implemented.

[0019] One or more of the above technical solutions have the following beneficial effects: (1) Based on feature engineering, the present invention obtains a line drawing and a color map for each image. A feature extractor is used to extract features from the original image, line drawing, and color map corresponding to each image, and a feature fusion device is used to perform reverse feature fusion. The line drawing can accurately depict the contour structure and brushstroke details of the mural, and the color map can quantify the color distribution pattern. The two complement the features of the original image and can fully cover the structural complexity, color specificity, and texture details of the mural. Therefore, compared with the existing technology, the present invention has better feature description accuracy and higher retrieval precision.

[0020] (2) The present invention employs a completely unsupervised, end-to-end retrieval method. From feature extraction of line drawings and color maps to feature fusion via reverse relative entropy calculation, no annotation information is required. Line drawings are automatically generated via gradient calculations, and color maps are automatically partitioned into color spaces based on clustering algorithms. Feature fusion is achieved via matrix operations, without the need for supervisory signal guidance. Therefore, compared to existing technologies, the present invention is more efficient and can better control retrieval costs.

[0021] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a flowchart of a mural image retrieval method based on unsupervised feature engineering in Example 1 of the present invention.

[0024] Figure 2 This is a flowchart for implementing feature engineering in Example 1 of the present invention.

[0025] Figure 3 This is a flowchart of obtaining a color map in Example 1 of the present invention.

[0026] Figure 4 This is a flowchart of reverse feature fusion using a feature fuser in Example 1 of the present invention. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0028] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0029] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0030] The overall idea proposed by the present invention is as follows: The present invention provides a mural image retrieval method based on unsupervised feature engineering. First, the original image is analyzed to obtain the line drawing and color map of the original image; then, the features of the original image, line drawing, and color map are extracted respectively; then, the three types of features are fused to obtain a comprehensive description feature; then, based on the comprehensive description feature technology, the similarity between the image and the images in the database is queried to obtain a similarity measure; finally, based on the comprehensive similarity measure, the retrieval results are returned.

[0031] Example 1 This embodiment discloses a mural image retrieval method based on unsupervised feature engineering.

[0032] like Figure 1 As shown in FIG, a mural image retrieval method based on unsupervised feature engineering includes: Step S1: Acquire mural images and build a mural image database; Step S2: traverse each image in the mural image database and extract features of each image based on feature engineering to construct a feature database; wherein the feature engineering implementation process is as follows: obtain the line drawing and color map of each image, use the feature extractor to extract features from the original image, line drawing and color map corresponding to each image, and use the feature fusion device to perform reverse feature fusion; Step S3: Obtain the query target, and extract the target features corresponding to the query target based on feature engineering; Step S4: measure the similarity between the features in the feature database and the target features in sequence, and return the search results in descending order of similarity.

[0033] Based on the above process, the present invention can improve the retrieval accuracy of mural images while saving retrieval costs. To facilitate understanding of the technical solution of the present invention, the specific implementation method of the technical solution of the present invention is further explained and illustrated below.

[0034] In step S1, mural images are acquired and a mural image database is constructed. Specifically, mural images are collected to obtain a mural image database. Among them, the mural image database The capacity is , and Represents the mural image database The images, i.e. .

[0035] In step S2, each image in the mural image database is traversed, and the features of each image are extracted based on feature engineering to construct a feature database.

[0036] By traversing each image in the mural image database , the feature engineering proposed by the present invention is used to extract its features and record them as , all image features constitute a feature database .in, Represents the mural image database The images; Represents the mural image database The images Features, ;database Represents a feature set.

[0037] like Figure 2 As shown in the figure, the implementation process of feature engineering is as follows: obtain the line drawing and color map of each image, use the feature extractor to extract features from the original image, line drawing and color map corresponding to each image, and use the feature fusion device to perform reverse feature fusion. This can be achieved through the following methods: Step S2-1: Obtain a line drawing of an image.

[0038] First, the Canny operator Gaussian kernel is used to filter the current image traversed in the mural image database, that is: the original image is recorded as , use the Canny operator to obtain the line graph, and use the Gaussian kernel shown in the following formula to correct the original image Perform filtering operations to obtain filtered images ,Right now: ; Next, the horizontal gradient of the current image after filtering is calculated based on the Sobel operator and vertical gradient ,Right now: ; ; Then, the magnitude of the gradient is determined based on the obtained horizontal and vertical gradients. and direction ,Right now: ; ; Finally, pixel suppression and edge enhancement are performed based on the gradient amplitude and direction. Specifically: First, the direction of the gradient Quantized into four main directions (0°, 45°, 90°, 135°), the gradient amplitude of the current pixel is compared with its two neighboring pixels along the gradient direction. If it is not the maximum value, it is suppressed (set to 0). Subsequently, high and low thresholds are set for edge detection and connection. Based on the comparison results of the current pixel with the set high and low thresholds, it is determined whether the edge is enhanced or not. That is, if the gradient amplitude of the current pixel is greater than the high threshold, it is a strong edge; if it is less than the low threshold, it is not an edge; if it is between the low and high thresholds, it is a weak edge.

[0039] Step S2-2: Obtain the color map of the image.

[0040] like Figure 3 As shown, obtaining the color map of the image includes: converting the current image traversed in the mural image database from the RGB space to the Lab space, treating the pixel color as a data point, and using the Kmeans algorithm to cluster the data points into five categories.

[0041] The Kmeans algorithm is used to cluster the data points into five categories. This involves randomly selecting five data points as initial cluster centers and iteratively performing two core operations: first, assigning all data points to the cluster with the closest cluster center. This can be achieved by calculating the Euclidean distance between the data point and each center. Second, recalculating the mean of all data points in each cluster and using this mean as the new cluster center. These two steps are repeated until the position of the cluster center no longer changes significantly. Finally, the cluster label and the converged cluster center are output for each data point. After clustering, the color of each cluster pixel is assigned to the color of the cluster center, resulting in a color map.

[0042] Step S2-3: Use a feature extractor to extract features from the original image, line image, and color map corresponding to each image.

[0043] Using the dimension The feature extractor extracts the original image features, line image features, and color map features respectively as shown in the following formula, and performs feature fusion to obtain the fused features.

[0044] ; ; ; in, Represents the original image features, Represents the line graph feature, Represents the color map feature; express No. A quantity, express No. A quantity, express No. A portion.

[0045] Step S2-4: Use a feature fuser to perform reverse feature fusion.

[0046] like Figure 4 As shown in FIG, a feature fuser is used to perform reverse feature fusion, including: performing relative entropy calculations on the line drawing features, original image features, and color map features corresponding to the current image, respectively, and constructing a fusion matrix based on the relative entropy calculation results; flattening the constructed fusion matrix to obtain fusion features.

[0047] The relative entropy calculation is performed on the line drawing features, original image features and color map features corresponding to the current image, including: 1) Calculate line graph features separately With line drawing features , original image features , color map features The relative entropy of: ; ; ; in, express and The relative entropy of express and The relative entropy of express and The relative entropy of .

[0048] 2) Calculate the original image features separately With line drawing features , original image features , color map features The relative entropy is: ; ; ; in, express and The relative entropy of express and The relative entropy of express and The relative entropy of .

[0049] 3) Calculate color map features separately With line drawing features , original image features , color map features The relative entropy is: ; ; ; in, express and The relative entropy of express and The relative entropy of express and The relative entropy of .

[0050] On this basis, the fusion matrix shown in the following formula can be constructed: ; By flattening the fusion matrix, the fusion features can be obtained .

[0051] In step S3, the query target is obtained, and the target features corresponding to the query target are extracted based on feature engineering. After that, the feature engineering proposed by the present invention can be used to extract target features ; Among them, the implementation process of feature engineering is consistent with step S2.

[0052] In step S4, similarity measurements are performed on the features in the feature database and the target features in sequence, and the search results are returned in descending order of similarity.

[0053] The similarity between the features in the feature database and the target features is measured, that is: ; in, Indicates the query target Compared with the mural image database images The similarity, Representation feature database The The features of the image, Indicates the query target characteristics.

[0054] Thus, the similarity measure can be constructed based on all similarity calculation results. ,Right now: ; in, Indicates the query target Similarity with the first image in the mural image database, Indicates the query target Similarity with the second image in the mural image database, Indicates the query target Compared with the mural image database Similarity of images.

[0055] Finally, the similarity measures are sorted and the search results are returned in descending order of similarity.

[0056] Example 2 This embodiment discloses a mural image retrieval system based on unsupervised feature engineering.

[0057] A mural image retrieval system based on unsupervised feature engineering, comprising: The mural image database construction module is configured to: obtain mural images and construct a mural image database; The feature engineering module is configured to: traverse each image in the mural image database and extract features of each image based on feature engineering to construct a feature database; wherein the feature engineering implementation process is as follows: obtain the line drawing and color map of each image, use the feature extractor to extract features from the original image, line drawing and color map corresponding to each image, and use the feature fusion device to perform reverse feature fusion; obtain the query target, and extract the target features corresponding to the query target based on feature engineering; The similarity measurement module is configured to measure the similarity between the features in the feature database and the target features in sequence, and return the search results in descending order of similarity. Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0058] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a mural image retrieval method based on unsupervised feature engineering as described in the first embodiment of the present disclosure.

[0059] Example 4 The purpose of this embodiment is to provide an electronic device.

[0060] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of a mural image retrieval method based on unsupervised feature engineering as described in the first embodiment of the present disclosure are implemented.

[0061] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0062] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0063] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A mural image retrieval method based on unsupervised feature engineering, characterized in that: include: Acquire mural images and build a mural image database; Traverse each image in the mural image database and extract features from each image based on feature engineering to build a feature database. The feature engineering implementation process is as follows: obtain the line drawing and color map of each image, use a feature extractor to extract features from the original image, line drawing, and color map corresponding to each image, and use a feature fusion device to perform reverse feature fusion. Obtain the query target and extract the target features corresponding to the query target based on feature engineering; The similarity between the features in the feature database and the target features is measured in turn, and the search results are returned in descending order of similarity.

2. The mural image retrieval method based on unsupervised feature engineering according to claim 1, characterized in that: Obtaining a line drawing of an image includes: first, filtering a current image traversed in a mural image database using a Canny operator Gaussian kernel; then, calculating a horizontal gradient and a vertical gradient of the filtered current image based on a Sobel operator; then, determining a gradient amplitude and a direction based on the obtained horizontal gradient and vertical gradient; and performing pixel suppression and edge enhancement according to the gradient amplitude and direction.

3. A mural image retrieval method based on unsupervised feature engineering as claimed in claim 2, characterized in that: Pixel suppression and edge enhancement are performed based on the amplitude and direction of the gradient, including: first, quantizing the direction of the gradient into four main directions, and determining whether the pixel is suppressed by comparing the gradient information of the pixels in the neighborhood of the current pixel; then, setting high and low thresholds for edge detection and connection, and determining whether the edge is enhanced based on the comparison results of the current pixel with the set high and low thresholds.

4. The mural image retrieval method based on unsupervised feature engineering according to claim 1, characterized in that: Obtaining a color map of an image includes: performing Lab space conversion on a current image traversed in a mural image database, treating pixel colors as data points, and performing a five-category clustering operation on the data points using a Kmeans algorithm.

5. The mural image retrieval method based on unsupervised feature engineering according to claim 4, characterized in that: The Kmeans algorithm is used to cluster the data points into five categories, including randomly selecting multiple data points as the initial cluster centers and iteratively performing two steps, namely: in the first step, all data points are assigned to the cluster where the nearest cluster center is located; in the second step, the mean of all data points in each cluster is recalculated and the mean is used as the new cluster center.

6. The mural image retrieval method based on unsupervised feature engineering according to claim 1, characterized in that: A feature fuser is used to perform reverse feature fusion, including: performing relative entropy calculations on the line drawing features, original image features, and color map features corresponding to the current image, respectively, and constructing a fusion matrix based on the relative entropy calculation results; flattening the constructed fusion matrix to obtain fusion features.

7. The mural image retrieval method based on unsupervised feature engineering according to claim 6, characterized in that: The relative entropy calculation is performed on the line drawing features, original image features and color map features corresponding to the current image, including: first, calculating the relative entropy of the line drawing features, the line drawing features, the original image features and the color map features; then, calculating the relative entropy of the original image features, the line drawing features, the original image features and the color map features; then, calculating the relative entropy of the color map features, the line drawing features, the original image features and the color map features.

8. A mural image retrieval system based on unsupervised feature engineering, using the mural image retrieval method according to any one of claims 1 to 7, characterized in that: include: The mural image database construction module is configured to: obtain mural images and construct a mural image database; The feature engineering module is configured to: traverse each image in the mural image database and extract features of each image based on feature engineering to construct a feature database; wherein the feature engineering implementation process is as follows: obtain the line drawing and color map of each image, use the feature extractor to extract features from the original image, line drawing and color map corresponding to each image, and use the feature fusion device to perform reverse feature fusion; obtain the query target, and extract the target features corresponding to the query target based on feature engineering; The similarity measurement module is configured to measure the similarity between the features in the feature database and the target features in sequence, and return the search results in descending order of similarity.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the mural image retrieval method based on unsupervised feature engineering as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the mural image retrieval method based on unsupervised feature engineering as described in any one of claims 1 to 7 are implemented.

Images