Cultural relic image retrieval method, system and equipment and storage medium

Abstract

The invention discloses a cultural relic image retrieval method, system and device and a storage medium, and relates to the field of image retrieval, and the method comprises the steps: inputting a cultural relic query image into a preset target detection model, obtaining the global fusion semantic feature and attribute information of the cultural relic query image outputted by the preset target detection model, screening alternative cultural relic information on the basis of the type of the attribute information with the highest confidence score, and selecting a preset number of alternative cultural relic information from the alternative cultural relic information as a retrieval result to be output on the basis of the similarity between the cultural relic features corresponding to the cultural relic query image and the alternative cultural relic features. According to the method, the global feature of the historical relic query image and the global fusion semantic feature of each detail feature are fused, the historical relic feature for retrieval is generated based on the global fusion semantic feature, the historical relic image retrieval precision is improved, alternative historical relic feature screening is performed based on the attribute information, and the retrieval efficiency is improved. Therefore, the precision and efficiency of historical relic image retrieval are improved.

Description

Technical Field

[0001] This application relates to the field of image retrieval technology, and in particular to a method, system, device and storage medium for retrieving images of cultural relics. Background Technology

[0002] With the development of digital technology, cultural relic image retrieval technology has shown broad application prospects in fields such as digital museum construction, cultural relic protection, and education and scientific research. Existing cultural relic image retrieval technologies mostly utilize convolutional neural networks (CNNs) trained on image classification datasets (ImageNet) to extract and identify the global texture features of user-uploaded cultural relic query images, thereby displaying cultural relic information that has a high degree of matching with the global texture features.

[0003] However, existing cultural relic image retrieval only uses global texture features as retrieval elements, ignoring the recognition accuracy of local features such as the shape and microscopic details of cultural relics in the image. This leads to a decrease in the accuracy of global texture features extracted by existing cultural relic image retrieval when the texture of the cultural relic in the user-uploaded query image is complex or partially damaged, thus reducing the retrieval accuracy of cultural relic image retrieval. Summary of the Invention

[0004] In view of the above problems, this application provides a method, system, device, and storage medium for retrieving cultural relics images, in order to improve the accuracy of cultural relics image retrieval. The specific solution is as follows:

[0005] The first aspect of this application provides a method for retrieving images of cultural relics, including:

[0006] Obtain images of cultural relics for search;

[0007] The cultural relic query image is input into a preset target detection model to obtain the global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model. The global fusion semantic features are fusion features generated by the preset target detection model after fusing the global features and detailed features of the cultural relic query image. The types of attribute information include the age attribute, material attribute and pattern attribute of the cultural relic query image.

[0008] Based on the type of the attribute information with the highest confidence score, candidate cultural relic information is filtered from the cultural relic database, and the candidate cultural relic information includes candidate cultural relic characteristics;

[0009] Based at least on the preset weight values ​​corresponding to the attribute information of each type, the global fusion semantic features are weighted to obtain the cultural relic features corresponding to the cultural relic query image;

[0010] Based on the similarity between the characteristics of the cultural relic and the characteristics of the candidate cultural relic, a preset number of candidate cultural relic information are selected from each candidate cultural relic information as the retrieval result output.

[0011] In one possible implementation, the step of inputting the cultural relic query image into a preset target detection model to obtain the global fused semantic features and attribute information of the cultural relic query image output by the preset target detection model includes:

[0012] The preset target detection model is used to extract feature information from the cultural relic query image. The feature information extraction operation includes:

[0013] The preset target detection model extracts detailed features and global features from the cultural relic query image;

[0014] The preset target detection model outputs the global fusion features of the cultural relic query image based on the detailed features and the global features.

[0015] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion features to obtain the global fusion semantic features of the cultural relic query image;

[0016] The preset target detection model performs spatial transformation and channel attention processing on the global fusion features to obtain the attribute information of the cultural relic query image.

[0017] In one possible implementation, the preset target detection model outputs global fusion features of the cultural relic query image based on the detailed features and the global features, including:

[0018] The preset target detection model performs channel concatenation on each of the detailed features and the global features to obtain global concatenated features; it then performs global average pooling and global max pooling on the global concatenated features to obtain average pooling features and max pooling features; the average pooling features and max pooling features are stacked along the channel dimension, and a non-linear activation function is used to normalize the weights of each channel of the stacked features to obtain the first weight of the global features and the second weight of each of the detailed features, the sum of the first weight and the second weight being 1; based on the first weight and the second weight, the detailed features and the global features are weighted and fused to obtain the global fused features of the cultural relic query image.

[0019] In one possible implementation, the preset target detection model extracts detailed features and global features from the cultural relic query image, including:

[0020] The preset target detection model uses a cross-stage local network to mark target boxes in the cultural relic query image, and extracts the detailed features of each target box and the global features of the cultural relic query image.

[0021] In one possible implementation, the training process of the preset target detection model includes:

[0022] Acquire multiple images of cultural relics, and mark the coordinate frame and attribute information of each image;

[0023] The labeled images of the artifacts are divided into a training set and a test set;

[0024] The initial target detection model is trained using the training set and the test set to obtain the preset target detection model. The input of the preset target detection model is the cultural relic query image, and the output is the global fusion semantic features and attribute information of the cultural relic query image.

[0025] In one possible implementation, training the initial object detection model using the training set and the test set includes:

[0026] During the training process of the initial target detection model using the training set and the test set, the loss values ​​output by each loss function are obtained at each training iteration.

[0027] By using the preset weight values ​​of each loss function, the loss values ​​output by each loss function are weighted and summed to obtain the joint loss value;

[0028] The parameters of the initial target detection model are tuned using the joint loss value.

[0029] In one possible implementation, the step of weighting the globally fused semantic features based at least on preset weight values ​​corresponding to the attribute information of each of the aforementioned types to obtain the cultural relic features corresponding to the cultural relic query image includes:

[0030] Obtain the attribute features of each type of attribute information extracted by the preset target detection model;

[0031] Obtain the first product of the attribute features of the attribute information of the type and the preset weight value of the attribute information of the type; obtain the second product of the global fusion semantic feature and the preset feature weight value of the global fusion semantic feature.

[0032] The characteristics of the cultural relic are obtained by summing the first product and the second product.

[0033] A second aspect of this application provides a cultural relic image retrieval system, comprising:

[0034] The image acquisition module is used to acquire images for cultural relic queries;

[0035] The data generation module is used to input the cultural relic query image into a preset target detection model to obtain the global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model. The global fusion semantic features are fusion features generated by the preset target detection model after fusing the global features and various detailed features of the cultural relic query image. The types of attribute information include the age attribute, material attribute and pattern attribute of the cultural relic query image.

[0036] The information filtering module is used to filter candidate cultural relic information from the cultural relic database based on the type of the attribute information with the highest confidence score. The candidate cultural relic information includes the characteristics of the candidate cultural relic.

[0037] The feature generation module is used to perform weighted processing on the global fusion semantic features based at least on the preset weight values ​​corresponding to the attribute information of each type, so as to obtain the cultural relic features corresponding to the cultural relic query image.

[0038] The information retrieval module is used to select a preset number of candidate cultural relics information from each candidate cultural relics information as retrieval results based on the similarity between the characteristics of the cultural relics and the characteristics of the candidate cultural relics.

[0039] In one possible implementation, the data generation module is configured as follows:

[0040] The preset target detection model is used to extract feature information from the cultural relic query image. The feature information extraction operation includes:

[0041] The preset target detection model extracts detailed features and global features from the cultural relic query image;

[0042] The preset target detection model outputs the global fusion features of the cultural relic query image based on the detailed features and the global features.

[0043] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion features to obtain the global fusion semantic features of the cultural relic query image;

[0044] The preset target detection model performs spatial transformation and channel attention processing on the global fusion features to obtain the attribute information of the cultural relic query image.

[0045] In one possible implementation, the preset target detection model in the data generation module is configured to output the global fusion features of the cultural relic query image based on the detailed features and the global features as follows:

[0046] The preset target detection model performs channel concatenation on each of the detailed features and the global features to obtain global concatenated features; it then performs global average pooling and global max pooling on the global concatenated features to obtain average pooling features and max pooling features; the average pooling features and max pooling features are stacked along the channel dimension, and a non-linear activation function is used to normalize the weights of each channel of the stacked features to obtain the first weight of the global features and the second weight of each of the detailed features, the sum of the first weight and the second weight being 1; based on the first weight and the second weight, the detailed features and the global features are weighted and fused to obtain the global fused features of the cultural relic query image.

[0047] In one possible implementation, the preset target detection model in the data generation module is configured to extract detailed features and global features from the cultural relic query image as follows:

[0048] The preset target detection model uses a cross-stage local network to mark target boxes in the cultural relic query image, and extracts the detailed features of each target box and the global features of the cultural relic query image.

[0049] In one possible implementation, the cultural relic image retrieval system further includes a model training module, which is configured during the training of the preset target detection model as follows:

[0050] Acquire multiple images of cultural relics, and mark the coordinate frame and attribute information of each image;

[0051] The labeled images of the artifacts are divided into a training set and a test set;

[0052] The initial target detection model is trained using the training set and the test set to obtain the preset target detection model. The input of the preset target detection model is the cultural relic query image, and the output is the global fusion semantic features and attribute information of the cultural relic query image.

[0053] In one possible implementation, the model training module is configured to train the initial object detection model using the training set and the test set as follows:

[0054] During the training process of the initial target detection model using the training set and the test set, the loss values ​​output by each loss function are obtained at each training iteration.

[0055] By using the preset weight values ​​of each loss function, the loss values ​​output by each loss function are weighted and summed to obtain the joint loss value;

[0056] The parameters of the initial target detection model are tuned using the joint loss value.

[0057] In one possible implementation, the feature generation module is configured as follows:

[0058] Obtain the attribute features of each type of attribute information extracted by the preset target detection model;

[0059] Obtain the first product of the attribute features of the attribute information of the type and the preset weight value of the attribute information of the type; obtain the second product of the global fusion semantic feature and the preset feature weight value of the global fusion semantic feature.

[0060] The characteristics of the cultural relic are obtained by summing the first product and the second product.

[0061] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0062] The memory is used to store computer programs;

[0063] The processor is used to execute the computer program so that the electronic device can implement the cultural relic image retrieval method of the first aspect or any implementation thereof.

[0064] The fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the cultural relic image retrieval method described in the first aspect or any implementation thereof.

[0065] By employing the aforementioned technical solutions, this application provides a method, system, device, and storage medium for retrieving cultural relics images. It processes the obtained cultural relic query images by configuring a preset target detection model, fusing global and detailed features of the query images to generate global fused semantic features. Compared to existing technologies, this application achieves the fusion extraction of global and detailed features. Simultaneously, by configuring the preset target detection model to output attribute information of various types of cultural relic query images, and configuring the type of attribute information with the highest confidence score among each type of attribute information, candidate cultural relic information is screened from the cultural relic database, improving the efficiency of cultural relic image retrieval. Subsequently, by configuring preset weight values ​​corresponding to each type of attribute information, the global fused semantic features are weighted to obtain the cultural relic features corresponding to the query images, improving the representation accuracy of cultural relic features for the detailed features of the query images. Finally, by configuring the similarity between the cultural relic features and the candidate cultural relic features in the candidate cultural relic information, a preset number of candidate cultural relic information are selected as the retrieval results, improving the accuracy of the cultural relic image retrieval results. Therefore, this application improves the retrieval accuracy of cultural relic image retrieval technology. Attached Figure Description

[0066] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0067] Figure 1 A flowchart of a method for retrieving cultural relic images provided in this application;

[0068] Figure 2 A schematic diagram of the architecture of a preset target detection model provided in this application;

[0069] Figure 3 A block diagram of a cultural relic image retrieval system provided in this application;

[0070] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0071] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0072] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0073] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0074] The first aspect of this application provides a method for retrieving images of cultural relics, such as... Figure 1 As shown, the method for retrieving cultural relic images includes:

[0075] S101. Obtain the image of the cultural relic for query.

[0076] It should be noted that in practical applications, the aforementioned images of cultural relics for retrieval can be user-uploaded images. Because user-uploaded images are affected by factors such as data source (e.g., low-resolution images from the internet), shooting equipment (low-resolution image acquisition equipment), and lighting conditions in the shooting scene (e.g., reflections, light spots), there is a risk that the global texture features of the cultural relics in the retrieval images may be missing or damaged. This, in turn, risks a decrease in retrieval accuracy for existing technologies that rely solely on global texture features as retrieval elements.

[0077] In one possible implementation, to improve the accuracy and efficiency of subsequent retrieval, preprocessing operations can be configured on the artifact query image before performing the subsequent steps S102 to S105. Specific types of these preprocessing operations include, but are not limited to, resizing, padding, and image normalization.

[0078] S102. Input the cultural relic query image into the preset target detection model to obtain the global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model. The global fusion semantic features are the fusion features generated by the preset target detection model after fusing the global features and detailed features of the cultural relic query image. The types of attribute information include the age attribute, material attribute and pattern attribute of the cultural relic query image.

[0079] It should be noted that, in practical applications, the aforementioned preset target detection model is a structured improvement of the existing YOLO (You Only Look Once) model. Since the YOLO model has high accuracy in labeling multi-scale features, this application introduces a preset target detection model based on an improved YOLO model for feature extraction in the context of cultural relic image retrieval. This improves the accuracy of extracting globally fused semantic features and attribute information from cultural relic query images, thereby enhancing subsequent retrieval accuracy.

[0080] It should be noted that in the context of cultural relic image retrieval, since cultural relics from different eras have different materials and patterns, this application configures a preset target detection model to output attribute information including era attributes, material attributes, and pattern attributes based on the input cultural relic query image. This attribute information is then used to assist in subsequent cultural relic image retrieval, thereby improving the accuracy of cultural relic image retrieval.

[0081] It should be noted that in practical applications, the aforementioned chronological, material, and pattern attributes can be categorized based on the comprehensive cultural relic information of the application scenario (such as museums or archives). Taking museums as an example, their chronological attributes can be divided into 15 categories: Shang, Zhou, Spring and Autumn and Warring States Periods, Qin, Han, Wei, Jin, Northern and Southern Dynasties, Sui, Tang, Five Dynasties and Ten Kingdoms, Song, Yuan, Ming, Qing, Modern and Contemporary, and Others. Material attributes can be divided into 11 categories: bronze, ceramics, jade, gold and silver, wood, lacquerware, stone, glass, textiles, bamboo, wood, ivory, horn, bone, and Others. Pattern attributes can be divided into 13 categories: animal face patterns, cloud and thunder patterns, dragon and phoenix patterns, intertwined vine patterns, meander patterns, floral patterns, geometric patterns, wave patterns, animal patterns, human figures, inscriptions, auspicious patterns, and Others.

[0082] It should be noted that this application fuses the global features and detailed features of the cultural relic query image by configuring a preset target detection model, so that the global fusion semantic features used for subsequent cultural relic image retrieval carry the global information and local texture detail information of the cultural relic query image. Compared with the existing technology that only uses global texture features as retrieval elements, this application improves the retrieval accuracy of cultural relic image retrieval.

[0083] S103. Based on the type of the attribute information with the highest confidence score, filter candidate cultural relic information from the cultural relic database. The candidate cultural relic information includes the characteristics of the candidate cultural relic.

[0084] It should be noted that, in practical applications, the aforementioned confidence scores represent the fit between the query image of a cultural relic and the categories under each type of attribute information, and can be generated by the normalized exponential function (softmax) in the preset target detection model. Taking the chronological attribute as an example, the confidence score for the Spring and Autumn and Warring States periods is 50, the confidence score for the Qin Dynasty is 97, and the confidence score for the Han Dynasty is 60. The confidence scores for other chronological arc-shaped categories are below the threshold of 45. Therefore, cultural relic information corresponding to the Qin Dynasty is selected from the cultural relic database as the above-mentioned candidate cultural relic information. Since the cultural relic database stores a large amount of cultural relic information, using a feature-by-feature comparison method to determine the retrieval results would lead to a decrease in the efficiency of cultural relic image retrieval. This application, however, reduces the number of comparisons and improves the efficiency of subsequent cultural relic image retrieval by configuring the type of the attribute information with the highest confidence score to select candidate cultural relic information from the cultural relic database.

[0085] It should be noted that in practical applications, since there may be multiple confidence scores for different types of attribute information, the argmax function can be used to automatically filter the type of attribute information with the highest confidence score in order to improve filtering efficiency.

[0086] It should be noted that, in practical application scenarios, the above-mentioned cultural relics database can be generated in a manner that includes steps A1 to A4.

[0087] Step A1: Collect images of all cultural relics in the application scenario. Then trigger step A2.

[0088] Step A2: Input each artifact image sequentially into the preset target detection model of step S102 above to obtain the global fusion semantic features and attribute information corresponding to each artifact image. Then trigger step A3.

[0089] Step A3: For each cultural relic image: based at least on the preset weight values ​​corresponding to the attribute information of each type of the cultural relic image, perform weighted processing on the global fusion semantic features corresponding to the cultural relic image to obtain the candidate cultural relic features corresponding to the cultural relic image. Then trigger step A4.

[0090] Step A4: Associate the cultural relic image, the corresponding candidate cultural relic features and attribute information into cultural relic information, and store each cultural relic information into a preset database to obtain the cultural relic database.

[0091] S104. Based at least on the preset weight values ​​corresponding to the attribute information of each type, the global fusion semantic features are weighted to obtain the cultural relic features corresponding to the cultural relic query image.

[0092] It should be noted that in practical applications, attribute information can serve as a retrieval element to distinguish different cultural relics. Therefore, this application configures preset weight values ​​based at least on the attribute information of each type to perform weighted processing on the globally fused semantic features, thereby obtaining cultural relic features that represent all retrieval elements of the cultural relic query image, thus improving the efficiency and accuracy of subsequent cultural relic image retrieval results.

[0093] S105. Based on the similarity between the characteristics of the cultural relic and the characteristics of the candidate cultural relic, select a preset number of candidate cultural relic information from each candidate cultural relic information as the retrieval result output.

[0094] It should be noted that, in practical applications, steps S103 and S104 can be executed sequentially or synchronously after step S102 is completed. This application does not impose excessive limitations or elaborate on the execution steps of steps S103 and S104.

[0095] It should be noted that, in practical applications, the similarity between the cultural relic features and the candidate cultural relic features in step S105 above can be obtained through various methods. These methods include, but are not limited to, cosine similarity, Euclidean distance, and Pearson correlation coefficient. This application does not elaborate further on the specific calculation methods and processes for the aforementioned similarity.

[0096] This application fuses global and detailed features of a cultural relic query image by configuring a preset target detection model. This allows the globally fused semantic features used for subsequent cultural relic image retrieval to carry both global information and detailed local texture information of the query image. Compared to existing technologies that only use global texture features as retrieval elements, this application improves the retrieval accuracy of cultural relic image retrieval. Subsequently, by configuring the type of the attribute information with the highest confidence score, candidate cultural relic information is filtered from the cultural relic database, thereby reducing the number of comparisons and improving the efficiency of subsequent cultural relic image retrieval. Furthermore, by configuring preset weight values ​​corresponding to at least each type of attribute information, the globally fused semantic features are weighted to obtain cultural relic features representing all retrieval elements of the query image, thus improving the efficiency and accuracy of determining the subsequent cultural relic image retrieval results. Finally, by configuring the similarity between the cultural relic features and the candidate cultural relic features, a preset number of candidate cultural relic information are selected as the retrieval results output, thus determining the cultural relic image retrieval results. Therefore, this application improves the retrieval accuracy and efficiency of cultural relic image retrieval technology.

[0097] In one possible implementation, the cultural relic query image is input into a preset target detection model to obtain the globally fused semantic features and attribute information of the cultural relic query image output by the preset target detection model, including:

[0098] The pre-defined target detection model is used to extract feature information from the images of cultural relics. The feature extraction operation includes:

[0099] The preset target detection model extracts detailed features and global features from the cultural relic query image;

[0100] The preset target detection model outputs global fusion features of the cultural relic query image based on detailed features and global features;

[0101] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion features to obtain the global fusion semantic features of the cultural relic query image;

[0102] The preset target detection model performs spatial transformation and channel attention processing on the global fusion features to obtain the attribute information of the cultural relic query image.

[0103] It should be noted that in practical applications, the traditional YOLO model, primarily used for image object detection and labeling, has poor capabilities in feature fusion and extraction. Therefore, this application improves the YOLO model structure, enabling the preset object detection model to output global fusion features of the cultural relic query image based on detailed and global features. The preset object detection model is configured to perform feature extraction and high-dimensional feature mapping on the global fusion features to obtain global fusion semantic features of the cultural relic query image. Furthermore, the preset object detection model is configured to perform spatial transformation and channel attention processing on the global fusion features to obtain attribute information of the cultural relic query image. This improves the preset object detection model's ability to fuse and extract features, thereby increasing the accuracy of the obtained global fusion semantic features and attribute information.

[0104] It should be noted that, in practical application scenarios, the above-mentioned preset target detection model can be implemented by extracting features and mapping high-dimensional features from global fusion features to obtain the global fusion semantic features of the cultural relic query image, which can be:

[0105] Detailed and global features are sequentially passed through two 3x3 convolutional layers and one 1x1 convolutional layer to extract detailed semantic information at different levels, thereby improving semantic richness. The extracted results are then fed into a fully connected (FC) convolutional layer to extract global semantics. High-dimensional feature mapping is then used to aggregate high-level global and detailed semantics, improving the information richness of the final global fused semantic features and thus enhancing the extraction accuracy.

[0106] It should be noted that in practical application scenarios, the above-mentioned preset target detection model can perform spatial transformation on the global fusion features by configuring a Spatial Transformer Network (STN). The STN's Spatial Transformer Layer can be used to perform spatial transformation on the global fusion features, and an attention network can be used to adjust the proportion of semantic features and detail features of adjacent layers in the global fusion features, so as to improve the accuracy of attribute information determination.

[0107] To facilitate understanding of the process by which the aforementioned pre-defined target detection model obtains globally fused semantic features and attribute information, an explanation is provided here in conjunction with one possible implementation of this application:

[0108] like Figure 2 The diagram shows the architecture of a pre-defined object detection model. The cultural relic query image 21 is input into the cross-stage local network 221 of the pre-defined object detection model 22 to extract each bounding box 222, each detailed feature 223 of each bounding box, and the global features 224 of the cultural relic query image. Subsequently, the attention module 225 of the pre-defined object detection model 22 performs channel concatenation on the obtained detailed features 223 and global features 224 to obtain global features. Global average pooling and global max pooling are then performed on the global concatenated features to obtain average pooling features and max pooling features. The average pooling features and max pooling features are stacked along the channel dimension, and the weights of each channel of the stacked features are normalized using a non-linear activation function to obtain the first weight of the global features and the second weight of each detailed feature. The sum of the first weight and the second weight is 1. Based on the first weight and the second weight, the detailed features and global features are weighted and fused to output the global fused features 226 of the cultural relic query image. The feature extraction module 227 of the preset target detection model 22 performs feature extraction and high-dimensional feature mapping on the global fusion feature 226 to obtain the global fusion semantic feature 228 and the high-dimensional feature 229. The attribute extraction module 230 of the preset target detection model 22 performs spatial transformation and channel attention processing on the global fusion feature 226 to output the attribute information 231 of the cultural relic query image.

[0109] In one possible implementation, the pre-defined object detection model outputs global fusion features of the cultural relic query image based on detailed features and global features, including:

[0110] The pre-defined target detection model performs channel concatenation on various detail features and global features to obtain global concatenated features. Global average pooling and global max pooling are then applied to the global concatenated features to obtain average pooling features and max pooling features. These average pooling and max pooling features are stacked along the channel dimension, and a non-linear activation function is used to normalize the weights of each channel of the stacked features, obtaining the first weight of the global features and the second weight of each detail feature. The sum of the first and second weights is 1. Based on the first and second weights, the detail features and global features are weighted and fused to obtain the global fused features of the cultural relic query image.

[0111] In one possible implementation, before stacking the average pooling features and max pooling features in the channel dimension, a multilayer perceptron network with two fully connected layers can be used to perform channel dimensionality reduction and channel dimensionality increase operations on the average pooling features and max pooling features respectively, in order to avoid the risk of channel number explosion and improve the reliability of feature extraction.

[0112] It should be noted that, in practical applications, the above method of normalizing the weights of each channel of the stacked features using a non-linear activation function to obtain the first weight of the global features and the second weights of each detailed feature can be implemented as follows:

[0113] As shown in the formulas: α = Sigmoid(fc(relu(fc(concat((max_pool(concat(C5,C3')),avg_pool(concat(C5,C3'))))))))), and β = 1-α, where α is the first weight, β is the second weight, Sigmoid is the activation function, fc is the fully connected layer, C5 is the global feature, C3' is the above-mentioned detail fusion feature, concat is the concatenation operation, max_pool is the max pooling feature extraction operation, and avg_pool is the average pooling feature extraction operation.

[0114] Since the stacking process described above only applies average pooling and max pooling features along the channel dimension, but does not fuse the information from the average pooling and max pooling features within the stacked features, the aforementioned nonlinear activation function extracts inter-channel dependencies and restores channel structure by configuring the two fully connected layers. This allows the inter-channel dependencies to represent the fusion information of detailed fusion features and global features, thereby improving the accuracy of the determined first weight by enhancing information richness. Furthermore, the fully connected layer that restores the channel structure ensures that the dimension of the obtained first weight remains consistent with the dimension of the global features.

[0115] It should be noted that in the field of cultural relic image retrieval, detailed features directly affect the accuracy of retrieval results. Therefore, this application achieves a balance between global and detailed information by configuring the aforementioned first and second weights, thereby improving the perception and extraction accuracy of the preset target detection model for detailed features (such as inscriptions and textures). For example, assuming α + β > 1, the weighted global fusion features may be over-amplified, leading to inaccurate predictions by the model. If α + β < 1, it may result in the loss of detailed information or insufficient global information. Ensuring α + β = 1 can prevent this imbalance, thereby ensuring the stability and accuracy of the model during training and use.

[0116] It should be noted that, in practical application scenarios, the above-mentioned process of weighted fusion of detailed features and global features based on the first and second weights to obtain the global fusion features of the cultural relic query image can be represented by the formula: Feature=α*C5+β*C3', where Feature is the global fusion feature of the cultural relic query image.

[0117] In one possible implementation, a pre-defined object detection model extracts detailed features and global features from the cultural relic query image, including:

[0118] The pre-defined target detection model uses a cross-stage local network to mark target boxes in the cultural relic query image and extracts the detailed features of each target box and the global features of the cultural relic query image.

[0119] In one possible implementation, the Cross-Stage Partial Network (CSPNet) described above is inherited from the traditional YOLO model. After marking the target bounding boxes in the cultural relic query image, the cross-stage partial network of the aforementioned preset object detection model can also output the coordinates of the marked target bounding boxes to the human-computer interaction interface to display detailed features and improve the user experience. The coordinates of the target bounding boxes are represented as [x, y, w, h], where x is the horizontal coordinate of the center point of the target bounding box in the coordinate system of the cultural relic query image, y is the vertical coordinate of the center point of the target bounding box in the coordinate system of the cultural relic query image, w is the width of the target bounding box, and h is the height of the target bounding box.

[0120] It should be noted that, in practical applications, the process by which the cross-stage local network of the aforementioned preset target detection model marks target boxes in the cultural relic query image and extracts the detailed features of each target box and the global features of the cultural relic query image can be as follows:

[0121] The local feature layer (C3) of the cross-stage local network extracts multiple texture features from the cultural relic query image, and the global feature layer (C5) extracts the global features of the cultural relic query image. The two 1*1 convolutions and 3*3 convolutions of the cross-stage local network take each texture feature and global feature as input to determine each target box and label it. The cross-stage local network performs multi-scale dilated convolution on multiple texture features to obtain each detail feature.

[0122] In one possible implementation, since the dimension of the local feature layer (C3) is smaller than that of the global feature layer (C5), in order to reduce the difficulty for the attention module of the subsequent preset object detection model to perform channel concatenation of each detailed feature and the global feature, after obtaining each detailed feature, the dimension of the obtained detailed fused feature can be made consistent with the dimension of the global feature by channel concatenation and 1*1 convolution to increase the dimension.

[0123] In one possible implementation, the training process of the aforementioned pre-defined target detection model includes:

[0124] Acquire multiple images of cultural relics and mark the coordinate frame and attribute information of each image;

[0125] The labeled images of the cultural relics were divided into training and testing sets;

[0126] The initial object detection model is trained using the training set and the test set to obtain the preset object detection model. The input of the preset object detection model is the cultural relic query image, and the output is the global fusion semantic features and attribute information of the cultural relic query image.

[0127] In one possible implementation, the initial object detection model is trained using a training set and a test set, including:

[0128] During the training of the initial object detection model using the training and test sets, the loss values ​​output by each loss function are obtained at each training iteration.

[0129] By using the preset weight values ​​of each loss function, the loss values ​​output by each loss function are weighted and summed to obtain the joint loss value;

[0130] The parameters of the initial target detection model are tuned using the joint loss value.

[0131] It should be noted that in practical applications, the loss functions of the aforementioned preset target detection model can be of various types. One example is provided here:

[0132] Since the main function of the above object detection model is to label the target box, output global fused semantic features and output attribute information, object detection loss function, feature extraction loss function and attribute classification loss function can be configured.

[0133] The target detection loss function can use the standard Intersection over Union (IOU) loss function: L reg =1-IOU(B, B'), calculate the regression error of the bounding box detection, where L reg Let B be the target detection loss value, B be the predicted target box, B' be the ground truth target box labeled in each training data in the training set, and IOU(B, B') be the intersection-union ratio of the predicted target box and the ground truth target box.

[0134] The feature extraction loss function can use the triplet loss function: L triplet =∑max(d(a i p i )- d(a i n i )+γ,0), with feature extraction loss value L triplet Measure the similarity of features. Where, a i p is the i-th anchor point sample in the training set. i It is the i-th positive sample in the training set, n i is the i-th negative sample in the training set, d is the distance between feature vectors (such as Euclidean distance), and γ is a hyperparameter representing the minimum margin between positive and negative samples.

[0135] The attribute classification loss function can calculate an independent cross-entropy loss value for each attribute. And configure different weights. Among them, L attr The cross-entropy loss value is an attribute information value, y j It's a real tag, p j It represents the predicted probability, where M is the number of categories under the attribute. It is the weight of the attribute information of type i.

[0136] In one possible implementation, the globally fused semantic features are weighted based on preset weight values ​​corresponding to each type of attribute information to obtain the cultural relic features corresponding to the cultural relic query image, including:

[0137] Obtain attribute features of various types of attribute information extracted by the preset target detection model;

[0138] Obtain the first product of the attribute features of the above-mentioned type of attribute information and the preset weight values ​​of the above-mentioned type of attribute information, and obtain the second product of the global fusion semantic features and the preset feature weight values ​​of the global fusion semantic features.

[0139] Summing the first and second products yields the characteristics of the cultural relic.

[0140] It should be noted that, in practical applications, the preset weight values ​​of the aforementioned attribute information and the preset feature weight values ​​of the globally fused semantic features can be pre-set by researchers based on the importance of different types of attributes. This application balances the proportion of attribute information and globally fused semantic features in the obtained cultural relic features by configuring the aforementioned preset weight values ​​and preset feature weight values, and summing each first product and second product. This allows the cultural relic features to represent the global semantic information of the cultural relic query image while also highlighting the attribute information of each type, thus improving the accuracy of cultural relic feature generation.

[0141] The second aspect of this application provides a cultural relic image retrieval system, such as... Figure 3 As shown, the cultural relic image retrieval system includes:

[0142] Image acquisition module 301 is used to acquire images for cultural relic queries;

[0143] The data generation module 302 is used to input the cultural relic query image into the preset target detection model to obtain the global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model. The global fusion semantic features are fusion features generated by the preset target detection model after fusing the global features and detailed features of the cultural relic query image. The types of attribute information include the age attribute, material attribute and pattern attribute of the cultural relic query image.

[0144] The information filtering module 303 is used to filter candidate cultural relic information from the cultural relic database based on the type of the attribute information with the highest confidence score. The candidate cultural relic information includes the characteristics of the candidate cultural relic.

[0145] The feature generation module 304 is used to perform weighted processing on the global fused semantic features based on at least the preset weight values ​​corresponding to the attribute information of each type, so as to obtain the cultural relic features corresponding to the cultural relic query image.

[0146] The information retrieval module 305 is used to select a preset number of candidate cultural relics information from each candidate cultural relics information as the retrieval result output based on the similarity between the cultural relics features and the candidate cultural relics features.

[0147] In one possible implementation, the data generation module 302 is configured as follows:

[0148] The pre-defined target detection model is used to extract feature information from the images of cultural relics. The feature extraction operation includes:

[0149] The preset target detection model extracts detailed features and global features from the cultural relic query image;

[0150] The preset target detection model outputs global fusion features of the cultural relic query image based on detailed features and global features;

[0151] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion features to obtain the global fusion semantic features of the cultural relic query image;

[0152] The preset target detection model performs spatial transformation and channel attention processing on the global fusion features to obtain the attribute information of the cultural relic query image.

[0153] In one possible implementation, the preset target detection model in the data generation module 302 is set to output the global fusion features of the cultural relic query image based on various detailed features and global features:

[0154] The pre-defined target detection model performs channel concatenation on various detail features and global features to obtain global concatenated features. Global average pooling and global max pooling are then applied to the global concatenated features to obtain average pooling features and max pooling features. These average pooling and max pooling features are stacked along the channel dimension, and a non-linear activation function is used to normalize the weights of each channel of the stacked features, obtaining the first weight of the global features and the second weight of each detail feature. The sum of the first and second weights is 1. Based on the first and second weights, the detail features and global features are weighted and fused to obtain the global fused features of the cultural relic query image.

[0155] In one possible implementation, the preset target detection model in the data generation module 302 is set to extract detailed features and global features from the cultural relic query image:

[0156] The pre-defined target detection model uses a cross-stage local network to mark target boxes in the cultural relic query image and extracts the detailed features of each target box and the global features of the cultural relic query image.

[0157] In one possible implementation, the cultural relic image retrieval system provided in the second aspect of this application further includes a model training module, which is configured during the training of the preset target detection model as follows:

[0158] Acquire multiple images of cultural relics and mark the coordinate frame and attribute information of each image;

[0159] The labeled images of the cultural relics were divided into training and testing sets;

[0160] The initial object detection model is trained using the training set and the test set to obtain the preset object detection model. The input of the preset object detection model is the cultural relic query image, and the output is the global fusion semantic features and attribute information of the cultural relic query image.

[0161] In one possible implementation, the model training module described above is configured to train the initial object detection model using the training and test sets as follows:

[0162] During the training of the initial object detection model using the training and test sets, the loss values ​​output by each loss function are obtained at each training iteration.

[0163] By using the preset weight values ​​of each loss function, the loss values ​​output by each loss function are weighted and summed to obtain the joint loss value;

[0164] The parameters of the initial target detection model are tuned using the joint loss value.

[0165] In one possible implementation, the feature generation module 304 provided in the second aspect of this application is configured as follows:

[0166] Obtain attribute features of various types of attribute information extracted by the preset target detection model;

[0167] Obtain the first product of the attribute features of the type attribute information and the preset weight values ​​of the type attribute information; obtain the second product of the global fusion semantic features and the preset feature weight values ​​of the global fusion semantic features.

[0168] Summing the first and second products yields the characteristics of the cultural relic.

[0169] A third aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0170] Memory is used to store computer programs;

[0171] The processor is used to execute computer programs to enable electronic devices to implement the cultural relic image retrieval method of the first aspect or any implementation thereof described above.

[0172] The structural schematic diagram of the electronic device provided in the third aspect of this application is as follows: Figure 4 As shown. The electronic devices in the embodiments of this application may include, but are not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0173] like Figure 4As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0174] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0175] The fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the cultural relic image retrieval method described in the first aspect or any implementation thereof.

[0176] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the cultural relic image retrieval methods provided in this application.

[0177] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0179] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0180] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An antique image retrieval method characterized by comprising: The method comprises the following steps: obtaining an artifact query image; inputting the artifact query image into a preset target detection model to obtain global fusion semantic features and attribute information of the artifact query image output by the preset target detection model, wherein the global fusion semantic features are fusion features generated by the preset target detection model after fusing global features and detail features of the artifact query image, and the types of the attribute information include age attributes, material attributes and pattern attributes of the artifact query image; based on the type of the attribute information with the highest confidence score, screening candidate artifact information from an artifact database, wherein the candidate artifact information comprises candidate artifact features; based on at least the preset weight value corresponding to each type of attribute information, performing weighted processing on the global fusion semantic features to obtain artifact features corresponding to the artifact query image; based on the similarity between the artifact features and the candidate artifact features, selecting a preset number of candidate artifact information from each candidate artifact information as a retrieval result output.

2. The method according to claim 1, wherein, The method comprises the following steps: using the preset target detection model to perform feature information extraction operation on the artifact query image, wherein the feature information extraction operation comprises: the preset target detection model extracts each detail feature in the artifact query image and the global feature of the artifact query image; the preset target detection model outputs the global fusion feature of the artifact query image based on each detail feature and the global feature; the preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion feature to obtain the global fusion semantic feature of the artifact query image; the preset target detection model performs spatial conversion and channel attention processing on the global fusion feature to obtain the attribute information of the artifact query image.

3. The method according to claim 2, wherein The method comprises the following steps: the preset target detection model concatenates each detail feature and the global feature to obtain a global concatenation feature; performs global average pooling and global maximum pooling on the global concatenation feature to obtain an average pooling feature and a maximum pooling feature; stacks the average pooling feature and the maximum pooling feature in the channel dimension, and normalizes the weight of each channel of the stacked feature using a nonlinear activation function to obtain a first weight of the global feature and a second weight of each detail feature, wherein the sum of the first weight and the second weight is 1; based on the first weight and the second weight, weighted fusion is performed on each detail feature and the global feature to obtain the global fusion feature of the artifact query image.

4. The method according to claim 2, wherein The method comprises the following steps: the preset target detection model extracts each detail feature in the artifact query image and the global feature of the artifact query image, comprising: The preset target detection model uses a cross-stage local network to mark a target frame of the cultural relic query image and extracts the detail features of each target frame and the global features of the cultural relic query image.

5. The method according to claim 2, wherein The training process of the preset target detection model comprises: a plurality of cultural relic images are obtained, and each cultural relic image is marked with a cultural relic coordinate frame and attribute information; each marked cultural relic image is divided into a training set and a test set; an initial target detection model is trained using the training set and the test set to obtain the preset target detection model, wherein the input of the preset target detection model is a cultural relic query image, and the output is global fusion semantic features and attribute information of the cultural relic query image.

6. The method according to claim 5, wherein The training of the initial target detection model using the training set and the test set comprises: During the training of the initial target detection model using the training set and the test set, the loss value output by each loss function at each training time is obtained; the loss values output by each loss function are weighted and summed using preset weight values corresponding to each loss function to obtain a joint loss value; the initial target detection model is parameter-optimized using the joint loss value.

7. The method according to claim 1, wherein The weighting processing of the global fusion semantic features using the preset weight values corresponding to each type of attribute information to obtain the cultural relic features corresponding to the cultural relic query image comprises: attribute features of each type of attribute information extracted by the preset target detection model are obtained; a first product of the attribute features of each type of attribute information and a preset weight value corresponding to each type of attribute information is obtained, and a second product of the global fusion semantic features and a preset feature weight value corresponding to the global fusion semantic features is obtained; each first product and second product is summed to obtain the cultural relic features.

8. An antique image retrieval system characterized by comprising: It comprises: an image obtaining module configured to obtain a cultural relic query image; a data generating module configured to input the cultural relic query image into a preset target detection model to obtain global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model, wherein the global fusion semantic features are fusion features generated by the preset target detection model after fusing global features and detail features of the cultural relic query image, and the types of attribute information include age attributes, material attributes, and pattern attributes of the cultural relic query image; an information screening module configured to screen candidate cultural relic information from a cultural relic database based on a type of attribute information with the highest confidence score, wherein the candidate cultural relic information comprises candidate cultural relic features; a feature generating module configured to perform weighting processing on the global fusion semantic features using preset weight values corresponding to each type of attribute information to obtain cultural relic features corresponding to the cultural relic query image; an information retrieving module configured to select a preset number of candidate cultural relic information from each candidate cultural relic information as a retrieval result output based on the similarity between the cultural relic features and the candidate cultural relic features.

9. An electronic device, comprising: It comprises at least one processor and a memory connected to the processor, wherein: The memory is configured to store a computer program. The processor is configured to execute the computer program to enable the electronic device to implement the cultural relic image retrieval method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device can implement the cultural relic image retrieval method according to any one of claims 1 to 7.

Images