An image retrieval method, system, device and storage medium for cultural relics

Abstract

The application discloses an ancient cultural relic image retrieval method, system, device and storage medium, and relates to the field of image retrieval, and comprises the following steps: inputting an ancient cultural relic query image into a preset target detection model, obtaining global fusion semantic features and attribute information of the ancient cultural relic query image output by the preset target detection model, screening candidate ancient cultural relic information based on the type of attribute information with the highest confidence score, and selecting a preset number of candidate ancient cultural relic information from each candidate ancient cultural relic information as a retrieval result output based on the similarity between the ancient cultural relic features corresponding to the ancient cultural relic query image and the candidate ancient cultural relic features. The global fusion semantic features of the ancient cultural relic query image are fused with the global features and the detail features, and the ancient cultural relic features used for retrieval are generated based on the global fusion semantic features, so that the ancient cultural relic image retrieval accuracy is improved, the candidate ancient cultural relic features are screened based on the attribute information, and the retrieval efficiency is improved. Therefore, the application improves the accuracy and efficiency of the ancient cultural relic image retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image retrieval, in particular to an artifact image retrieval method, system, device and storage medium. BACKGROUND

[0002] With the development of digital technology, artifact image retrieval technology shows a wide application prospect in the fields of digital museum construction, artifact protection, education and scientific research. The existing artifact image retrieval technology is mostly to use a convolutional neural network (CNN) trained based on an image classification dataset (ImageNet) to extract and identify the global texture features of the user-uploaded artifact query image, so as to display the artifact information with higher matching degree of global texture features.

[0003] However, since the existing artifact image retrieval only uses global texture features as retrieval elements, the recognition accuracy of local features such as artifact shape and microscopic details is ignored, which leads to a decrease in the accuracy of global texture features extracted by the existing artifact image retrieval in the case that the texture of the artifact in the user-uploaded artifact query image is complex or locally damaged, thereby reducing the retrieval accuracy of artifact image retrieval. SUMMARY

[0004] In view of the above problems, the present application provides an artifact image retrieval method, system, device and storage medium to achieve the purpose of improving the artifact image retrieval accuracy. The specific scheme is as follows:

[0005] The first aspect of the present application provides an artifact image retrieval method, comprising:

[0006] obtaining an artifact query image;

[0007] inputting the artifact query image into a preset target detection model to obtain global fusion semantic features and attribute information of the artifact query image output by the preset target detection model, wherein the global fusion semantic features are fusion features generated by the preset target detection model after fusing global features and each detail feature of the artifact query image, and the types of the attribute information include the age attribute, the material attribute and the pattern attribute of the artifact query image;

[0008] based on the type of attribute information with the highest confidence score, filtering candidate artifact information from an artifact database, the candidate artifact information including candidate artifact features;

[0009] based on at least the preset weight value corresponding to each type of attribute information, performing weighted processing on the global fusion semantic features to obtain artifact features corresponding to the artifact query image;

[0010] Based on the similarity between the cultural relic feature and the candidate cultural relic feature, a preset number of candidate cultural relic information is selected from each of the candidate cultural relic information as a retrieval result output.

[0011] In a possible implementation, the inputting of the cultural relic query image into the preset target detection model to obtain the global fusion semantic feature and the attribute information of the cultural relic query image output by the preset target detection model includes:

[0012] The feature information extraction operation of the cultural relic query image by the preset target detection model includes:

[0013] The preset target detection model extracts each detail feature in the cultural relic query image and a global feature of the cultural relic query image;

[0014] The preset target detection model outputs a global fusion feature of the cultural relic query image based on each of the detail features and the global feature;

[0015] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion feature to obtain a global fusion semantic feature of the cultural relic query image;

[0016] The preset target detection model performs spatial conversion and channel attention processing on the global fusion feature to obtain the attribute information of the cultural relic query image.

[0017] In a possible implementation, the preset target detection model outputs a global fusion feature of the cultural relic query image based on each of the detail features and the global feature, including:

[0018] The preset target detection model performs channel splicing on each of the detail features and the global feature to obtain a global splicing feature; performs global average pooling and global maximum pooling on the global splicing feature respectively to obtain an average pooling feature and a maximum pooling feature; stacks the average pooling feature and the maximum pooling feature in a channel dimension, and normalizes the weight of each channel of the stacked feature by using a nonlinear activation function to obtain a first weight of the global feature and a second weight of each of the detail features, and the sum of the first weight and the second weight is 1; based on the first weight and the second weight, each of the detail features and the global feature is weighted and fused to obtain the global fusion feature of the cultural relic query image.

[0019] In a possible implementation, the preset target detection model extracts each detail feature in the cultural relic query image and a global feature of the cultural relic query image, including:

[0020] The preset target detection model uses a cross-stage local network to mark a target frame of the cultural relic query image and extracts the detail features of each target frame and global features of the cultural relic query image.

[0021] In a possible implementation, the training process of the preset target detection model includes:

[0022] A plurality of cultural relic images are obtained, and each cultural relic image is marked with a cultural relic coordinate frame and attribute information.

[0023] Each marked cultural relic image is divided into a training set and a test set.

[0024] The training set and the test set are used to train an initial target detection model to obtain the preset target detection model, wherein the input of the preset target detection model is a cultural relic query image, and the output is global fusion semantic features and attribute information of the cultural relic query image.

[0025] In a possible implementation, the training of the initial target detection model using the training set and the test set includes:

[0026] During the training of the initial target detection model using the training set and the test set, loss values output by each loss function at each training time are obtained.

[0027] Each loss value output by each loss function is weighted and summed using a preset weight value of each loss function to obtain a joint loss value.

[0028] The initial target detection model is parameter-optimized using the joint loss value.

[0029] In a possible implementation, the weighting processing of the global fusion semantic features using at least the preset weight value corresponding to the attribute information of each type to obtain cultural relic features corresponding to the cultural relic query image includes:

[0030] Attribute features of the attribute information of each type extracted by the preset target detection model are obtained.

[0031] A first product of the attribute features of the attribute information of the type and a preset weight value of the attribute information of the type is obtained, and a second product of the global fusion semantic features and a preset feature weight value of the global fusion semantic features is obtained.

[0032] Each first product and second product is summed to obtain the cultural relic features.

[0033] The second aspect of the present application provides a cultural relic image retrieval system, which includes:

[0034] an image obtaining module configured to obtain an artifact query image;

[0035] a data generating module configured to input the artifact query image into a preset target detection model to obtain global fusion semantic features and attribute information of the artifact query image output by the preset target detection model, wherein the global fusion semantic features are fusion features generated by the preset target detection model after fusing global features and various detail features of the artifact query image, and types of the attribute information include an age attribute, a material attribute, and a pattern attribute of the artifact query image;

[0036] an information screening module configured to screen candidate artifact information from an artifact database based on a type of the attribute information with the highest confidence score, the candidate artifact information including candidate artifact features;

[0037] a feature generating module configured to perform weighted processing on the global fusion semantic features based on preset weight values corresponding to the types of the attribute information to obtain artifact features corresponding to the artifact query image;

[0038] an information retrieving module configured to select a preset number of candidate artifact information as a retrieval result output from the candidate artifact information based on similarities between the artifact features and the candidate artifact features.

[0039] In a possible implementation, the data generating module is configured to:

[0040] perform a feature information extraction operation on the artifact query image by using the preset target detection model, the feature information extraction operation including:

[0041] extracting, by the preset target detection model, various detail features in the artifact query image and global features of the artifact query image;

[0042] outputting, by the preset target detection model, global fusion features of the artifact query image based on the various detail features and the global features;

[0043] performing, by the preset target detection model, feature extraction and high-dimensional feature mapping on the global fusion features to obtain global fusion semantic features of the artifact query image;

[0044] performing, by the preset target detection model, spatial conversion and channel attention processing on the global fusion features to obtain the attribute information of the artifact query image.

[0045] In a possible implementation, when outputting the global fusion features of the artifact query image based on the various detail features and the global features, the preset target detection model in the data generating module is configured to:

[0046] The preset target detection model concatenates each of the detail features and the global feature in a channel dimension to obtain a global concatenated feature; performs global average pooling and global maximum pooling on the global concatenated feature respectively to obtain an average pooled feature and a maximum pooled feature; stacks the average pooled feature and the maximum pooled feature in a channel dimension, and normalizes the weight of each channel of the stacked feature by using a nonlinear activation function to obtain a first weight of the global feature and a second weight of each of the detail features, wherein the sum of the first weight and the second weight is 1; and performs weighted fusion on each of the detail features and the global feature based on the first weight and the second weight to obtain a global fused feature of the cultural relic query image.

[0047] In a possible implementation, when extracting each detail feature in the cultural relic query image and the global feature of the cultural relic query image, the preset target detection model in the data generation module is configured to:

[0048] The preset target detection model uses a cross-stage local network to mark a target frame of the cultural relic query image, and extracts the detail feature of each target frame and the global feature of the cultural relic query image.

[0049] In a possible implementation, the cultural relic image retrieval system further includes a model training module, and the model training module is configured to, in a training process of the preset target detection model:

[0050] Obtain a plurality of cultural relic images, and mark each of the cultural relic images with a cultural relic coordinate frame and attribute information;

[0051] Divide each of the marked cultural relic images into a training set and a test set;

[0052] Train an initial target detection model by using the training set and the test set to obtain the preset target detection model, wherein the input of the preset target detection model is a cultural relic query image, and the output of the preset target detection model is a global fused semantic feature of the cultural relic query image and attribute information.

[0053] In a possible implementation, the model training module is configured to, when training the initial target detection model by using the training set and the test set:

[0054] In the process of training the initial target detection model by using the training set and the test set, obtain a loss value output by each loss function in each training;

[0055] Perform weighted summation on the loss values output by each of the loss functions by using a preset weight value of each of the loss functions to obtain a joint loss value;

[0056] The initial target detection model is parameterized using the joint loss value.

[0057] In a possible implementation, the feature generation module is configured to:

[0058] obtain attribute features of the attribute information of each type extracted by the preset target detection model;

[0059] obtain a first product of the attribute features of the attribute information of the type and a preset weight value of the attribute information of the type, and obtain a second product of the global fusion semantic features and a preset feature weight value of the global fusion semantic features;

[0060] sum the first products and the second products to obtain the cultural relic features.

[0061] The third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0062] The memory is configured to store a computer program;

[0063] The processor is configured to execute the computer program, so that the electronic device can implement the cultural relic image retrieval method of the first aspect or any implementation manner of the first aspect.

[0064] The fourth aspect of the present application provides a computer storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the cultural relic image retrieval method of the first aspect or any implementation manner of the first aspect.

[0065] By the technical scheme, the application provides an ancient cultural relic image retrieval method, system, device and storage medium. The global feature and the detail feature of the ancient cultural relic query image are fused by configuring a preset target detection model to process the obtained ancient cultural relic query image, and the global fusion semantic feature of the ancient cultural relic query image is generated, so that the fusion extraction of the global feature and the detail feature is realized compared with the prior art. At the same time, the attribute information of each type of the ancient cultural relic query image is output by configuring the preset target detection model, and the type of the attribute information with the highest confidence score is configured based on each type of attribute information, the candidate ancient cultural relic information is filtered from the ancient cultural relic database, and the ancient cultural relic image retrieval efficiency is improved. Then, the global fusion semantic feature is weighted and processed by configuring the preset weight value corresponding to each type of attribute information, the ancient cultural relic feature corresponding to the ancient cultural relic query image is obtained, and the representation accuracy of the ancient cultural relic feature to the detail feature of the ancient cultural relic query image is improved. Finally, the preset number of candidate ancient cultural relic information is selected as the retrieval result output from each candidate ancient cultural relic information by configuring the similarity between the ancient cultural relic feature and the candidate ancient cultural relic feature in the candidate ancient cultural relic information, and the accuracy of the ancient cultural relic image retrieval result is improved. It can be seen that the application improves the retrieval accuracy of the ancient cultural relic image retrieval technology. BRIEF DESCRIPTION OF DRAWINGS

[0066] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0067] Figure 1 A flowchart of an ancient cultural relic image retrieval method provided by the application;

[0068] Figure 2 An architecture diagram of a preset target detection model provided by the application;

[0069] Figure 3 A block diagram of an ancient cultural relic image retrieval system provided by the application;

[0070] Figure 4 A structural diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0071] The embodiments of the application are described below with reference to the accompanying drawings. The terms used in the embodiment part of the application are only used to explain the specific embodiments of the application, and are not intended to limit the application.

[0072] The embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art can know that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0073] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or equipment containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0074] The first aspect of the present application provides an artifact image retrieval method, as shown in the figure, the artifact image retrieval method comprises: Figure 1 As shown in the figure, the artifact image retrieval method comprises:

[0075] S101, obtaining an artifact query image.

[0076] It should be noted that in actual application scenarios, the above-mentioned artifact query image can be an artifact image uploaded by a user. Due to the influence of data sources (such as network low-resolution images), shooting devices (low-resolution image acquisition devices), shooting scene light (such as reflection, light spot, etc.), and other factors, there is a risk of missing or damaging the global texture features of the artifacts in the artifact query image, which in turn leads to the risk of reduced retrieval accuracy of the existing technology that relies only on global texture features as retrieval elements.

[0077] In one possible implementation, in order to improve the subsequent retrieval accuracy and efficiency, a pre-processing operation can be configured on the artifact query image before the subsequent steps S102 to S105 are executed. The specific types of the above-mentioned pre-processing operation include but are not limited to: resizing (Resize), padding (Padding), image normalization (Image Normalization), etc.

[0078] S102, inputting the artifact query image into a preset target detection model to obtain global fusion semantic features and attribute information of the artifact query image output by the preset target detection model, wherein the global fusion semantic features are fusion features generated by the preset target detection model after fusing the global features and each detail feature of the artifact query image, and the types of the attribute information include the age attribute, the material attribute and the pattern attribute of the artifact query image.

[0079] It should be noted that in the actual application scenario, the above preset target detection model is obtained by structurally improving the existing YOLO (You Only Look Once) model. Since the YOLO model has high labeling accuracy for multi-scale features, the preset target detection model improved based on the YOLO model is introduced for feature extraction in the cultural relic image retrieval scene, which improves the extraction accuracy of the global fusion semantic features and attribute information of the cultural relic query image, and further improves the subsequent retrieval accuracy.

[0080] It should be noted that in the cultural relic image retrieval scene, since the materials and patterns of cultural relics of different eras are different, the preset target detection model is configured to output attribute information including era attribute, material attribute and pattern attribute based on the input cultural relic query image, so as to assist the subsequent cultural relic image retrieval by using the attribute information to improve the cultural relic image retrieval accuracy.

[0081] It should be noted that in the actual application scenario, the above era attribute, material attribute and pattern attribute can be divided based on the full cultural relic information of the application scene (such as a museum, an archive). Taking a museum as an example, the era attribute can be divided into 15 categories such as Shang, Zhou, Spring and Autumn Period, Warring States Period, Qin, Han, Wei, Jin, Southern and Northern Dynasties, Sui, Tang, Five Dynasties and Ten Kingdoms, Song, Yuan, Ming, Qing, and Modern Times, and others. The material attribute can be divided into 11 categories such as bronze, ceramics, jade, gold and silver, wooden ware, lacquerware, stone, glass, fabric, bamboo and wood, and others. The pattern attribute can be divided into 13 categories such as beast face pattern, cloud and thunder pattern, dragon and phoenix pattern, winding branch pattern, spiral pattern, floral pattern, geometric pattern, sea water pattern, animal pattern, human pattern, inscription, auspicious pattern, and others.

[0082] It should be noted that the preset target detection model is configured to fuse the global features and detailed features of the cultural relic query image, so that the global fusion semantic features used for subsequent cultural relic image retrieval carry the global information and local texture detail information of the cultural relic query image, so that the present application improves the retrieval accuracy of cultural relic image retrieval compared with the prior art which only uses global texture features as retrieval elements.

[0083] S103, based on the type of the attribute information with the highest confidence score, filtering candidate cultural relic information from the cultural relic database, the candidate cultural relic information including candidate cultural relic features.

[0084] It should be noted that in the actual application scenario, the confidence score represents the adaptability of the cultural relic query image to the category under each type of attribute information, which can be generated by the normalization exponential function (softmax) in the preset target detection model. Taking the dynasty attribute as an example, the confidence score corresponding to the Spring and Autumn Period is 50, the confidence score corresponding to the Qin Dynasty is 97, the confidence score corresponding to the Han Dynasty is 60, and the confidence scores of the other dynasty categories are lower than the threshold value 45. Then, the cultural relic information corresponding to the Qin Dynasty is selected from the cultural relic database as the above-mentioned candidate cultural relic information. Since a large amount of cultural relic information is stored in the cultural relic database, if a one-by-one feature comparison method is used to determine the retrieval result, the cultural relic image retrieval efficiency will be reduced. The present application selects candidate cultural relic information from the cultural relic database by configuring the type of the attribute information with the highest confidence score, thereby reducing the number of comparisons and improving the subsequent cultural relic image retrieval efficiency.

[0085] It should be noted that in the actual application scenario, since there can be multiple confidence scores for different types of attribute information, to improve the screening efficiency, the type of attribute information with the highest confidence score can be automatically selected by the argmax function.

[0086] It should be noted that in the actual application scenario, the generation method of the above-mentioned cultural relic database can include the following steps A1 to A4.

[0087] Step A1, collect all cultural relic images of cultural relics in the application scenario. And trigger step A2.

[0088] Step A2, input each cultural relic image into the preset target detection model of step S102 in sequence, obtain the global fusion semantic feature and attribute information corresponding to each cultural relic image. And trigger step A3.

[0089] Step A3, for each cultural relic image: at least based on the preset weight value corresponding to the attribute information of each type of the cultural relic image, the global fusion semantic feature corresponding to the cultural relic image is weighted and processed to obtain the candidate cultural relic feature corresponding to the cultural relic image. And trigger step A4.

[0090] Step A4, associate the cultural relic image, the candidate cultural relic feature and the attribute information corresponding to the cultural relic image as cultural relic information, and store each cultural relic information in a preset database to obtain a cultural relic database.

[0091] S104, at least based on the preset weight value corresponding to each type of attribute information, the global fusion semantic feature is weighted and processed to obtain the cultural relic feature corresponding to the cultural relic query image.

[0092] It should be noted that in actual application scenarios, since the attribute information can be used as a retrieval element to distinguish different cultural relics. Therefore, the present application performs weighted processing on the global fusion semantic feature based on at least the preset weight value corresponding to each type of attribute information, thereby obtaining a cultural relic feature representing the full retrieval element of the cultural relic query image, thereby improving the determination efficiency and accuracy of the subsequent cultural relic image retrieval result.

[0093] In step S105, based on the similarity between the cultural relic feature and the candidate cultural relic feature, a preset number of candidate cultural relic information is selected from each candidate cultural relic information as a retrieval result output.

[0094] It should be noted that in actual application scenarios, the above steps S103 and S104 can be executed in sequence or synchronously after step S102 is executed. The present application does not make excessive limitation and elaboration on the execution steps of the above steps S103 and S104.

[0095] It should be noted that in actual application scenarios, the similarity between the cultural relic feature and the candidate cultural relic feature in the above step S105 can be obtained in various ways. The ways to obtain the above similarity include but are not limited to: cosine similarity (Cosine Similarity), Euclidean distance (Euclidean Distance), Pearson correlation coefficient (Pearson Correlation Coefficient), etc. The present application does not make excessive limitation and elaboration on the specific calculation method and calculation process of the above similarity.

[0096] The present application configures a preset target detection model to fuse the global feature and the detail feature of the cultural relic query image, so that the global fusion semantic feature used for subsequent cultural relic image retrieval carries the global information and local texture detail information of the cultural relic query image, so that the present application improves the retrieval accuracy of cultural relic image retrieval compared with the prior art which only uses global texture feature as a retrieval element. Subsequently, by configuring the type based on the attribute information with the highest confidence score, the candidate cultural relic information is filtered from the cultural relic database, thereby reducing the comparison quantity and improving the subsequent cultural relic image retrieval efficiency. And, by configuring at least the preset weight value corresponding to each type of attribute information, the global fusion semantic feature is weighted processed, thereby obtaining a cultural relic feature representing the full retrieval element of the cultural relic query image, thereby improving the determination efficiency and accuracy of the subsequent cultural relic image retrieval result. Finally, by configuring the similarity between the cultural relic feature and the candidate cultural relic feature, a preset number of candidate cultural relic information is selected from each candidate cultural relic information as a retrieval result output, realizing the determination of the cultural relic image retrieval result. It can be seen that the present application improves the retrieval accuracy and efficiency of the cultural relic image retrieval technology.

[0097] In a possible implementation, the cultural relic query image is input into a preset target detection model, global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model are obtained, including:

[0098] The preset target detection model is used for feature information extraction operation on the cultural relic query image, and the feature information extraction operation includes:

[0099] The preset target detection model extracts each detail feature in the cultural relic query image and the global feature of the cultural relic query image;

[0100] The preset target detection model outputs the global fusion feature of the cultural relic query image based on each detail feature and the global feature;

[0101] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion feature to obtain the global fusion semantic feature of the cultural relic query image;

[0102] The preset target detection model performs spatial conversion and channel attention processing on the global fusion feature to obtain the attribute information of the cultural relic query image.

[0103] It should be noted that in actual application scenarios, since the traditional YOLO model is mainly used for image target detection and labeling, its feature fusion and extraction capability is poor. Therefore, the YOLO model structure is improved in the present application, so that the preset target detection model realizes output of the global fusion feature of the cultural relic query image based on each detail feature and the global feature, the preset target detection model is configured to perform feature extraction and high-dimensional feature mapping on the global fusion feature to obtain the global fusion semantic feature of the cultural relic query image, and the preset target detection model is configured to perform spatial conversion and channel attention processing on the global fusion feature to obtain the attribute information of the cultural relic query image, thereby improving the feature fusion and extraction capability of the preset target detection model, and further improving the precision of the obtained global fusion semantic feature and attribute information.

[0104] It should be noted that in actual application scenarios, the above implementation of the preset target detection model performing feature extraction and high-dimensional feature mapping on the global fusion feature to obtain the global fusion semantic feature of the cultural relic query image can be:

[0105] The detail features and the global features are sequentially passed through two 3*3 convolution layers and one 1*1 convolution layer arranged in series to extract different levels of detail semantic information and improve semantic richness. Then, the extraction result is input into an FC convolution layer to extract global semantics. Through high-dimensional feature mapping, high-level global semantics and detail semantics are aggregated, the information richness of the finally obtained global fusion semantic feature is improved, and the extraction precision of the global fusion semantic feature is further improved.

[0106] It should be noted that in an actual application scenario, the above-mentioned preset target detection model performs spatial conversion on the global fusion feature, which can be realized by configuring a spatial transformation network (STN), using the spatial transformer layer (STN) of the STN to perform spatial conversion on the global fusion feature, and using the attention network to adjust the proportion of semantic features and detail features of adjacent layers in the global fusion feature, so as to improve the determination accuracy of attribute information.

[0107] In order to facilitate the understanding of the process of obtaining the global fusion semantic feature and the attribute information of the above-mentioned preset target detection model, a possible implementation of the present application is described as follows:

[0108] As shown in Figure 2 , it is an architecture diagram of a preset target detection model. The cultural relic query image 21 is input into the cross-stage local network 221 of the preset target detection model 22 to extract each target frame 222, each detail feature 223 of each target frame, and the global feature 224 of the cultural relic query image. Subsequently, the attention module 225 of the preset target detection model 22 performs channel splicing on the obtained each detail feature 223 and global feature 224 to obtain the global splicing feature, and performs global average pooling and global maximum pooling on the global splicing feature respectively to obtain the average pooling feature and the maximum pooling feature; the average pooling feature and the maximum pooling feature are stacked in the channel dimension, and the weight of each channel of the stacked feature is normalized by using a nonlinear activation function to obtain the first weight of the global feature and the second weight of each detail feature, and the sum of the first weight and the second weight is 1; based on the first weight and the second weight, the each detail feature and the global feature are weighted and fused to output the global fusion feature 226 of the cultural relic query image. The feature extraction module 227 of the preset target detection model 22 performs feature extraction and high-dimensional feature mapping on the global fusion feature 226 to obtain the global fusion semantic feature 228 and the high-dimensional feature 229. The attribute extraction module 230 of the preset target detection model 22 performs spatial conversion and channel attention processing on the global fusion feature 226 to output the attribute information 231 of the cultural relic query image.

[0109] In a possible implementation, the preset target detection model outputs the global fusion feature of the cultural relic query image based on each detail feature and global feature, including:

[0110] The preset target detection model performs channel splicing on each detail feature and the global feature to obtain a global spliced feature; performs global average pooling and global maximum pooling on the global spliced feature respectively to obtain an average pooled feature and a maximum pooled feature; stacks the average pooled feature and the maximum pooled feature in a channel dimension, and utilizes a nonlinear activation function to normalize the weight of each channel of the stacked feature, to obtain a first weight of the global feature and a second weight of each detail feature, and the sum of the first weight and the second weight is 1; based on the first weight and the second weight, each detail feature and the global feature are weighted and fused to obtain a global fusion feature of the cultural relic query image.

[0111] In a possible implementation, before stacking the average pooled feature and the maximum pooled feature in the channel dimension, a multilayer perceptron network including two fully connected layers can also be utilized to sequentially perform a channel dimension reduction operation and a channel dimension increase operation on the average pooled feature and the maximum pooled feature respectively, to avoid the risk of channel number explosion and improve the reliability of feature extraction.

[0112] It should be noted that in actual application scenarios, the implementation of utilizing the nonlinear activation function to normalize the weight of each channel of the stacked feature to obtain the first weight of the global feature and the second weight of each detail feature can be:

[0113] As shown in the formula: a = Sigmoid(fc (relu (fc (concat ((max_pool (concat (C5, C3'))), avg_pool (concat (C5, C3'))))) ) ) and the formula: b = 1-a, where a is the first weight, b is the second weight, Sigmoid is the activation function, fc is the fully connected layer, C5 is the global feature, C3' is the above-mentioned detail fusion feature, concat is the splicing operation, max_pool is the maximum pooled feature extraction operation, and avg_pool is the average pooled feature extraction operation.

[0114] Since the above stacking process is only on the average pooled feature and the maximum pooled feature in the channel dimension, but the information of the average pooled feature and the maximum pooled feature in the stacked feature is not fused. Therefore, the above nonlinear activation function extracts the inter-channel dependency and restores the channel structure by configuring the above two fc fully connected layers, so as to utilize the inter-channel dependency to represent the fusion information of the detail fusion feature and the global feature, to improve the accuracy of the determined first weight in the manner of improving the information richness, and utilize the fully connected layer for restoring the channel structure to make the dimension of the obtained first weight consistent with the dimension of the global feature.

[0115] It should be noted that in the field of cultural relic image retrieval, detailed features directly affect the accuracy of the retrieval results. Therefore, by configuring the first weight and the second weight, the balance of global information and detailed information is achieved, and the perception accuracy and extraction accuracy of the preset target detection model for detailed features (such as inscriptions and textures) are improved. For example, assuming that α + β > 1, the weighted global fusion features may be over-amplified, resulting in inaccurate predictions by the model. If α + β < 1, it may result in loss of detailed information or insufficient global information. Ensuring that α + β = 1 can prevent such imbalance, thereby ensuring the stability and accuracy during model training and use.

[0116] It should be noted that in actual application scenarios, the implementation process of the above-mentioned weighting fusion of each detailed feature and global feature based on the first weight and the second weight to obtain the global fusion feature of the cultural relic query image can be as shown in the formula: Feature = α * C5 + β * C3', wherein Feature is the global fusion feature of the cultural relic query image.

[0117] In one possible implementation, the preset target detection model extracts each detailed feature in the cultural relic query image and the global feature of the cultural relic query image, including:

[0118] The preset target detection model uses the cross-stage local network to mark the target frame of the cultural relic query image and extracts the detailed features of each target frame and the global feature of the cultural relic query image.

[0119] In one possible implementation, the above-mentioned cross-stage local network (Cross Stage Partial Network, CSPNet) is inherited from the traditional YOLO model. After the cross-stage local network of the above-mentioned preset target detection model marks the target frame of the cultural relic query image, the coordinates of the marked target frame can be output to the human-computer interaction interface to frame and display the detailed features, improving the user experience. The coordinates of the target frame are represented as [x, y, w, h], x is the horizontal coordinate of the center point of the target frame in the coordinate system of the cultural relic query image, y is the vertical coordinate of the center point of the target frame in the coordinate system of the cultural relic query image, w is the width of the target frame, and h is the height of the target frame.

[0120] It should be noted that in actual application scenarios, the cross-stage local network of the above-mentioned preset target detection model marks the target frame of the cultural relic query image and extracts the detailed features of each target frame and the global feature of the cultural relic query image. The implementation process can be:

[0121] The cross-stage local network extracts multiple texture features of the cultural relic query image through a local feature layer (C3), and extracts global features of the cultural relic query image through a global feature layer (C5); the cross-stage local network determines each target frame and labels through two 1*1 convolution and 3*3 convolution in series with each texture feature and global feature as input; and the cross-stage local network obtains each detail feature through multi-scale atrous convolution on the multiple texture features.

[0122] In a possible implementation, since the dimension of the local feature layer (C3) is smaller than the dimension of the global feature layer (C5), in order to reduce the difficulty of the attention module of the subsequent preset target detection model in channel splicing each detail feature and global feature, after obtaining each detail feature, the dimension of the obtained detail fusion feature can be fused through channel splicing and 1*1 convolution to make the dimension of the obtained detail fusion feature consistent with the dimension of the global feature.

[0123] In a possible implementation, the training process of the preset target detection model includes:

[0124] Obtaining multiple cultural relic images and labeling each cultural relic image with cultural relic coordinate frames and attribute information;

[0125] Dividing each labeled cultural relic image into a training set and a test set;

[0126] Training the initial target detection model using the training set and the test set to obtain the preset target detection model, wherein the input of the preset target detection model is a cultural relic query image, and the output is a global fusion semantic feature and attribute information of the cultural relic query image.

[0127] In a possible implementation, training the initial target detection model using the training set and the test set includes:

[0128] During the training of the initial target detection model using the training set and the test set, obtaining loss values output by each loss function at each training time;

[0129] Weighted summing the loss values output by each loss function using a preset weight value of each loss function to obtain a joint loss value;

[0130] Parameter tuning of the initial target detection model using the joint loss value.

[0131] It should be noted that in actual application scenarios, the types of the loss functions of the preset target detection model can be various, and one of them is provided as an example herein:

[0132] Since the main function of the above-mentioned target detection model is to label target frames, output global fusion semantic features, and output attribute information, the target detection loss function, the feature extraction loss function, and the attribute classification loss function can be configured.

[0133] The target detection loss function can use the standard Intersection over Union (IOU) loss function: L reg =1-IOU(B, B'), calculate the regression error of the bounding box detection, where L reg Let B be the target detection loss value, B be the predicted target box, B' be the ground truth target box labeled in each training data in the training set, and IOU(B, B') be the intersection-union ratio of the predicted target box and the ground truth target box.

[0134] The feature extraction loss function can use the triplet loss function: L triplet =∑max(d(a i p i )- d(a i n i )+γ,0), with feature extraction loss value L triplet Measure the similarity of features. Where, a i p is the i-th anchor point sample in the training set. i It is the i-th positive sample in the training set, n i is the i-th negative sample in the training set, d is the distance between feature vectors (such as Euclidean distance), and γ is a hyperparameter representing the minimum margin between positive and negative samples.

[0135] The attribute classification loss function can calculate an independent cross-entropy loss value for each attribute. And configure different weights. Among them, L attr The cross-entropy loss value is an attribute information value, y j It's a real tag, p j It represents the predicted probability, where M is the number of categories under the attribute. It is the weight of the attribute information of type i.

[0136] In one possible implementation, the globally fused semantic features are weighted based on preset weight values ​​corresponding to each type of attribute information to obtain the cultural relic features corresponding to the cultural relic query image, including:

[0137] Obtain attribute features of various types of attribute information extracted by the preset target detection model;

[0138] Obtain the first product of the attribute features of the above-mentioned type of attribute information and the preset weight values ​​of the above-mentioned type of attribute information, and obtain the second product of the global fusion semantic features and the preset feature weight values ​​of the global fusion semantic features;

[0139] Summing the first and second products yields the characteristics of the cultural relic.

[0140] It should be noted that in actual application scenarios, the preset weight values of the attribute information of the above type and the preset feature weight values of the global fusion semantic feature can be weight values pre-set by R&D personnel based on the importance of different types of attributes. The preset weight values and the preset feature weight values are configured, and the first product and the second product are summed, so as to balance the proportion of the attribute information and the global fusion semantic feature in the obtained cultural relic feature, so that the cultural relic feature not only represents the global semantic information of the cultural relic query image, but also highlights each type of attribute information, thereby improving the generation accuracy of the cultural relic feature.

[0141] The second aspect of the present application provides a cultural relic image retrieval system, as shown in the following figure: Figure 3 The cultural relic image retrieval system includes:

[0142] An image obtaining module 301 is configured to obtain a cultural relic query image.

[0143] A data generating module 302 is configured to input the cultural relic query image into a preset target detection model to obtain global fusion semantic features and attribute information of the cultural relic query image output by the preset target detection model, wherein the global fusion semantic features are fusion features generated by the preset target detection model after fusing the global features and the detail features of the cultural relic query image, and the types of the attribute information include the age attribute, the material attribute and the pattern attribute of the cultural relic query image.

[0144] An information screening module 303 is configured to screen candidate cultural relic information from a cultural relic database based on the type of the attribute information with the highest confidence score, and the candidate cultural relic information includes candidate cultural relic features.

[0145] A feature generating module 304 is configured to perform weighted processing on the global fusion semantic features based on the preset weight values corresponding to each type of attribute information to obtain a cultural relic feature corresponding to the cultural relic query image.

[0146] An information retrieval module 305 is configured to select a preset number of candidate cultural relic information as a retrieval result output from each candidate cultural relic information based on the similarity between the cultural relic feature and the candidate cultural relic feature.

[0147] In one possible implementation, the data generating module 302 is configured to:

[0148] perform a feature information extraction operation on the cultural relic query image by using the preset target detection model, and the feature information extraction operation includes:

[0149] extracting the global features of the cultural relic query image and the detail features in the cultural relic query image by using the preset target detection model.

[0150] The preset target detection model outputs global fusion features of the cultural relic query image based on the detail features and the global feature.

[0151] The preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion features to obtain global fusion semantic features of the cultural relic query image.

[0152] The preset target detection model performs spatial conversion and channel attention processing on the global fusion features to obtain attribute information of the cultural relic query image.

[0153] In one possible implementation, when the preset target detection model in the data generation module 302 outputs global fusion features of the cultural relic query image based on the detail features and the global feature, it is set to:

[0154] The preset target detection model performs channel concatenation on the detail features and the global feature to obtain global concatenation features; performs global average pooling and global maximum pooling on the global concatenation features respectively to obtain average pooling features and maximum pooling features; stacks the average pooling features and the maximum pooling features in the channel dimension, and normalizes the weight of each channel of the stacked features using a nonlinear activation function to obtain a first weight of the global feature and a second weight of each detail feature, and the sum of the first weight and the second weight is 1; based on the first weight and the second weight, the detail features and the global feature are weighted and fused to obtain the global fusion features of the cultural relic query image.

[0155] In one possible implementation, when the preset target detection model in the data generation module 302 extracts the detail features of each target frame in the cultural relic query image and the global feature of the cultural relic query image, it is set to:

[0156] The preset target detection model uses the cross-stage local network to mark the target frame of the cultural relic query image, and extracts the detail features of each target frame and the global feature of the cultural relic query image.

[0157] In one possible implementation, the cultural relic image retrieval system provided in the second aspect of the present application further comprises a model training module, and the model training module is set to:

[0158] Obtain a plurality of cultural relic images, and mark the cultural relic coordinate frame and the attribute information of each cultural relic image;

[0159] Divide the marked cultural relic images into a training set and a test set;

[0160] Train the initial target detection model using the training set and the test set to obtain the preset target detection model, and the input of the preset target detection model is the cultural relic query image, and the output is the global fusion semantic features and the attribute information of the cultural relic query image.

[0161] In a possible implementation, the model training module is configured to, when training the initial target detection model by using the training set and the test set:

[0162] In the process of training the initial target detection model by using the training set and the test set, loss values output by the loss functions in each training are obtained.

[0163] The loss values output by the loss functions are weighted and summed by using preset weight values of the loss functions respectively, to obtain a joint loss value.

[0164] The initial target detection model is parameter-optimized by using the joint loss value.

[0165] In a possible implementation, the feature generation module 304 provided in the second aspect of the present application is configured to:

[0166] obtain attribute features of the attribute information of each type extracted by the preset target detection model;

[0167] obtain a first product of the attribute features of the attribute information of each type and a preset weight value of the attribute information of each type, and obtain a second product of the global fusion semantic feature and a preset feature weight value of the global fusion semantic feature;

[0168] sum the first products and the second products to obtain the cultural relic feature.

[0169] The third aspect of the present application provides an electronic device, including at least one processor and a memory connected with the processor, wherein:

[0170] The memory is configured to store a computer program;

[0171] The processor is configured to execute the computer program, so that the electronic device can implement the cultural relic image retrieval method of the first aspect or any implementation manner of the first aspect.

[0172] The structural diagram of the electronic device provided in the third aspect of the present application is shown in Figure 4 The electronic device in the embodiment of the present application can include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers and the like. Figure 4 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0173] As Figure 4As shown, the electronic device can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. In a state in which the electronic device is powered on, various programs and data required for operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0174] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a memory card, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 The electronic device is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0175] The fourth aspect of the present application provides a computer storage medium, the storage medium carrying one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the relic image retrieval method of the first aspect or any implementation manner of the first aspect.

[0176] The embodiment of the present application also provides a computer program product including computer readable instructions, when the computer readable instructions run on an electronic device, the electronic device implements any one of the relic image retrieval methods provided by the embodiments of the present application.

[0177] In addition, it should be noted that the device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines.

[0178] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.

[0179] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially.

[0180] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as training device, data center, etc. integrated with one or more available media sets. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD) or semiconductor medium (such as solid state disk (SSD)) etc.

Claims

1. An antique image retrieval method characterized by comprising: The method comprises the following steps: obtaining an artifact query image; inputting the artifact query image into a preset target detection model to obtain global fusion semantic features and attribute information of the artifact query image output by the preset target detection model, wherein the global fusion semantic features are obtained by feature extraction and high-dimensional feature mapping on global fusion features generated by the preset target detection model after fusing global features and detail features of the artifact query image, the attribute information is generated based on the global fusion features by the preset target detection model, and types of the attribute information include age attribute, material attribute and pattern attribute of the artifact query image; based on the type of the attribute information with the highest confidence score, screening candidate artifact information from an artifact database, wherein the candidate artifact information comprises candidate artifact features; based on preset weight values of the attribute information of each type and a preset feature weight value of the global fusion semantic features, performing weighted processing on attribute features of the attribute information and the global fusion semantic features to obtain artifact features corresponding to the artifact query image; based on the similarity between the artifact features and the candidate artifact features, selecting a preset number of candidate artifact information from each candidate artifact information as a retrieval result output.

2. The method according to claim 1, wherein, The method comprises the following steps: performing feature information extraction operation on the artifact query image by using the preset target detection model, wherein the feature information extraction operation comprises: the preset target detection model extracts each detail feature in the artifact query image and global features of the artifact query image; the preset target detection model outputs global fusion features of the artifact query image based on each detail feature and the global features; the preset target detection model performs feature extraction and high-dimensional feature mapping on the global fusion features to obtain global fusion semantic features of the artifact query image; the preset target detection model performs spatial conversion and channel attention processing on the global fusion features to obtain the attribute information of the artifact query image.

3. The method according to claim 2, wherein The method comprises the following steps: the preset target detection model performs channel splicing on each detail feature and the global features to obtain global splicing features; performs global average pooling and global maximum pooling on the global splicing features respectively to obtain average pooling features and maximum pooling features; stacks the average pooling features and the maximum pooling features in the channel dimension, and normalizes the weight of each channel of the stacked features by using a nonlinear activation function to obtain a first weight of the global features and a second weight of each detail feature, wherein the sum of the first weight and the second weight is 1; based on the first weight and the second weight, weighted fusion is performed on each detail feature and the global features to obtain global fusion features of the artifact query image.

4. The method according to claim 2, wherein The preset target detection model extracts each detail feature in the cultural relic query image and a global feature of the cultural relic query image, including: The preset target detection model uses a cross-stage local network to mark a target frame of the cultural relic query image and extracts the detail feature of each target frame and the global feature of the cultural relic query image.

5. The method according to claim 2, wherein The training process of the preset target detection model includes: Obtaining a plurality of cultural relic images, and marking each cultural relic image with a cultural relic coordinate frame and attribute information; Dividing each marked cultural relic image into a training set and a test set; Training an initial target detection model using the training set and the test set to obtain the preset target detection model, wherein the input of the preset target detection model is a cultural relic query image, and the output is a global fusion semantic feature and attribute information of the cultural relic query image.

6. The method according to claim 5, wherein The training of the initial target detection model using the training set and the test set includes: During the training of the initial target detection model using the training set and the test set, obtaining a loss value output by each loss function at each training time; Using a preset weight value of each loss function to weight and sum the loss values output by each loss function to obtain a joint loss value; Using the joint loss value to optimize the parameters of the initial target detection model.

7. The method according to claim 1, wherein The preset weight value of the attribute information of each type and the preset feature weight value of the global fusion semantic feature are used to weight process the attribute feature of the attribute information and the global fusion semantic feature to obtain the cultural relic feature corresponding to the cultural relic query image, including: Obtaining the attribute feature of each type of attribute information extracted by the preset target detection model; Obtaining a first product of the attribute feature of the attribute information of the type and a preset weight value of the attribute information of the type, and a second product of the global fusion semantic feature and a preset feature weight value of the global fusion semantic feature; Summing each first product and second product to obtain the cultural relic feature.

8. An antique image retrieval system characterized by comprising: It includes: An image obtaining module is configured to obtain a cultural relic query image; A data generating module is configured to input the cultural relic query image into a preset target detection model to obtain a global fusion semantic feature and attribute information of the cultural relic query image output by the preset target detection model, wherein the global fusion semantic feature is obtained by feature extraction and high-dimensional feature mapping on a global fusion feature generated by the preset target detection model after fusing global features and detail features of the cultural relic query image, the attribute information is generated based on the global fusion feature by the preset target detection model, and the types of the attribute information include the age attribute, the material attribute, and the pattern attribute of the cultural relic query image; An information screening module is configured to screen candidate cultural relic information from a cultural relic database based on a type of attribute information with the highest confidence score, wherein the candidate cultural relic information includes candidate cultural relic features. The feature generation module is configured to perform weighted processing on the attribute features of the attribute information and the global fused semantic features based on preset weight values of the attribute information of each type and preset feature weight values of the global fused semantic features, to obtain cultural relic features corresponding to the cultural relic query image; The information retrieval module is configured to select a preset number of candidate cultural relic information from the candidate cultural relic information as a retrieval result output based on similarities between the cultural relic features and the candidate cultural relic features.

9. An electronic device, comprising: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program, so that the electronic device can implement the cultural relic image retrieval method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device can implement the cultural relic image retrieval method according to any one of claims 1 to 7.

Images