Image retrieval method combining domain ontology and multi-scale features

Abstract

The invention discloses an image retrieval method combining a domain ontology and multi-scale features, and the method comprises the steps: analyzing a domain where a data retrieval object is located, constructing the domain ontology used for expressing concepts and relationships thereof in the domain according to domain knowledge, and carrying out the structural expression of an image in a database; the domain ontology is used for carrying out structured representation on the images, and the images with high similarity are marked as rough retrieval images; for the query image and the coarse retrieval images, extracting global features of the images by using a DINOv2 model, and marking the three coarse retrieval images with the highest similarity as global feature retrieval images; for the query image and the global feature retrieval image, extracting local feature points of the images by using a SuperPoint network, and obtaining feature point matching pairs between the images by using a SuperGlue network; and calculating the local feature similarity between the query image and the global feature retrieval image, and outputting the image with the highest similarity as a final retrieval result.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to an image retrieval method combining domain ontology and multi-scale features. Background Art

[0002] Image retrieval technology has received extensive attention in recent years and is mainly used in the fields of Internet, medical image analysis, security monitoring, etc. However, with the rapid increase in data volume, traditional keyword-based retrieval methods face huge challenges and cannot effectively handle the diversity and complexity of images.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0004] The present invention provides an image retrieval method combining domain ontology and multi-scale features, which effectively improves the efficiency and accuracy of retrieval by using domain ontology to represent images in a structured manner and extracting multi-scale features. This method can not only narrow the retrieval scope according to semantic structure information, thus ensuring the speed of image retrieval, but also make full use of the multi-scale feature information of the image, thus solving the shortcomings of traditional methods in processing complex image data and ensuring the accuracy of image retrieval.

[0005] The purpose of the present invention is achieved through the following technical solutions.

[0006] The image retrieval method based on domain ontology and multi-scale features includes the following steps: Step 1: Analyze the domain of the data retrieval object in the database, build a domain ontology for expressing the concepts and their relationships in the domain based on the domain knowledge of space target situation awareness, and perform structured expression on the images in the database; Step 2: For the query image input by the user, the domain ontology is used to perform a structural representation of the query image, and the structural similarity between the query image and the image in the database in the ontology is calculated. The image with higher similarity is recorded as the coarse retrieval image; Step 3: For the query image and the rough retrieval image, the DINOv2 model is used to extract the global features of the image, and the cosine similarity between the features is calculated. The three rough retrieval images with the highest similarity are recorded as global feature retrieval images. Step 4: For the query image and the global feature retrieval image, the local feature points of the image are extracted using the SuperPoint network, and the feature point matching pairs between the images are obtained using the SuperGlue network; Step 5: Calculate the local feature similarity between the query image and the global feature retrieval image, that is, the cosine similarity of the feature point matching pairs, and output the image with the highest similarity as the final retrieval result.

[0007] In the method described, in step 1, knowledge in the field is collected and analyzed to generate a concept list, the hierarchical relationship between concepts is determined, and a tree structure is used to represent it. The nodes in the ontology tree are concepts, and the edges in the ontology tree are the relationships between concepts. Then, the database image is associated with the ontology tree, and semantic tags are added to the image to achieve structured expression of the database image.

[0008] In the method described, in step 2, semantic tags are added to the query image according to the domain ontology to achieve a structured representation of the query image; then, the structural similarity between the query image and the image in the database in the ontology tree is calculated. The higher the score, the greater the similarity. The screening threshold is set to 80% of the highest score. Images with similarity scores greater than the threshold are recorded as coarse retrieval images. The structural similarity calculation formula is as follows:

[0009] in, is the ontology tree of the query image, is the ontology tree of the database image, and are the nodes corresponding to the two trees respectively. is the weight corresponding to the node, is the node similarity, that is, the concept similarity, and the calculation formula is as follows:

[0010] in, is the sum of the number of edges in the shortest path connecting two nodes. Refers to The weight corresponding to the edge.

[0011] In the method described, in step 3, the global feature vector output by the DINOv2 model Size , the cosine similarity calculation formula is:

[0012] in, and are the global feature vectors of image A and image B respectively, Represents the vector modulus. Calculate the cosine similarity of the global features of the query image and the rough matching image, and sort them according to the score. The smaller the score, the higher the similarity. The three rough matching images with the highest similarity are recorded as global feature retrieval images.

[0013] In the method described, in step 4, the feature points extracted by the SuperPoint network include the key point coordinates and the descriptor vector , which together describe the local features of the image; secondly, the SuperGlue network matches the feature points of the two images, and the indexes of the successfully matched feature points in the corresponding images are saved for subsequent calculation of local feature similarity.

[0014] In the method described above, in step 5, the local feature similarity between images is measured by the cosine similarity of the feature point matching pairs, and the calculation formula is as follows:

[0015] in, and are the feature points of image A and image B respectively, is the number of feature point matching pairs, Represents the vector modulus length and is sorted according to the score. The smaller the score, the higher the similarity. The image with the highest similarity is output as the final retrieval result.

[0016] In the method, in step 1, knowledge in the field is collected and analyzed to generate a concept list, the hierarchical relationship between concepts is determined, and a tree structure is used to represent it. The nodes in the ontology tree are concepts, and the edges in the ontology tree are the relationships between concepts. Then, the database image is associated with the ontology tree, and semantic tags are added to the image to achieve structured expression of the database image. The specific steps are as follows: ① Analyze domain knowledge: Through consultation with domain experts, literature review and data analysis, collect and analyze relevant knowledge in the domain and generate a preliminary list of concepts, including main entities and attributes.

[0017] ② Define domain concepts and relationships: clarify the definition and attributes of each concept, determine the relationship between concepts, and establish the basic framework of the domain ontology.

[0018] ③ Build the structure of the domain ontology: establish a hierarchical structure for concepts and relationships to facilitate the organization and retrieval of knowledge, and use a tree diagram to organize concepts and relationships. The nodes in the tree are concepts, and the edges in the tree are the relationships between concepts.

[0019] ④Semantic annotation: Associate database images with concepts and relationships in the ontology, and make each image correspond to one or more concepts in the ontology by adding semantic tags to the images.

[0020] In step 2, semantic tags are added to the query image according to the domain ontology to achieve a structured representation of the query image; then, the structural similarity between the query image and the database image in the ontology tree is calculated. The higher the score, the greater the similarity. The screening threshold is set to 80% of the highest score. Images with similarity scores greater than this threshold are recorded as coarse retrieval images. The formula for calculating structural similarity is as follows:

[0021] in, is the ontology tree of the query image, is the ontology tree of the database image, and are the nodes corresponding to the two trees respectively. is the weight corresponding to the node, is the node similarity, that is, the concept similarity, and the calculation formula is as follows:

[0022] in, is the sum of the number of edges in the shortest path connecting two nodes. Refers to The weight corresponding to the edge.

[0023] In step 3, DINOv2 is a basic model for computer vision. It learns robust and universal visual features in the large-scale high-quality image dataset "LVD-142M". These features are applicable to different image distributions and different tasks and can be used without further fine-tuning. The DINOv2 network structure includes: Image block embedding layer: The image is divided into image blocks of fixed size, and each image block is embedded into a high-dimensional feature space through linear transformation; Encoder: It consists of multiple self-attention layers and feed-forward networks. Each self-attention layer captures the global features of the image by calculating the attention weights between image blocks. The encoder outputs a series of feature embedding vectors representing different regions of the image. Category tag: Based on the input image block, a category tag for the final feature representation is introduced, which aggregates global information during the encoding process of the model; Output head: Output features for classification, regression or feature matching.

[0024] We use the distilled DINOv2 model, which reduces the number of parameters from the original 1.1 billion to 300 million, which can not only speed up the model's reasoning speed, but also ensure better feature extraction capabilities. The global feature vector output by the DINOv2 model is Size , the cosine similarity calculation formula is:

[0025] in, and are the global feature vectors of image A and image B respectively, Represents the vector modulus. Calculate the cosine similarity of the global features of the query image and the rough matching image, and sort them according to the score. The smaller the score, the higher the similarity. The three rough matching images with the highest similarity are recorded as global feature retrieval images.

[0026] In step 4, the feature points extracted by the SuperPoint network contain the key point coordinates and the descriptor vector , which together describe the local features of the image; secondly, the SuperGlue network matches the feature points of the two images, and the indexes of the successfully matched feature points in the corresponding images are saved for subsequent calculation of local feature similarity.

[0027] SuperPoint is an end-to-end convolutional neural network for image key point detection and descriptor generation. It uses a shared encoder based on a convolutional neural network to extract image feature maps, and then generates key points and descriptors through two decoders. First, the input image is converted to a grayscale image and normalized to meet the input requirements of the SuperPoint model; second, the image is input into the SuperPoint model, and the image feature map is extracted through a shared convolutional encoder; then, SuperPoint outputs a key point heat map, selects the local maximum value in the heat map as the key point, and selects the key points with significant responses according to the threshold; finally, the feature map is branched through the descriptor generation branch to extract the local descriptor corresponding to each key point. The local descriptor is a high-dimensional feature vector that captures the local image information around the key point.

[0028] SuperGlue is a feature matching algorithm based on graph neural network, which is used for image feature point matching. SuperGlue finds the best matching pair between the feature points of a pair of images through the self-attention mechanism. First, the key point positions and corresponding descriptors extracted from the SuperPoint model are used as input; then, the key points and descriptors of each image are converted into a graph structure, where nodes represent key points and edges represent the relationship between them; secondly, SuperGlue applies the self-attention mechanism on the graph structure to calculate the similarity of all key point pairs between images, so as to find potential key point matching pairs between different images; finally, using the matching probability matrix output by SuperGlue, the key point pair with the highest probability is found as the final matching result.

[0029] In step 5, the local feature similarity between images is measured by the cosine similarity of the feature point matching pairs, and the calculation formula is as follows:

[0030] in, and are the feature points of image A and image B respectively, is the number of feature point matching pairs, Represents the vector modulus. Finally, the images are sorted according to the scores. The smaller the score, the higher the similarity. The image with the highest similarity is output as the final retrieval result.

[0031] Compared with the prior art, the present invention has the following advantages: ① Efficient retrieval process: Through the semantic hierarchical structured representation of the domain ontology, the image library can be preliminarily screened, the search scope can be narrowed, and the search accuracy and efficiency can be improved; ② Accurate feature representation: The DINOv2 model is used to extract the global features of the image, and the SuperPoint network and SuperGlue network are used to extract the local features of the image. By combining multi-scale features, it can provide a comprehensive image feature description and improve the accuracy of image retrieval; ③ Strong flexibility and adaptability: The domain ontology method can be flexibly extended to image retrieval tasks in different fields, while the DINOv2 pre-trained model can generate general visual features that are suitable for different image distributions and different tasks.

[0032] The above description is only an overview of the technical solution of the present invention. In order to make the technical means of the present invention clearer and to enable those skilled in the art to implement it according to the contents of the specification, and to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described below by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The drawings in the specification are only for the purpose of illustrating the preferred embodiments and are not considered to be limitations of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings.

[0034] In the attached picture: Figure 1 A flow chart of an image retrieval method combining domain ontology and multi-scale features provided by the present invention; Figure 2 A schematic diagram of the space target situation awareness domain ontology constructed in step 1 in an embodiment of the present invention; Figure 3 The rough search image result outputted in step 2 in the embodiment of the present invention; Figure 4 It is a feature attention heat map of the DINOv2 model for the spatial target image in an embodiment of the present invention; Figure 5 are the three rough retrieval images with the highest similarity outputted in step 3 of the embodiment of the present invention; Figure 6 It is the SuperPoint and SuperGlue feature point extraction and matching results and the corresponding local similarity scores output by steps 4-5; Figure 7 These are the image retrieval result diagrams under different working conditions.

[0035] The present invention is further explained below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0036] The following will refer to the attached Figures 1 to 7 Specific embodiments of the present invention are described in more detail. Although specific embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0037] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present invention, but the description is based on the general principles of the specification and is not intended to limit the scope of the present invention. The scope of protection of the present invention shall be determined by the attached claims.

[0038] To facilitate understanding of the embodiments of the present invention, further explanation will be given below by taking specific embodiments as examples in conjunction with the accompanying drawings, and each of the accompanying drawings does not constitute a limitation on the embodiments of the present invention.

[0039] like Figure 1 As shown, the present invention provides an image retrieval method combining domain ontology and multi-scale features, comprising the following steps: Step 1: Analyze the domain of the data retrieval object, construct a domain ontology to express the concepts and their relationships in the domain based on domain knowledge, and perform structured expression on the images in the database; Step 2: For the query image input by the user, the domain ontology is used to perform a structural representation of the image, and the structural similarity between the image and the database image in the ontology is calculated. The image with the higher similarity is recorded as the coarse retrieval image; Step 3: For the query image and the rough retrieval image, the DINOv2 model is used to extract the global features of the image, and the cosine similarity between the features is calculated. The three rough retrieval images with the highest similarity are recorded as global feature retrieval images. Step 4: For the query image and the global feature retrieval image, the local feature points of the image are extracted using the SuperPoint network, and the feature point matching pairs between the images are obtained using the SuperGlue network; Step 5: Calculate the local feature similarity between the query image and the global feature retrieval image, that is, the cosine similarity of the feature point matching pairs, and output the image with the highest similarity as the final retrieval result.

[0040] The following is an example process of space target image retrieval.

[0041] Figure 2 Schematic diagram of the space target situation awareness domain ontology constructed in step 1. The concepts in the space target situation awareness domain include satellite name, structural components, payload, space debris, background, etc., and these concepts contain other sub-concepts. For example, structural components can be divided into solar panels, satellite bodies, etc., and payloads can be divided into sensors, controllers, etc. By sorting out the connections between these concepts and using a tree structure to represent them, the nodes in the ontology tree are concepts, and the edges in the ontology tree are the relationships between concepts. Finally, the database image is associated with the ontology tree, and semantic tags are added to the image to achieve structured expression of the database image.

[0042] Figure 3 is the rough retrieval image result output from step 2. First, semantic tags are added to the query image according to the domain ontology of space target situation awareness to achieve the structured representation of the query image; then, the structural similarity between the query image and the database image in the ontology tree is calculated. The higher the score, the greater the similarity. The calculation formula is as follows:

[0043] in, is the ontology tree of the query image, is the ontology tree of the database image, and are the nodes corresponding to the two trees respectively. is the weight corresponding to the node, is the node similarity, that is, the concept similarity, and the calculation formula is as follows:

[0044] in, is the sum of the number of edges in the shortest path connecting two nodes. Refers to The weight corresponding to the edge.

[0045] In this example, the maximum score is 100%, and the screening threshold is 80% of the maximum score, so the threshold is set to 80%, and images with similarity scores greater than the threshold are recorded as coarse retrieval images.

[0046] Figure 4 It is the feature attention heat map of the DINOv2 model for the space target image. In terms of space target situation awareness, for the present invention, through the feature attention map of the space target, it can be found that the DINOv2 model can well obtain the shape structure information of the space target and has a certain anti-interference ability to the space background.

[0047] Figure 5 are the three coarse retrieval images with the highest similarity output from step 3. The global feature vector output by the DINOv2 model Size , the cosine similarity calculation formula is:

[0048] in, and are the global feature vectors of image A and image B respectively, Represents the vector modulus. Calculate the cosine similarity of the global features of the query image and the rough matching image, and sort them according to the score. The smaller the score, the higher the similarity. The three rough matching images with the highest similarity are recorded as global feature retrieval images.

[0049] Figure 6 It is the SuperPoint and SuperGlue feature point extraction and matching results and the corresponding local similarity scores output by step 4-5. Combining SuperPoint feature point detection and SuperGlue feature point matching, it is possible to obtain feature point pairs between images. The local feature similarity between images is measured by the cosine similarity of feature point matching pairs, and the calculation formula is as follows:

[0050] in, and are the feature points of image A and image B respectively, is the number of feature point matching pairs, Represents the vector modulus. Finally, the images are sorted according to the scores. The smaller the score, the higher the similarity. The image with the highest similarity is output as the final retrieval result.

[0051] Figure 7 The following are the image retrieval results for space target situational awareness under different working conditions. In order to verify the robustness and accuracy of this method, image retrieval experiments were conducted under different lighting conditions, including simulated images with uniform lighting, real images with low lighting, and real images with strong lighting. The experimental results show that this method combines the semantic information of the domain ontology and the rich description capabilities of multi-scale features, and can achieve accurate retrieval of images under different lighting conditions, which is suitable for space target image retrieval tasks.

[0052] In one embodiment, the method includes, Firstly, the domain in which the data retrieval object is located is analyzed, and a domain ontology for expressing the concepts and their relationships within the domain is constructed based on domain knowledge, and the images in the database are structurally expressed; secondly, for the query image input by the user, the domain ontology is used to structurally represent the image, and its structural similarity with the database image in the ontology is calculated. The images with higher similarity are recorded as coarse retrieval images; then, for the query image and the coarse retrieval image, the DINOv2 model is used to extract the global features of the image, and the cosine similarity between the features is calculated. The three coarse retrieval images with the highest similarity are recorded as global feature retrieval images; furthermore, for the query image and the global feature retrieval image, the SuperPoint network is used to extract the local feature points of the image, and the SuperGlue network is used to obtain the feature point matching pairs between the images; finally, the local feature similarity between the query image and the global feature retrieval image, that is, the cosine similarity of the feature point matching pairs, is calculated, and the image with the highest similarity is output as the final retrieval result.

[0053] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. An image retrieval method combining domain ontology and multi-scale features, characterized in that: The steps include: Step 1: Analyze the domain of the data retrieval object in the database, build a domain ontology for expressing the concepts and their relationships in the domain based on the domain knowledge of space target situation awareness, and perform structured expression on the images in the database; Step 2: For the query image input by the user, the domain ontology is used to perform a structural representation of the query image, and the structural similarity between the query image and the image in the database in the ontology is calculated. The image with higher similarity is recorded as the coarse retrieval image; Step 3: For the query image and the rough retrieval image, the DINOv2 model is used to extract the global features of the image, and the cosine similarity between the features is calculated. The three rough retrieval images with the highest similarity are recorded as global feature retrieval images. Step 4: For the query image and the global feature retrieval image, the local feature points of the image are extracted using the SuperPoint network, and the feature point matching pairs between the images are obtained using the SuperGlue network; Step 5: Calculate the local feature similarity between the query image and the global feature retrieval image, that is, the cosine similarity of the feature point matching pairs, and output the image with the highest similarity as the final retrieval result.

2. The method according to claim 1, characterized in that: Preferably, in step 1, knowledge in the field is collected and analyzed to generate a concept list, the hierarchical relationship between concepts is determined, and a tree structure is used to represent it. The nodes in the ontology tree are concepts, and the edges in the ontology tree are the relationships between concepts. Then, the database image is associated with the ontology tree, and semantic tags are added to the image to achieve structured expression of the database image.

3. The method according to claim 2, characterized in that In step 2, semantic tags are added to the query image according to the domain ontology to achieve a structured representation of the query image. Then, the structural similarity between the query image and the database image in the ontology tree is calculated. The higher the score, the greater the similarity. The screening threshold is set to 80% of the highest score. Images with similarity scores greater than this threshold are recorded as coarse retrieval images. The formula for calculating structural similarity is as follows: ， in, is the ontology tree of the query image, is the ontology tree of the database image, and are the nodes corresponding to the two trees respectively. is the weight corresponding to the node, is the node similarity, that is, the concept similarity, and the calculation formula is as follows: ， in, is the sum of the number of edges in the shortest path connecting two nodes. Refers to The weight corresponding to the edge.

4. The method according to claim 1, characterized in that In step 3, the global feature vector output by the DINOv2 model is Size , the cosine similarity calculation formula is: ， in, and are the global feature vectors of image A and image B respectively, Represents the vector modulus; calculate the cosine similarity of the global features of the query image and the rough matching image, and sort them according to the score. The smaller the score, the higher the similarity. The three rough matching images with the highest similarity are recorded as global feature retrieval images.

5. The method according to claim 1, characterized in that In step 4, the feature points extracted by the SuperPoint network contain the key point coordinates and the descriptor vector , which together describe the local features of the image; secondly, the SuperGlue network matches the feature points of the two images, and the indexes of the successfully matched feature points in the corresponding images are saved for subsequent calculation of local feature similarity.

6. The method according to claim 1, characterized in that In step 5, the local feature similarity between images is measured by the cosine similarity of the feature point matching pairs, and the calculation formula is as follows: ， in, and are the feature points of image A and image B respectively, is the number of feature point matching pairs, Represents the vector modulus length and is sorted according to the score. The smaller the score, the higher the similarity. The image with the highest similarity is output as the final retrieval result.

Images