A heterogeneous image matching method based on meta-template knowledge base

Abstract

The present invention relates to a heterogeneous image matching method based on a meta-template knowledge base, and belongs to the technical field of image recognition and matching. The method includes: constructing a heterogeneous image knowledge base based on a meta-template: establishing a structure for building a knowledge base based on a meta-template standard; forming a meta-template knowledge base for heterogeneous images; segmenting heterogeneous images and extracting underlying features; carrying out image feature mapping; constructing a heterogeneous image pyramid model based on a meta-template knowledge base; generating a linear quadtree index of the pyramid model; performing heterogeneous image matching training and prediction based on a pseudo-twin network; the method can solve heterogeneous matching problems in more image domains by forming multiple groups of pseudo-twin networks compared to existing optical and millimeter wave heterogeneous matching; solving the problems of rapid storage, understanding, retrieval and reading of massive image data, effectively supporting the matching of heterogeneous images in different fields, and providing an innovative and feasible technical approach for heterogeneous image recognition and matching.

Description

Technical Field

[0001] The invention relates to a heterogeneous image matching method based on a meta-template knowledge base, belonging to the technical field of image recognition and matching. Background Art

[0002] With the rapid development of science and technology, image recognition and matching technology has become a very important technology in modern information processing, especially in the field of image information processing. The number of image data of the same area obtained by different types of sensors such as visible light, near infrared, short-wave infrared, thermal infrared, microwave, millimeter wave is increasing. In order to fully, effectively and comprehensively utilize various types of heterogeneous image data, the research on image recognition and matching technology is also on the rise. The recognition and matching of heterogeneous images (images of different standard specifications collected by different sensors) has become a hot research issue in the fields of machine vision, intelligent manufacturing, intelligent transportation, military applications, etc.

[0003] With the booming development of the Internet and big data and the rapid improvement of computer hardware, the data that can be used for image recognition and matching has shown a geometric growth. Such massive image data requires an effective management mechanism to solve the problems of efficient storage, effective understanding, rapid retrieval and high-speed reading of massive image data. Due to the richness of features and content contained in images, it is difficult to use key words to describe semantics, forming a semantic gap in images. In order to achieve effective understanding of image semantics, meta-template-based image segmentation, underlying feature extraction, image feature mapping and other technologies are used to form a knowledge base of massive image data to solve the problems of rapid storage, understanding, retrieval and reading of massive image data.

[0004] From the perspective of spectral resolution, spatial resolution and temporal resolution, the spectral domain of heterogeneous images has expanded from early visible light, near infrared, short-wave infrared, thermal infrared, microwave, etc. to electromagnetic spectra such as millimeter waves; the band domain has expanded from black and white photography, 3 bands, 4 bands, and 7 bands to the currently commonly used 100μm-200μm. Using Fourier spectral analysis technology, it can reach more than a thousand bands, and the band width is 0.4μm, 0.1μm, and 5nm in the early days. The spatial resolution includes standards such as 80m, 30m, 20m, 5m, 3m, 1m, and 0.15m, which leads to the technical complexity of heterogeneous image recognition and matching. Based on the generation results of the meta-template knowledge base of heterogeneous images, a multi-resolution hierarchical pyramid model of different sensors is established to solve the normalized modal processing problem of heterogeneous images.

[0005] The recognition and matching of heterogeneous images is a necessary condition for heterogeneous image fusion analysis. Its accuracy directly affects the results of subsequent applications such as target detection, target recognition, and change detection. Although certain achievements have been made in the field of image recognition and matching, due to radiation differences, geometric deformation, and nonlinear changes caused by different sensors and different viewpoints, the recognition and matching of heterogeneous images is still a very challenging technical problem. The coarse matching and fine matching based on the pseudo-twin network structure are integrated to solve the problem of fast and accurate matching of heterogeneous massive large images. Summary of the invention

[0006] The purpose of the present invention is to address the low matching efficiency and matching rate of multi-source heterogeneous images such as visible light, infrared, microwave, and millimeter waves, which cannot meet the image matching needs in the fields of machine vision, intelligent manufacturing, intelligent transportation, and military applications. A heterogeneous image matching method based on a meta-template knowledge base is proposed, a meta-template knowledge base of massive heterogeneous images is analyzed, a multi-resolution hierarchical pyramid model of different sensors is constructed based on the meta-template knowledge base, and coarse matching and fine matching based on a pseudo-twin network structure are integrated to achieve matching for heterogeneous images.

[0007] In order to achieve the above object, the present invention adopts the following scheme:

[0008] The heterogeneous image matching method based on the meta-template knowledge base comprises the following steps:

[0009] S1. Constructing a meta-template-based heterogeneous image knowledge base, specifically: establishing a structure for building a knowledge base based on a meta-template standard and forming a meta-template knowledge base for heterogeneous images, including the following sub-steps:

[0010] S11. Establish a structure for building a knowledge base based on meta-template standards;

[0011] The meta template structure includes data sets, data set series, metadata, metadata elements, metadata entities, metadata subsets, core metadata, full set metadata, extended metadata, dedicated standards, metadata dictionaries and metadata hierarchies;

[0012] The data set is a set of data that can be identified; the data set series is a set of several data sets that use the same specification and can be uniformly classified; metadata is data that describes data and can describe the attribute information of data; metadata elements are the basic units of metadata, equivalent to an attribute in the unified modeling language, and metadata elements are unique in the same metadata entity; the metadata entity, a set of one or more metadata elements, is a set of metadata elements that describe the same type of data features; metadata subsets are a set of multiple related metadata entities and elements; core metadata is the minimum metadata elements that must be retained when a basic description of a data set is made, and is a part of the full set metadata elements; full set metadata is all metadata elements when a data set is described in detail; extended metadata is metadata entities or metadata elements that are extended using the consistent principle when the full set metadata cannot meet the application; the dedicated standard is a standard for selecting subsets, entities, and elements from the full set metadata to meet specific application fields or users; the metadata dictionary is a relationship between metadata subsets, entities, and elements in a certain hierarchical language structure, and defines and explains metadata classes and attributes using names / roles, definitions, constraints, maximum occurrences, data types, and value ranges;

[0013] Among them, the unified modeling language, namely Universal Modeling Language, referred to as UML;

[0014] S12, forming a meta-template knowledge base of heterogeneous images;

[0015] The meta-template knowledge base includes 6 meta-data sets, including image meta-database, image identification information, image quality information, image reference system information, image content information and image source information;

[0016] The image meta-template library is a root entity that defines meta-templates related to various image information resources; the image identification information is the image data set identification information that describes the basic information of the image data set; the image quality information is the image data quality information that provides overall evaluation information of the image data quality; the image reference system information is the image reference system information that provides a description of the space and time reference system used by the image data; the image content information is the image content information that provides element classification information and describes the data layer and image data features; the image source information is the image source information that provides platform information for acquiring image data, sensor type information, sensor feature information, image correction information, data log information and related attribute information of the image product;

[0017] The image data set is a data set containing image information, and is not limited to the 6 meta data sets included in the meta template knowledge base;

[0018] S2. Segment the heterogeneous image and extract the underlying features to extract the underlying features of the image, specifically:

[0019] S21, segmenting the heterogeneous image, specifically:

[0020] Adaptively process the pixels and roughness in the image, determine the quantization number of the image color, and obtain the extreme value of the color quantization LUV component; divide the LUV component equally according to the color quantization number to form the initial color quantization value of the image, and form the metadata element of the color quantization value of the image;

[0021] Perform region growth, merge each sub-region according to the normalized blending distance, and when the merging stop condition is reached, the automatic segmentation of the image stops; after the automatic segmentation stops, perform manually assisted image sub-region merging to segment and form the image metadata root entity of the image and the image segmentation region set;

[0022] Among them, the blending distance includes color distance and roughness distance;

[0023] The segmented region set of the image includes a number of metadata root entity data, and there is a mutual correlation relationship between the data;

[0024] S22, extracting underlying features of the image, and storing the extracted underlying features of the image in a meta-template knowledge base of heterogeneous images, specifically including:

[0025] In terms of color, extract the color mean;

[0026] On the texture, the entropy, contrast, energy and contrast moment of the gray-level co-occurrence matrix are extracted;

[0027] In terms of shape, the shape invariant moment features of the segmented area are extracted;

[0028] The underlying features of these images are used as the entity resolution module to store them in the meta-template knowledge base of heterogeneous images;

[0029] S3. Carry out image feature mapping, specifically:

[0030] S31, using the feature data in the meta-template knowledge base as a training sample data set, using an inductive learning method to construct a mapping rule for the metadata root entity, and using the rule as a decision rule file for template root entity parsing;

[0031] S32, using the underlying feature data of the image in the meta-template knowledge base as the input of the mapping, and the determination rule of the metadata root entity and the description file for the entity form the output of the mapping;

[0032] S33, normalizing the metadata root entity data; after the normalization process, discretizing each data, and forming a discrete description for each value, to obtain discrete description data;

[0033] The normalization prevents a certain component from being too large, so that each component has the same influence in the system;

[0034] S34, constructing a decision tree using the discretized description data and generating a judgment rule for a corresponding entity; using the discretized entity description data to form a corresponding feature description table;

[0035] S35, perform higher-level semantic analysis to obtain a reasonable understanding of the image feature mapping, specifically:

[0036] For each region after image segmentation, the mapping result of the region is obtained according to the underlying features of the color, texture and shape of the region; when there is more than one result, a candidate result queue is obtained;

[0037] For the metadata root entity data structure, a solution is selected from the candidate solution set of each segmented region, and the cumulative sum of the path weights connecting these solutions is maximized;

[0038] Based on the meta-template knowledge base, a reasonable understanding of the image feature mapping is obtained;

[0039] S4. Constructing a heterogeneous image pyramid model based on a meta-template library knowledge base, specifically: constructing the image resolution layer by layer using a magnification method or a specified resolution method, the resolution of the pyramid gradually decreases from the bottom layer to the top layer, and the image area range represented remains unchanged, thereby forming multiple resolution levels, including the following sub-steps:

[0040] S41, performing heterogeneous image domain division, that is, determining the domain to which the multi-source heterogeneous images belong;

[0041] S42, determining the initial resolution of the image as r according to the meta-template knowledge base 0 ;

[0042] S43, constructing the resolution of each layer of the pyramid model, generating an adaptive magnification matrix according to the image adaptation requirements to construct a layered pyramid model;

[0043] The resolution of each pyramid layer is formed by using a 2-fold equal step length to form an m matrix, and a hierarchical pyramid model is constructed. S43 is specifically: the original resolution r of the heterogeneous image 0 As the 0th layer of the pyramid, the data in the magnification m matrix are all greater than 1, ensuring that the resolution of the high-level pyramid is higher than the resolution of the low-level pyramid. Based on the image resolution of the 0th layer of the pyramid, The first layer is generated by combining pixels into one pixel, and the image is divided into blocks to form a first layer image block matrix. The second layer image block matrix is ​​generated by the same method on the basis of the first layer, until an n-layer pyramid model is generated;

[0044] S44, generating a linear quadtree index of the pyramid model, and performing spatial recursive decomposition;

[0045] The quadtree is a tree structure in which each non-leaf node has at most four branches. The quadtree structure is a hierarchical data structure whose common feature is spatial recursive decomposition.

[0046] S44. Specifically include:

[0047] Combine the position coordinates of the image block and the size of the image block to calculate the coordinates of the lower left pixel, the upper right pixel, and the coordinates of the image block to which the pixel belongs; calculate the number of rows and columns of the image block matrix according to the number of rows and columns of the pixel matrix, and calculate the total number of blocks in each layer; calculate the storage offset according to the starting offset of each layer, the number of bytes of the image block, and the storage offset of the image block;

[0048] The indexing process is as follows: extract the image block that overlaps with the regional coverage from the lth layer, obtain the pixel coordinates of the lower left corner and the upper right corner of the lth layer, calculate the storage offset of the image block, and complete the index calculation of the linear quadtree;

[0049] So far, from S1 to S4, the problem of normalized modality processing of images has been solved by modeling the pyramid model of multi-source heterogeneous images;

[0050] S5. Construct a heterogeneous image matching network based on a pseudo twin network, specifically using pseudo twin network structure matching, including three parts: a coarse matching network, a fine matching network, and a mismatch elimination network;

[0051] The coarse matching network and the fine matching network both adopt a pseudo twin network structure;

[0052] The mismatch removal network uses a combination of multiple convolutional and pooling layers to transform the mismatch removal task into a thermal Figure 2 For classification problems, we can filter the diffuse distribution heat map by setting the network output probability confidence to eliminate false matches;

[0053] S6, performing heterogeneous image matching training, specifically: during training, the data set is first input into the fine matching network, and then the coarse matching network and the wrong matching elimination network are trained based on the training results of the fine matching network;

[0054] S7, performing heterogeneous image matching prediction, specifically: during the prediction, the large image is first input into the coarse matching network to obtain a possible suitable matching area, then the fine matching network predicts the registration point, and finally the final high-precision but sparse matching area is obtained through the mismatch elimination network;

[0055] So far, from S1 to S7, a heterogeneous image matching method based on meta-template knowledge base has been completed.

[0056] Beneficial Effects

[0057] Compared with the existing heterogeneous image matching methods, the heterogeneous image matching method based on the meta-template knowledge base described in the present invention has the following beneficial effects:

[0058] 1. The method adopts meta-template, deep learning and other technical methods, and realizes the matching of multi-source heterogeneous images such as visible light, infrared, microwave, and millimeter wave by forming multiple sets of pseudo-twin networks. Compared with the existing optical and millimeter wave heterogeneous matching, it can solve heterogeneous matching problems in more image domains;

[0059] 2. The heterogeneous image matching method has made breakthroughs in image segmentation, underlying feature extraction and image feature mapping based on meta-templates, and a heterogeneous image pyramid model based on a meta-template knowledge base of heterogeneous images; it has solved the problems of rapid storage, understanding, retrieval and reading of massive image data, the normalization modality processing of heterogeneous images, and the rapid and accurate retrieval of heterogeneous massive large images;

[0060] 3. The heterogeneous image matching method can effectively support the matching of heterogeneous images in the fields of machine vision, intelligent manufacturing, intelligent transportation, military applications, etc., and provides an innovative and feasible technical approach for heterogeneous image recognition and matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a UML class diagram of a meta-template knowledge base of heterogeneous images in a heterogeneous image matching method based on a meta-template knowledge base of the present invention;

[0062] Figure 2 It is a structural diagram of an image block pyramid model in a heterogeneous image matching method based on a meta-template knowledge base of the present invention;

[0063] Figure 3 It is a schematic diagram of a quadtree index structure of a pyramid model in a heterogeneous image matching method based on a meta-template knowledge base of the present invention;

[0064] Figure 4 It is a schematic diagram of a pseudo twin network structure in a heterogeneous image matching method based on a meta-template knowledge base of the present invention;

[0065] Figure 5It is a schematic diagram of a heterogeneous image matching technology based on a pseudo twin network in a heterogeneous image matching method based on a meta-template knowledge base of the present invention;

[0066] Figure 6 It is a technical roadmap of heterogeneous image matching based on a meta-template knowledge base in a heterogeneous image matching method based on a meta-template knowledge base of the present invention. DETAILED DESCRIPTION

[0067] A heterogeneous image matching method based on a meta-template knowledge base of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Example 1

[0069] The matching efficiency and matching rate of multi-source heterogeneous images such as visible light, infrared, microwave, and millimeter wave are low, which cannot meet the image matching requirements in the fields of machine vision, intelligent manufacturing, intelligent transportation, and military applications. This embodiment adopts a heterogeneous image matching method based on a meta-template knowledge base, analyzes and forms a meta-template knowledge base of massive heterogeneous images, builds a multi-resolution hierarchical pyramid model of different sensors based on the meta-template knowledge base, and realizes the integration of coarse matching and fine matching based on a pseudo-twin network structure, so as to realize matching for heterogeneous images. Figure 6 The technical roadmap of heterogeneous image matching based on meta-template knowledge base is shown in the figure.

[0070] This embodiment includes constructing a meta-template-based heterogeneous image knowledge base, constructing a heterogeneous image pyramid model based on the meta-template knowledge base, and adopting heterogeneous image matching based on a pseudo-twin network. Specifically, the following scheme is adopted:

[0071] Construct a meta-template-based heterogeneous image knowledge base, specifically: establish a knowledge base structure based on the meta-template standard and form a meta-template knowledge base for heterogeneous images, such as Figure 1 The meta-template knowledge base of heterogeneous images is shown in the UML class diagram, which includes the following sub-steps:

[0072] S1. Establish a structure for building a knowledge base based on meta-template standards;

[0073] The meta template structure includes data sets, data set series, metadata, metadata elements, metadata entities, metadata subsets, core metadata, full set metadata, extended metadata, dedicated standards, metadata dictionaries and metadata hierarchies;

[0074] The data set is a set of data that can be identified; the data set series is a set of several data sets that use the same specification and can be uniformly classified; metadata is data that describes data and can describe the attribute information of data; metadata elements are the basic units of metadata, equivalent to an attribute in the unified modeling language, and metadata elements are unique in the same metadata entity; the metadata entity, a set of one or more metadata elements, is a set of metadata elements that describe the same type of data features; metadata subsets are a set of multiple related metadata entities and elements; core metadata is the minimum metadata elements that must be retained when a basic description of a data set is made, and is a part of the full set metadata elements; full set metadata is all metadata elements when a data set is described in detail; extended metadata is metadata entities or metadata elements that are extended using the consistent principle when the full set metadata cannot meet the application; the dedicated standard is a standard for selecting subsets, entities, and elements from the full set metadata to meet specific application fields or users; the metadata dictionary is a relationship between metadata subsets, entities, and elements in a certain hierarchical language structure, and defines and explains metadata classes and attributes using names / roles, definitions, constraints, maximum occurrences, data types, and value ranges;

[0075] Among them, the unified modeling language, namely Universal Modeling Language, referred to as UML;

[0076] Forming a meta-template knowledge base of heterogeneous images;

[0077] The meta-template knowledge base includes 6 meta-data sets, including image meta-database, image identification information, image quality information, image reference system information, image content information and image source information;

[0078] The image meta-template library is a root entity that defines meta-templates related to various image information resources; the image identification information is the image data set identification information that describes the basic information of the image data set; the image quality information is the image data quality information that provides overall evaluation information of the image data quality; the image reference system information is the image reference system information that provides a description of the space and time reference system used by the image data; the image content information is the image content information that provides element classification information and describes the data layer and image data features; the image source information is the image source information that provides platform information for acquiring image data, sensor type information, sensor feature information, image correction information, data log information and related attribute information of the image product;

[0079] The image data set is a data set containing image information, and is not limited to the 6 meta data sets included in the meta template knowledge base;

[0080] S2. Segment the heterogeneous image and extract the underlying features to extract the underlying features of the image, specifically:

[0081] S21, segmenting the heterogeneous image, specifically:

[0082] Adaptively process the pixels and roughness in the image, determine the quantization number of the image color, and obtain the extreme value of the color quantization LUV component; divide the LUV component equally according to the color quantization number to form the initial color quantization of the image and the metadata element of the color quantization value of the image;

[0083] Perform region growth and merge each sub-region according to the normalized mixed distance;

[0084] When the merging stop condition is reached, the automatic segmentation of the image stops; after the automatic segmentation stops, the image sub-regions are merged with manual assistance to form the image metadata root entity of the image;

[0085] Among them, the blending distance includes color distance and roughness distance;

[0086] S22, extracting underlying features of the image; specifically including:

[0087] In terms of color, extract the color mean;

[0088] On the texture, the entropy, contrast, energy and contrast moments of the gray-level co-occurrence matrix are extracted;

[0089] In terms of shape, the shape invariant moment features of the segmented area are extracted;

[0090] The underlying features of these images are stored in the meta-template knowledge base of heterogeneous images as the entity resolution module

[0091] S3. Carry out image feature mapping, specifically:

[0092] S31, image feature mapping uses feature data in the meta-template knowledge base as a training sample data set, adopts an inductive learning method, constructs a mapping rule for the metadata root entity, and uses the rule as a decision rule file for data root entity resolution;

[0093] The underlying feature data in the meta-template knowledge base is used as the input of the mapping, and the decision rules of the metadata root entity data and the description file of the entity form the output of the mapping;

[0094] S32. For the metadata root entity data, normalize it to avoid a certain component being too large, so that each component has the same influence in the system; after normalization, discretize each data and form a discrete description for each value; then use the discretized description data to build a decision tree and generate the judgment rules for the corresponding entity. And use the discretized entity description data to form a corresponding feature description table;

[0095] S33, perform higher-level semantic parsing work, for each region after image segmentation, obtain the mapping result of the region according to the underlying features such as color, texture and shape of the region, and obtain a candidate result queue when there is more than one result;

[0096] For the metadata root entity data structure, a solution is selected from the candidate solution set of each segmented region, and the cumulative sum of the path weights connecting these solutions is maximized;

[0097] The segmented region set of an image is composed of several metadata root entity data and has a mutual correlation. Based on the meta-template knowledge base, a reasonable understanding of the image feature mapping is obtained;

[0098] So far, from S1 to S3, the construction of a meta-template-based heterogeneous image knowledge base has been realized; that is, for massive image data, a standard definition and description of image meta-templates are established, a knowledge base solution for image meta-templates is constructed, and a standard meta-template knowledge base structure for heterogeneous images is formed. Image segmentation and feature extraction are performed on multi-source heterogeneous image data to achieve standardized and rapid storage, understanding, retrieval and reading capabilities of image data in the meta-template knowledge base. Feature mapping of massive heterogeneous images is completed, and the judgment rules for metadata root entities and the mapping output for entity description files are formed.

[0099] S4, build a heterogeneous image pyramid model based on the meta-template library knowledge base, such as Figure 2 The structure of the image block pyramid model is shown in the figure, specifically:

[0100] S41. Heterogeneous image pyramid model is constructed by using the magnification method or the specified resolution method for image resolution stratification. The resolution of the pyramid gradually decreases from the bottom to the top layer, and the image area range represented remains unchanged, thereby forming multiple resolution levels, specifically:

[0101] S42. When constructing an image pyramid, first perform heterogeneous image domain division to determine the domains to which multi-source heterogeneous images such as visible light, infrared, microwave, and millimeter waves belong.

[0102] S43, determining the initial resolution of the image as r according to the meta-template knowledge base 0 , the pyramid's layering ratio is m matrix;

[0103] When S43 is implemented, the resolution of each pyramid layer forms an m matrix: m = [m 1 ,m 2 ,m 3 ,...,m n ];

[0104] A hierarchical pyramid model is formed using a 2-fold equal step length, specifically: the original resolution of the heterogeneous image is r 0 As the 0th layer of the pyramid, the data in the magnification m matrix are all greater than 1, ensuring that the resolution of the high-level pyramid is higher than the resolution of the low-level pyramid. Based on the image resolution of the 0th layer of the pyramid, The first layer is generated by synthesizing pixels into 1 pixel, and the image is divided into blocks to form the first layer image block matrix. The second layer image block matrix is ​​generated by the same method on the basis of the first layer until an n-layer pyramid model is generated; among them, the resolution of the first layer is r l for:

[0105]

[0106] Indicates m l The integer rounded up after the square root, 1≤l≤n; n is the highest level of the pyramid model;

[0107] S44, generating a linear quadtree index of the pyramid model, and performing spatial recursive decomposition;

[0108] The quadtree is a tree structure in which each non-leaf node has at most four branches. The quadtree structure is a hierarchical data structure whose common feature is spatial recursive decomposition.

[0109] When S44 is implemented, it specifically includes: according to the size of the image block Size i , combined with the position coordinates of the image block (x t ,y t ), the pixel coordinates of the lower left corner of the i-th image block (x lbi ,y lbi ) and the upper right pixel coordinate (x rti ,y rti ):

[0110]

[0111] The image block or according to the pixel coordinates (x i ,y i ), calculate the coordinates of the image block to which the pixel belongs (x t ,y t ):

[0112]

[0113]

[0114] is an integer rounded down;

[0115] According to the number of rows r of the pixel matrix i and the number of columns c i , calculate the number of rows r of the image block matrix t and the number of columns c t , calculate the total number of blocks in each layer; calculate the storage offset, based on the starting offset of each layer as os l , the number of bytes of the image block is bs t , image block (x t ,y t ,l) storage offset calculation:

[0116] offset=(y t ·r tl +x t ) ·bs t +os l

[0117]

[0118] The indexing process is to extract the region (x lbi ,y lbi ) and (x rti ,y rti ) overlapped image blocks, and the lower left corner pixel (x tlb ,y tlb ) coordinates and the upper right pixel (x trt ,y trt ) coordinates, calculate the storage offset of the image block, and complete the index calculation of the linear quadtree.

[0119] So far, from the above steps, a heterogeneous image pyramid model based on the meta-template library knowledge base has been realized; that is, considering the complexity of heterogeneous images in image acquisition, image format, resolution, etc., a hierarchical processing method of the heterogeneous image pyramid model is adopted to form an image block pyramid model structure in the image domain; a linear quadtree indexing method of the pyramid model is established to achieve rapid indexing of massive heterogeneous images; by modeling the pyramid model of multi-source heterogeneous images, the problem of normalized modal processing of images is solved.

[0120] S5. Conduct heterogeneous image matching based on pseudo twin network, specifically:

[0121] Use Figure 4The pseudo-twin network structure matching is divided into three parts: coarse matching, fine matching and false matching elimination. The reason for using the pseudo-twin network is that the imaging mechanism of heterogeneous images, especially millimeter waves, is very different from that of optical images, resulting in images with very different radiation and geometric characteristics. The similarity of image features extracted by conventional matching methods is difficult to guarantee, and it is difficult to improve the accuracy of image matching. The pseudo-twin network refers to a network structure with the same dual-branch network structure but independent weight parameters. The pseudo-twin network can learn modality-specific features, and therefore can better adapt to the different feature representations of heterogeneous images such as millimeter waves and optical images to a certain extent, so the pseudo-twin network is used.

[0122] The coarse matching and fine matching both adopt pseudo twin network structures. The coarse matching uses VGG11 skeleton and binary cross entropy loss; the fine matching uses multi-scale feature extraction, spatial feature dimension reduction and mean square error loss; the mismatch elimination network uses a combination of multiple convolution and pooling layers, adopts Sigmoid activation function and binary cross entropy loss function, and transforms the mismatch elimination task into thermal model. Figure 2 For classification problems, we can filter the diffuse distribution heat map by setting the network output probability confidence to eliminate false matches;

[0123] Figure 5 As shown in the schematic diagram of heterogeneous image matching based on pseudo twin network, both the coarse matching network (5a) and the fine matching network (5b) adopt the pseudo twin network structure, and the mismatch elimination network (5c) uses a combination of convolution and pooling layers;

[0124] S6. Perform heterogeneous image matching training and prediction: During training, the data set is first input into the fine matching network, and then the coarse matching network and the false matching elimination network are trained based on the training results of the fine matching network; during prediction, a large image is first input into the coarse matching network to obtain a possible suitable matching area, and then the fine matching network predicts the registration point, and finally the false matching elimination network is used to obtain the final high-precision but sparse matching area;

[0125] Through S6, heterogeneous image matching based on pseudo twin network is carried out; that is, considering the large differences in radiation and geometric features of heterogeneous images, combined with the requirements of heterogeneous image data recognition and matching for timeliness, heterogeneous image matching based on pseudo twin network is adopted, and three steps of coarse matching, fine matching and mismatch elimination are adopted respectively, to gradually realize the generation of adaptive feature points for large-scale and multi-feature image data, the refined analysis of adaptive areas, and the judgment and elimination of mismatch areas. The technical method of deep learning training process is used to achieve high-precision and high-time heterogeneous image recognition and matching.

[0126] In order to meet the requirements of accuracy and real-time performance of massive heterogeneous image recognition and matching in the fields of machine vision, intelligent manufacturing, intelligent transportation, military applications, etc., this paper proposes a heterogeneous image matching method based on a meta-template knowledge base. According to the differences in indicators of heterogeneous images in multiple dimensions, a standardized image meta-template is used to form a knowledge base. The pyramid model is used to adapt to normalized modal processing, and the heterogeneous image matching technology based on a pseudo-twin network is used to achieve accurate matching of heterogeneous images. The relevant key technologies are correct and feasible, and have good application effects:

[0127] (1) Aiming at massive heterogeneous image data, a meta-template-based heterogeneous image knowledge base design method is proposed, which solves the problem of standardized processing of heterogeneous images and realizes the rapid storage, understanding, retrieval and reading process of massive heterogeneous image data.

[0128] (2) Considering the complexity of heterogeneous images, a heterogeneous image pyramid model based on the meta-template knowledge base is proposed. Relying on the results of the meta-template knowledge base, it forms the ability to normalize massive heterogeneous images.

[0129] (3) According to the timeliness requirements of heterogeneous image data recognition and matching, and considering the differences in radiation and geometric features of heterogeneous images, a heterogeneous image matching technology based on a pseudo-twin network is proposed. Through training and testing of coarse and fine matching of heterogeneous images and analysis and judgment of false matching elimination, it has the ability to recognize and match heterogeneous images with high precision and timeliness.

[0130] The above is only a preferred embodiment of the present invention, and the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A heterogeneous image matching method based on meta-template knowledge base, Features: The steps include: S1. Constructing a meta-template-based heterogeneous image knowledge base, including the following sub-steps: S11. Establish a structure for building a knowledge base based on meta-template standards; Meta template structure, including dataset, dataset series, metadata, metadata element, metadata entity, metadata subset, core metadata, full set metadata, extended metadata, dedicated standard, metadata dictionary and metadata hierarchy; S12, forming a meta-template knowledge base of heterogeneous images; The meta-template knowledge base includes 6 meta-data sets, including image meta-database, image identification information, image quality information, image reference system information, image content information and image source information; S2. Segment the heterogeneous image and extract the underlying features to extract the underlying features of the image, specifically: S21, segmenting the heterogeneous image, specifically: Adaptively process the pixels and roughness in the image, determine the quantization number of the image color, and obtain the extreme value of the color quantization LUV component; divide the LUV component equally according to the color quantization number to form the initial color quantization value of the image, and form the metadata element of the color quantization value of the image; Perform region growth and merge each sub-region according to the normalized mixed distance. When the merging stop condition is reached, the automatic segmentation of the image stops. After the automatic segmentation stops, the image sub-regions are merged with manual assistance to segment and form the image metadata root entity of the image and the segmented region set of the image; S22, extracting underlying features of the image, and storing the extracted underlying features of the image in a meta-template knowledge base of heterogeneous images; S3. Carry out image feature mapping, specifically: S31, using the feature data in the meta-template knowledge base as a training sample data set, using an inductive learning method to construct a mapping rule for the metadata root entity, and using the rule as a decision rule file for template root entity parsing; S32, using the underlying feature data of the image in the meta-template knowledge base as the input of the mapping, and the determination rules of the metadata root entity data and the description file for the entity form the output of the mapping; S33, normalizing the metadata root entity data; after the normalization process, discretizing each data, and forming a discrete description for each value, to obtain discrete description data; S34, constructing a decision tree using the discretized description data and generating a judgment rule for a corresponding entity; using the discretized entity description data to form a corresponding feature description table; S35, performing higher-level semantic analysis to obtain a reasonable understanding of the image feature mapping; S4. Constructing a heterogeneous image pyramid model based on a meta-template library knowledge base, specifically: constructing the image resolution layer by layer using a magnification method or a specified resolution method, the resolution of the pyramid gradually decreases from the bottom layer to the top layer, and the image area range represented remains unchanged, thereby forming multiple resolution levels, including the following sub-steps: S41, performing heterogeneous image domain division, that is, determining the domain to which the multi-source heterogeneous images belong; S42, determining the initial resolution of the image according to the meta-template knowledge base ; S43, constructing the resolution of each layer of the pyramid model, generating an adaptive magnification matrix according to the image adaptation requirements to construct a layered pyramid model; S44, generating a linear quadtree index of the pyramid model, and performing spatial recursive decomposition; The quadtree is a tree structure in which each non-leaf node has at most four branches. The quadtree structure is a hierarchical data structure whose common feature is spatial recursive decomposition. S5. Construct a heterogeneous image matching network based on a pseudo twin network, specifically using pseudo twin network structure matching, including three parts: a coarse matching network, a fine matching network, and a mismatch elimination network; S6, performing heterogeneous image matching training, specifically: during training, the data set is first input into the fine matching network, and then the coarse matching network and the wrong matching elimination network are trained based on the training results of the fine matching network; S7. Perform heterogeneous image matching prediction. Specifically, during prediction, the large image is first input into the coarse matching network to obtain the possible suitable matching area, and then the fine matching network predicts the matching points. Finally, the final high-precision but sparse matching area is obtained through the mismatch elimination network.

2. The heterogeneous image matching method according to claim 1, Features: S1 specifically includes: establishing a structure for building a knowledge base based on the meta-template standard and forming a meta-template knowledge base for heterogeneous images; S11 The data set is a data set that can be identified; the data set series is a set of several data sets that adopt the same specification and can be uniformly classified; Metadata is data that describes data and can describe the attribute information of data; Metadata elements are the basic units of meta templates, equivalent to an attribute in the Unified Modeling Language. Metadata elements are unique in the same metadata entity. The metadata entity includes a set of one or more metadata elements, which is a set of metadata elements that describe the same type of data features; A metadata subset is a collection of multiple related metadata entities and elements; Core metadata is the minimum metadata elements that must be retained when making a basic description of a dataset and is part of the full set metadata elements; Full set metadata refers to all metadata elements used to describe a data set in detail; Extended metadata refers to metadata entities or metadata elements that are extended using consistent principles when the full set metadata cannot satisfy the application. The dedicated standard is a standard for selecting subsets, entities and elements from the entire set of metadata to meet the needs of specific application fields or users; the metadata dictionary uses a certain hierarchical language structure to define and explain the relationships between metadata subsets, entities and elements, and uses names / roles, definitions, constraints, maximum number of occurrences, data types and value ranges to define and explain metadata classes and attributes.

3. The heterogeneous image matching method according to claim 1, Features: The image metadata database in S12 is a root entity that defines various image information resource meta-templates; the image identification information is the data set identification information of the image; the image quality information provides overall evaluation information of the image data quality for the image data quality information; the image reference system information provides a description of the spatial and temporal reference system used by the image data for the image reference system information; the image content information provides element classification information for the image content information and describes the data layer and image data characteristics; the image source information provides the platform information for obtaining the image data, sensor type information, sensor feature information, image correction information, data log information and related attribute information of the image product for the image source information.

4. The heterogeneous image matching method according to claim 1, Features: In S21, the mixed distance includes the color distance and the roughness distance; the segmented region set of the image includes a plurality of metadata root entity data, and there is a mutual correlation relationship between the data.

5. The heterogeneous image matching method according to claim 1, Features: S22 specifically includes: in terms of color, extracting the color mean; in terms of texture, extracting the entropy, contrast, energy and contrast moment of the grayscale co-occurrence matrix; in terms of shape, extracting the shape invariant moment features of the segmented area; and storing the underlying features of these images as the entity parsing module in the meta-template knowledge base of heterogeneous images.

6. The heterogeneous image matching method according to claim 1, Features: The normalization in S33 avoids a certain component from being too large, so that each component has the same influence in the system; S35 specifically includes: for each region after the image segmentation, obtaining the mapping result of the region according to the underlying features of the color, texture and shape of the region; when there is more than one result, obtaining a candidate result queue; for the metadata root entity data structure, selecting a solution from the candidate solution set of each segmented region, and maximizing the cumulative sum of the path weights connecting these solutions; Based on the meta-template knowledge base, a reasonable understanding of image feature mapping is obtained.

7. The heterogeneous image matching method according to claim 1, Features: S43 uses 2 times the equal step length to form the resolution structure of each layer of the pyramid m Matrix, build a hierarchical pyramid model, specifically: the original resolution of heterogeneous images r 0 As the 0th level of the pyramid, the multiplier m The data in the matrix are all greater than 1, ensuring that the resolution of the high-level pyramid is higher than the resolution of the low-level pyramid. Based on the image resolution of the 0th level of the pyramid, The first layer is generated by combining pixels into one pixel, and the image is divided into blocks to form the first layer image block matrix. The second layer image block matrix is ​​generated by the same method based on the first layer until n Layer pyramid model.

8. The heterogeneous image matching method according to claim 1, Features: S44 specifically includes: calculating the lower left pixel coordinates, the upper right pixel coordinates and the coordinates of the image block to which the pixel belongs respectively according to the image block size in combination with the position coordinates of the image block; calculating the number of rows and columns of the image block matrix according to the number of rows and columns of the pixel matrix, and calculating the total number of blocks in each layer; calculating the storage offset according to the starting offset of each layer, the number of bytes of the image block and the storage offset of the image block.

9. The heterogeneous image matching method according to claim 1, Features: The indexing process of S44 is as follows: Extract the image patches that overlap with the regional coverage from the layer to obtain the coordinates of the bottom-left pixel and the top-right pixel of the l layer, calculate the storage offset of the image patch, and complete the indexing calculation of the linear quadtree.

10. The heterogeneous image matching method according to claim 1, Features: S5 The coarse matching network and the fine matching network both adopt a pseudo-twin network structure; the mismatch elimination network uses a combination of multiple convolutional and pooling layers to transform the mismatch elimination task into a heat map binary classification problem, and eliminates mismatches by setting the network output probability confidence to filter the distributed diffuse heat map.

Images