A paradigm graph embedding method of multi-spectral full-element features

Abstract

The present application belongs to the field of image processing, and discloses a paradigm graph embedding method of multispectral full-element features, which is used to obtain three aspects of feature maps of multispectral data, and the specific steps are as follows: firstly, the spatial information and spectral information features in the multispectral image are fused, and then the fused feature information is subjected to data feature extraction and dimension reduction through a method based on manifold learning and space spectrum embedding. The infrared band spectral data, spatial pixels and spectral dimension pixel information in the reduced multispectral data are respectively subjected to infrared spectral band feature extraction, spatial superpixel segmentation technology and construction of a spectral feature graph adjacency matrix, so that the physical feature map of spectral data, the spatial feature map and the spectral feature map are obtained, and finally the paradigm graph embedding method of full-element features of multispectral data is realized. The method can obtain more dimensional feature information and improve the model capability for processing multispectral images.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a paradigm graph embedding method for multispectral full-element features. Background Technology

[0002] Multispectral images are images containing information from multiple spectral bands. Unlike single-band images, multispectral images provide more comprehensive and accurate information, including reflectance, emissivity, and absorptivity, and can acquire rich spatial and spectral information. Multispectral images contain both spatial and spectral information about the target, overcoming the limitations of single-modal images.

[0003] Graph embedding technology refers to embedding a graph into a vector space, representing it as a low-dimensional vector, while preserving the graph's structural information. Graph neural networks can then be used to perform feature processing on the established graph structure, and it is now widely used in various downstream tasks such as image classification and object detection.

[0004] Traditional feature extraction methods for multispectral images typically extract relatively simple and coarse features, failing to adequately represent the complex information of the image. They also perform poorly for nonlinearly related features, have limitations in representing multi-scale features, exhibit poor model generalization ability, and have limited feature dimensionality. Therefore, it is necessary to propose a method for extracting rich multispectral features. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a paradigm map embedding method for multispectral full-element features. The method can extract three dimensions of information from multispectral images: physical features, spatial features, and spectral features, and construct feature maps. This provides a new method to address the previous underutilization of multispectral information dimensions.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0007] A paradigm graph embedding method based on multispectral full-element features, characterized by the following steps:

[0008] (S1) Obtain multispectral images of N bands using a multispectral camera;

[0009] (S2) The fusion form of band array coding and spatial pixel neighborhood coding of multispectral image is represented by augmented vector representation;

[0010] (S3) The fused data information is used to complete the weight allocation of the similarity of the spectral features of different pixels by using spatial spectral embedding, and the local neighborhood space and spectral information are classified and feature dimensionality reduced by manifold learning;

[0011] (S4), using the dimension-reduced spectral data, combine with the infrared spectral features to extract the physical feature map of the spectral data;

[0012] (S5), using the dimension-reduced spectral data, perform feature transformation on the spatial pixel information of the multispectral data. The superpixel neighbor node information can be determined by using the linear iterative clustering method. Based on the spatial connectivity relationship of the superpixels, construct the edge connection relationship between the nodes to extract the spatial feature map;

[0013] (S6), using the dimension-reduced spectral data, combine with the spectral feature similarity of the target, design the adjacency matrix, sample and reorganize from different spectral band dimensions to obtain the target spectral feature distribution, and effectively represent the spectral data residing on the smooth manifold by using the graph neural network.

[0014] Preferably, in step (S1), the multispectral camera is an imaging camera that can simultaneously collect 3 or more spectral bands.

[0015] Preferably, in step (S1), the N is an integer greater than or equal to 3.

[0016] Preferably, in step (S3), L labeled augmented vectors of the fused spectral and spatial information are used as training data (where 3 < L < 10); after normalization, the elements in the augmented vector are classified into the same category in the supervised mode, and the pixel local neighborhood is constructed by using the k-nearest neighbor algorithm. Further combine with the manifold learning framework to encode the manifold local topology of the neighborhood data.

[0017] Preferably, in step (S4), the physical features can be equivalent temperature, equivalent area, and the physical features can be represented as a graph by graph embedding methods such as random walk.

[0018] Preferably, in step (S5), the spatial feature map is as follows: first perform superpixel segmentation on the multispectral image, calculate the spatial distance and spectral distance between pixels, update the clustering center position and range by balancing the weights of the spatial distance and the spectral distance and then perform iteration to obtain the multispectral image composed of superpixels. Connect the adjacent superpixel points to form edges, and construct a spatial graph from the edges.

[0019] Preferably, in step (S6), the spectral feature map can be constructed by constructing unsupervised, supervised, and semi-supervised adjacency matrices.

[0020] Compared with the prior art, the present invention provides a paradigm graph embedding method for multispectral all-element features, and the beneficial effects are as follows:

[0021] The proposed method extracts physical, spatial, and spectral features from multispectral images to construct a feature map. Traditional multispectral image feature extraction methods typically extract relatively simple features, failing to adequately represent the complex information of the image. They also have limitations in representing multi-scale features, exhibiting poor model generalization ability and limited feature dimensionality. To address these limitations, this method fully utilizes the physical, spatial, and spectral features extracted from multispectral images, enabling the acquisition of deeper image information and the extraction of multidimensional multispectral information, thereby improving the generalization ability of subsequent model processing. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the paradigm diagram embedding of multispectral full-element features according to the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0024] See Figure 1 The present invention provides a paradigm graph embedding method based on multispectral full-element features, comprising the following steps:

[0025] (S1) Obtain multispectral images of N bands using a multispectral camera;

[0026] (S2) The fusion form of band array coding and spatial pixel neighborhood coding of multispectral image is represented by augmented vector representation;

[0027] (S3) The fused data information is used to complete the weight allocation of the similarity of the spectral features of different pixels by using spatial spectral embedding, and the local neighborhood space and spectral information are classified and feature dimensionality reduced by manifold learning;

[0028] (S4) Using the dimensionality-reduced spectral data, combine infrared spectral features to extract the physical feature map of the spectral data;

[0029] (S5) Using the dimensionality-reduced spectral data, the spatial pixel information of the multispectral data is transformed into features. The linear iterative clustering method can be used to determine the superpixel neighbor node information. Based on the spatial connectivity of the superpixels, the edge connection relationship between nodes is constructed, and the spatial feature map is extracted.

[0030] (S6) Using the dimensionality-reduced spectral data and combining the similarity of the target's spectral features, an adjacency matrix is ​​designed. The target's spectral feature distribution is obtained by sampling and recombining from different spectral band dimensions. The spectral data residing on the smooth manifold is effectively represented by a graph neural network.

[0031] See Figure 1In step (S1), the acquired multispectral data consists of multispectral images of four bands.

[0032] See Figure 1 In step (S2), the augmented vector of the fused spectral and spatial information can be represented by the following formula:

[0033] x=(u,v,b1,b2,...,b B )=(x 1 ,x 2 ,...,x B+2 ) T (1)

[0034] Where h(u,v) is a pixel on the image grid plane, (b1,b2,b3,b...) B () is a band array.

[0035] In addition, in this embodiment we obtained images of 4 bands, so B=4.

[0036] See Figure 1 In step (S3), the feature extraction and dimensionality reduction of the spatial spectral information embedding and manifold learning are performed using augmented vectors of L labeled pixels. As training data, after normalization, for any x i In supervised mode, classify the same category and construct local neighborhoods of pixels using the k-nearest neighbor algorithm. Then, combine the manifold learning framework to encode the local topology of the neighborhood data for feature dimensionality reduction.

[0037]

[0038] Among them, y i For x i The learned m-dimensional features, {W ij} represents the input data and Di = ∑ j W ij Positive weights of similarity between them, original x i and x j The feature similarity constraint between them can be obtained by dimensionality reduction of y i and y j The characteristics between them are maintained by a Euclidean distance metric.

[0039] In a specific embodiment, the augmented vector contains 6 labeled elements.

[0040] Furthermore, the weights based on the spatial spectrum polynomial local domain embedding can be calculated using Laplacian embedding and locality-preserving projection, i.e.:

[0041]

[0042] See Figure 1 In step (S4), the physical feature map, including the physical features of equivalent temperature and equivalent area, is represented as a graph by a random walk graph embedding method.

[0043] See Figure 1 In step (S5), in the spatial feature map, the multispectral image is first segmented into superpixels using the SLIC algorithm. By calculating the spatial distance and spectral distance between pixels and balancing the weights, the superpixel cluster centers and range boundaries are iteratively updated. The iteration stops when the error between the new cluster center and the old cluster center is less than a certain range, resulting in a multispectral image data composed of superpixels. The edge connection relationship between nodes is constructed based on the spatial connectivity relationship of the superpixels.

[0044] Specifically, the expressions for spatial distance and spectral distance are:

[0045]

[0046]

[0047]

[0048]

[0049] In the formula, d c For the spectral squared difference, d(S) x ,S y ) represents the spectral angular distance, d s The square of the distance is balanced by adjusting m, which is typically 50.

[0050] In a specific embodiment, the number of iterations is 15, and the error between the cluster center and the old cluster center is less than 0.01.

[0051] See Figure 1 In step (S6), the spectral feature map is constructed by a semi-supervised adjacency matrix method.

[0052] Specifically, this method is built upon information provided by a limited amount of labeled data and a large amount of unlabeled data. It uses a Dirichlet process hybrid model based on variational inference to construct pseudo-labels and constructs a spatial spectral adjacency matrix based on the intrinsic clustering algorithm in the data samples.

[0053] Ultimately, three types of physical feature maps, spatial feature maps, and spectral feature maps were constructed, thus realizing a paradigm graph embedding method for full-element features of multispectral data.

[0054] The above describes preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above description. All technical solutions falling within the spirit and principle of the present invention are within the protection scope of the present invention. It should be noted that any improvements and modifications made without departing from the principle of the present invention should also be considered within the protection scope of the present invention.

Claims

1. A paradigmatic graph embedding method for multi-spectral all-element features, characterized in that, The method comprises the following steps: (S1), obtaining a multi-spectral image of N bands by using a multi-spectral camera; (S2), representing a fusion form of band array coding of the multi-spectral image and spatial pixel neighborhood coding of the multi-spectral image by an augmented vector; (S3), completing weight distribution of similarity of different pixel spectral features by using space-spectrum embedding on the fused data information, and performing similarity classification and feature dimension reduction on local neighborhood space and spectral information by manifold learning; (S4), based on the obtained low-dimensional spectral embedding representation, mapping physical features of the multi-spectral data, and extracting physical features reflecting target physical properties, wherein the physical features include equivalent temperature and equivalent area; introducing random walk, spectral embedding or equivalent graph embedding method into the physical features as node attributes, representing the physical features in a graph structure, and generating a spectral physical feature graph for describing distribution relationship of target physical properties; (S5), based on the obtained low-dimensional spectral embedding representation, performing feature conversion on spatial pixel information of the multi-spectral image, constructing a spatial feature graph, and performing superpixel segmentation on the multi-spectral image, taking spatial position and corresponding low-dimensional spectral feature of a pixel as clustering features, and using a linear iterative clustering method to calculate weights according to spatial distance and spectral distance between pixels, and updating positions and ranges of clustering centers through iteration to obtain a multi-spectral image composed of multiple superpixel regions; taking the superpixel regions as nodes in a graph, constructing edge connection relationship between nodes according to spatial connectivity relationship between adjacent superpixels, and generating a spatial feature graph for representing spatial structure relationship of the multi-spectral data; (S6), based on the obtained low-dimensional spectral embedding representation, constructing a spectral feature graph for describing spectral relationship of the multi-spectral data in combination with spectral feature similarity of the target under different spectral bands, taking low-dimensional spectral embedding features of pixels or superpixels as node features in the graph, calculating connection weights between nodes according to spectral feature similarity degrees between different nodes, and constructing a spectral adjacency matrix; wherein the spectral adjacency matrix is constructed in an unsupervised manner, a supervised manner or a semi-supervised manner according to different application scenarios to describe association relationship between different spectral feature nodes; based on the spectral adjacency matrix, a graph neural network is introduced to perform feature propagation and aggregation on the spectral feature graph to obtain an embedding representation for representing internal spectral structure relationship of the multi-spectral data, thereby realizing norm graph embedding representation of the multi-spectral data in a spectral dimension.

2. The multi-spectral full-factorial paradigm graph embedding method of claim 1, wherein, In step (S1), the multi-spectral camera is an imaging camera capable of simultaneously collecting 3 or more spectral bands.

3. The multi-spectral full-factorial paradigm graph embedding method of claim 1, wherein, In step (S1), N is an integer greater than or equal to 3.

4. The multi-spectral full-factorial paradigm graph embedding method of claim 1, wherein, In step (S3), the augmented vectors of the fused spectral and spatial information after L marking are taken as training data, wherein L is greater than 3 and less than 10; after normalization, elements in the augmented vector are classified into the same category in a supervised mode, a local neighborhood of pixels is constructed by a k-nearest neighbor algorithm, and a manifold local topology of neighborhood data is further coded in combination with a manifold learning framework.

Images