Multi-modal remote sensing image segmentation method based on MRF semantic reasoning

Abstract

The invention provides a multi-modal remote sensing image segmentation method based on MRF semantic reasoning. The method is used for solving the technical problem that an existing multi-modal data classification method is difficult to achieve semantic information reasoning of different modal data on the same position. The method comprises the following steps: carrying out successive training learning on multi-modal data to obtain an initial semantic segmentation result of each modal, carrying out deep feature fusion on the multi-modal data to obtain deep fusion features of the multi-modal data, then establishing a self-learning updated multi-modal semantic reasoning model under an MRF framework, and carrying out multi-modal semantic reasoning based on the multi-modal semantic reasoning model. And updating a multi-modal semantic segmentation result through cooperation of the multi-modal depth features and semantic reasoning. According to the method, fusion of multi-modal data features and semantic association information among different modals are fully considered, and accurate reasoning and classification of geographic objects are realized through the synergistic effect of the CNN model and the Markov random field. Experimental results show that the model provided by the invention has better performance on different data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal remote sensing data semantic segmentation, and in particular to a multimodal remote sensing image segmentation method based on MRF semantic reasoning. Background Art

[0002] Remote sensing data obtained by different sensors have great differences in spectral characteristics, resolution, etc. These data provide multi-angle and multi-level radiation information of the observation scene. Due to the complexity of the types and distribution of land objects, it is difficult to accurately interpret remote sensing images using only data from a single sensor. In order to solve this problem, many researchers have begun to focus on the fusion processing of multimodal data to provide more comprehensive and multi-dimensional earth observation data. Hyperspectral imaging (HSI) expands the spectral dimension of remote sensing data. Different spectral bands can show sensitivity to different land objects, so that HSI can effectively distinguish different types of land cover, including vegetation, water bodies, buildings, etc. Therefore, HSI data has a wide range of applications in environmental monitoring, agricultural management, natural resource surveys and other fields. Synthetic aperture radar (SAR) and laser radar (LiDAR) are both typical active remote sensing technologies that detect scenes by emitting electromagnetic waves. SAR can transmit microwave signals without being affected by cloud cover and can observe the surface in bad weather such as cloudy days and insufficient light conditions such as night. Therefore, SAR can provide all-weather surface information, which gives it unique advantages in forestry, agriculture, urban planning and other fields, and can be used for terrain measurement, change detection and ocean monitoring. LiDAR obtains distance information between the ground and objects by emitting laser signals to the ground and measuring the time for the light beam to return. This high-precision distance measurement makes LiDAR widely used in terrain modeling, urban planning, forest survey and other fields. LiDAR data can generate high-precision digital elevation models and provide detailed terrain information. It can also be used for three-dimensional modeling of objects such as buildings and trees. The comprehensive use of multi-source remote sensing data such as hyperspectral, SAR and LiDAR can integrate the complementarity between different modal data, make up for the limitations of single sensors in observation content and observation time, and achieve more comprehensive earth observation. This multimodal data fusion method provides a more comprehensive and three-dimensional information basis for applications such as accurate land object classification, environmental monitoring and urban planning, but it also brings challenges in data fusion and semantic information extraction.

[0003] Semantic segmentation based on remote sensing data is one of the core tasks of intelligent interpretation of remote sensing images. Its goal is to mark each pixel in the image as a semantic category with a certain property. Semantic segmentation with the help of computers can help people interpret surface features in a more detailed and accurate manner. Through semantic segmentation, accurate identification of buildings, roads, green spaces, etc. in cities can be achieved, providing detailed geographic information for urban planning and management. Multimodal remote sensing data can depict surface information from multiple perspectives, but how to perform semantic reasoning on the complementarity between different modal data requires certain theoretical guidance. Existing feature-level fusion and decision-level fusion are usually completed based on data-driven network models. Different modal data can complement each other. For example, the recognition of buildings in hyperspectral images may be affected by surrounding roads, and the roof and road of the building may be confused from the perspective of features, but lidar images can accurately detect the elevation information of the objects, which can help distinguish these different objects with the same appearance features. However, current methods usually fuse multimodal data based on deep learning models, and most of them lack clear strategies and theoretical frameworks for the interaction of semantic information between multimodal data. As a data and knowledge driven method, the Markov random field (MRF) model uses the theoretical framework of MRF to interact semantic information between different modalities to promote the intelligent interpretation of multimodal data, which is a difficult and hot issue worthy of study.

[0004] The invention patent with application number 202410200897.6 discloses a remote sensing image semantic segmentation method based on multimodal data fusion, including: obtaining multimodal remote sensing data of the target area; the multimodal remote sensing data includes at least two different types of remote sensing image data; preprocessing the multimodal remote sensing data to obtain preprocessed data; extracting features from the preprocessed data of each modality to obtain features of each modality; fusing the features of each modality to obtain fused features; inputting the fused features into a trained semantic segmentation network to obtain segmentation results. The above invention can make full use of the complementarity of each modality data to improve the accuracy and robustness of semantic segmentation. However, the above invention improves the results by fusion of features of multimodal remote sensing data, without considering the reasoning relationship between the semantic information corresponding to the multimodal data, and considering that the features and semantic information of the multimodal data can further improve the accuracy and interpretability of the results. Summary of the invention

[0005] In view of the technical problem that the existing multimodal data classification methods are difficult to realize the semantic information reasoning of different modal data for the same location, the present invention proposes a multimodal remote sensing image segmentation method based on MRF semantic reasoning. Under the theoretical framework of Markov random field, the multimodal deep fusion features are combined with the semantic information reasoning corresponding to different modal data. Firstly, high-quality fusion features are extracted by deep learning method, and the single modal data is initially classified one by one by a semantic segmentation network based on convolutional neural network (CNN). On this basis, the reasoning unit of geographic object is constructed. The initial result obtained by the deep learning network can make the corresponding reasoning unit have basic semantic information. At the same time, the semantic relationship matrix constructed by self-learning strategy is used to capture the dynamic interactive relationship of semantic knowledge between different modalities, so as to improve the modeling ability of spatial information, thereby improving the accuracy and consistency of classification results.

[0006] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a multimodal remote sensing image segmentation method based on MRF semantic reasoning, the steps of which are as follows:

[0007] Step 1: Read different modal data of multimodal remote sensing images, use CNN to train each modal image data separately, and obtain the initial semantic segmentation results of different modal data;

[0008] Step 2: Multimodal image data is deeply fused through a multimodal feature fusion classification network to obtain multimodal deep fusion features; multiple inference units are generated based on the initial semantic segmentation results of any single modality, and an MRF model is constructed based on the inference units;

[0009] Step 3: The initial semantic segmentation results of different modalities obtained in step 1 and the deep fusion features of multimodal data obtained in step 2 are respectively used as the label field and feature field in the MRF model to model a multimodal MRF model;

[0010] Step 4: By self-learning to update the interactive relationship between different modal semantics and the spatial contextual relationship within the same modality, the multimodal MRF model established in step 3 is iteratively solved to obtain the semantic segmentation result of multimodal data.

[0011] Preferably, the method for obtaining the initial semantic segmentation result in step 1 is: different modal data I of the multimodal remote sensing image a =(I 1 ,I 2 ) is m×n in size, and the image data I 1 ,I 2 The number of spectral bands are d1, d2, and d1, d2∈N +, N + is a set of positive integers; CNN is used to image data I 1 ,I 2 The initial semantic segmentation results of different modalities are obtained by training and learning respectively and recorded as Y = (Y 1 ,Y 2 ), and Y (l) The label fields corresponding to different modalities of deep fusion features, l = {l1,l2}, l1,l2∈{1,2};

[0012] The method for obtaining multi-modal deep fusion features is as follows: for image data of different modalities with N pixels in each band, the qth layer of features learned through the CNN network is represented as:

[0013]

[0014] in, represents the feature of the cth pixel in the lth mode in the qth layer, I l,c represents the feature of the cth pixel in the lth mode, p is the total number of layers of the CNN network; the function f(·) is the ReLU activation function, and They are weight parameters and bias terms respectively;

[0015] The feature-level cross-reconstruction strategy is selected to fuse the features of different modalities. The fusion method is:

[0016]

[0017] in, Represents the pixel-level fusion feature of the qth layer, pixel-level fusion feature is the deep fusion feature I′, Represents the features of the cth pixel in the lth modality in the qth layer.

[0018] Preferably, the method for generating multiple inference units is: using a mean shift algorithm to divide the initial semantic segmentation result of any single modality to obtain inference units, and the deep fusion features corresponding to the position of each inference unit are used as data features of the inference unit;

[0019] The MRF model is a graph model.

[0020] Preferably, the method for constructing the MRF model according to the inference unit is: the acquired inference unit is represented as R={R s ,s∈S}, each reasoning unit R s Represents a vertex v in the graph model s , each vertex corresponds to an area of ​​the image, and the graph model is constructed as a graph G = {V, E}, where V = {Rs , s∈S} represents the vertex set, S is the position index set of each vertex, E={e s,t |s,t∈S} represents the edge set between vertices, and edge e s,t Represents vertex v s and vertex v t The spatial relationship of s and vertex v t If the edge e is adjacent, s,t = 1, if vertex v s and vertex v t If they are not adjacent, then edge e s,t =0.

[0021] Preferably, the method for obtaining the semantic segmentation result of multimodal data is: using a self-learning strategy to obtain the interaction between two points in the energy function describing the second-order potential function of the MRF model in the multi-layer logical model, so as to achieve modeling of spatial relationships within a single modality; calculating the interaction relationship between labels through conditional probability to achieve reasoning between multimodal semantics, and using an object-level MRF model to obtain the semantic segmentation result of multimodal data.

[0022] Preferably, the method of using the object-level MRF model to obtain the semantic segmentation result of multimodal data is:

[0023] The deep fusion feature is used as the feature field I = {I s |s∈S}, the labeling fields of different modalities l={l1,l2},l1,l2∈{1,2},and l1≠l2,I s Represents vertex R s The eigenvector of s The dimension of is determined by the deep fusion feature I′; the label variable is a random variable representing the mark at position s in the lth modality, corresponding to the initial semantic segmentation result Y 1 ,Y 2 , label variable Take values ​​from the category label set Λ={1,2,…,K}, where K represents the total number of semantic categories and S={s} represents the location index set of image data; Represents a random field of markers A specific implementation of , then for the MRF model of multimodal data semantic reasoning proposed in the present invention, the optimal implementation of different modalities is achieved through the objective function method:

[0024]

[0025] in, They represent the optimal realization of different modes, They represent the posterior probabilities of the labeling results of different modalities under the data feature conditions, Represent the feature likelihood functions of different modes respectively; They represent the prior probabilities of different modal labeling results respectively; are the conditional probabilities of the current modality under the labeling of another modality; They respectively represent the feature information of multimodal image data based on the labeling results of different modalities, represents the intra-layer relationship of the l1th modal semantic layer, represents the intra-layer relationship of the l2th modal semantic layer, They represent the semantic interaction information between different modalities respectively.

[0026] Preferably, for the multimodal feature field, the feature information of the multimodal image data is expressed as:

[0027]

[0028] After the division of the reasoning unit, it is further expressed as R = {R s |s∈S}, s is the position node, when vertex v s The marker variable When h represents a value in the category label set Λ={1,2,…,K}, assuming that the feature vectors of the pixels contained in each over-segmented region as the inference unit are independent and identically distributed and obey the Gaussian distribution, then the likelihood function of the object feature vector is equivalent to the product of the likelihood functions of the feature vectors of the pixels contained in the object, that is, the conditional probability It is expressed as:

[0029]

[0030] So the characteristic information:

[0031]

[0032] Among them, d is the number of bands of deep fusion feature I′, μ h and Σ h Respectively represent the mean and variance of the feature marked as h; I′ s , I′ t represent the spectral values ​​at position s and position t respectively, Represents the probability of the feature distribution when the label at position s is h.

[0033] Preferably, the intra-layer relationship E2(X l ) is calculated as:

[0034]

[0035] in,

[0036]

[0037] in, represents the spatial contextual relationship of position s in the lth modal semantic layer, and the intra-layer relationship E2(X l ) is determined by the spatial context of each position in the layer Composite, represents the potential function of the double-point group at position s and its neighboring position t, N s ={t|t∈S,e s,t =1} represents the set of neighboring locations of location s, denote the tags of the lth modal semantic layer at position s and position t, respectively, and e s,t represents the edge between position s and position t, e s,t =1 means that position s is adjacent to position t, b s,t Indicates region R s and region R t The length of the common boundary of represents the spatial relationship between category i and category j, Represents the weight of the spatial relationship between category i and category j in the tagging result of the l-th modal semantic layer.

[0038] Preferably, the spatial relationship between category i and category j is:

[0039]

[0040] in,

[0041]

[0042] m i,j represents the penalty coefficient, represents the probability that the l1th modal semantic layer is labeled as i when the l2th modal semantic layer at position s is labeled as category j, represents the number of pixels labeled as i in the l1th modal semantic layer when the l2th modal semantic layer at position s is labeled as category j, Represents the number of pixels labeled j at the l2th modality semantic layer at position s;

[0043] The weight of the spatial relationship between category i and category j in the tagging result of the lth modal semantic layer for:

[0044]

[0045] Preferably, for any position s∈S, the semantic interaction information The relationship between different modal markers is expressed as follows:

[0046]

[0047] Among them, α is the weight relationship between different modal categories, and its value is the average area of ​​the category:

[0048]

[0049] γ i,j Represents the distance between markers, and the discrete distance calculation method is:

[0050]

[0051] The beneficial effects of the present invention are as follows: first, based on the deep learning model, the multimodal data is trained and learned successively to obtain the initial semantic segmentation results of different modalities respectively, and the deep fusion features of the multimodal data are extracted with the help of the deep learning model and the multimodal data is subjected to deep feature fusion. Then, under the theoretical framework of MRF, the multimodal deep fusion features are combined with the semantic reasoning between different modalities to establish a self-learning and updated multimodal semantic reasoning model, and the semantic segmentation results driven by the multimodal features and the semantic reasoning between different modalities are obtained. The multimodal semantic segmentation results are updated by the multimodal deep features and the semantic reasoning, which improves the modeling ability of the MRF model for spatial information and the accuracy of the classification results, thereby making certain progress in the semantic reasoning of multimodal data. The semantic segmentation model driven by multimodal data and multimodal knowledge reasoning proposed in the present invention can comprehensively utilize deep learning and Markov random field theory, and designs a new remote sensing image semantic segmentation framework that combines multimodal data features with semantic reasoning. By comparing with other different types of multimodal data classification models, the results verify the effectiveness of the model.

[0052] The present invention has made certain innovations and progress based on the combination of multimodal deep fusion features and Markov random field theory. The present invention fully considers the fusion of multimodal data features and the semantic association information between different modalities in its design. Through the synergy of the CNN model and the Markov random field, accurate reasoning and classification of geographic objects are achieved. Experimental results show that the proposed model has achieved good performance on different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a flow chart of the present invention.

[0055] Figure 2 It is a schematic diagram of the multimodal MRF semantic reasoning model of the present invention.

[0056] Figure 3 A schematic diagram of the initial semantic classification result obtained by the present invention.

[0057] Figure 4 A schematic diagram of obtaining multi-modal deep fusion features according to the present invention.

[0058] Figure 5 This is the classification result of the present invention on the Houston 2013 dataset.

[0059] Figure 6 This is the classification result of the present invention on the Berlin dataset. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] like Figure 1 As shown in the figure, a multimodal remote sensing image segmentation method based on MRF semantic reasoning is firstly to learn the multimodal data separately through deep learning method to obtain the initial semantic segmentation result of the multimodal data; secondly, the image data features are fused through deep learning method, and the inference units of each geographic object on the fusion features of different modalities are extracted respectively with the obtained initial semantic segmentation results; finally, an MRF multimodal reasoning model is established based on the obtained inference units, and the semantic reasoning of the same modality and different modalities is performed between the classification results of different modalities by combining the fusion features of multimodal. The steps of this method are as follows:

[0062] Step 1: Read different modal data of multimodal remote sensing images, use CNN network to train each modal image data separately, and obtain the initial semantic segmentation results of different modal data.

[0063] Different modal data of multimodal remote sensing images I a =(I 1 ,I 2 ) is m×n in size, the number of semantic categories is K, and the image data is I 1 ,I 2The number of spectral bands are d1, d2, and d1, d2∈N + , N + is a positive integer set; let image data I 1 is a hyperspectral image HSI with spectral band number d1, and the image data of another modality I 2 The number of spectral bands is d2, such as Lidar or SAR, generally d2 = 1, that is, Lidar or SAR, image data I 2 It is a single-band full-color image that reflects the elevation information of the surface of the object. Figure 3 As shown, the CNN network is used to analyze the image data I 1 ,I 2 The initial semantic segmentation results of different modalities are obtained by training and learning respectively and recorded as Y = (Y 1 ,Y 2 ), and Y (l) , l={l1,l2}, l1,l2∈{1,2} corresponds to the label fields of different modes of deep fusion features. The initial semantic segmentation results obtained by the CNN network have basic semantic information, which can help the model generate reasoning units with basic geographical meanings. At the same time, these initial results provide initial semantic segmentation results for subsequent semantic reasoning.

[0064] Step 2: Multimodal image data is subjected to deep feature fusion through a multimodal feature fusion classification network to obtain multimodal deep fusion features; multiple inference units are generated based on the initial semantic segmentation results of any single modality, and the deep fusion features corresponding to each inference unit are used as a vertex in the MRF model.

[0065] The multimodal image data is trained and learned using a multimodal feature fusion classification network to obtain the features of data of different modalities, and the features of data of different modalities are fused to obtain the deep fusion feature I′.

[0066] After the hyperspectral image and radar data are combined, the dimension is often high. In the present invention, the fusion feature of each pixel in the image is obtained through the deep learning module. Assume that an image data I of size m×n l , the fusion features of each pixel in the image data are obtained through the CNN network used for multi-modal feature fusion in the deep learning module. Image data of different modalities l∈{1,2} is used to mark the information of different data. l=1 represents the first mode, and l=2 represents the second mode. It has d1 and d2 band information respectively, indicating the number of spectral bands of different modes. Each band has N pixels of two-mode image data. The t-th layer feature learned by CNN network can be expressed as:

[0067]

[0068] in, represents the feature of the cth pixel in the lth mode in the qth layer, I l,c represents the feature of the cth pixel in the lth mode, and p is the total number of layers of the CNN network. The function f(·) is defined as the ReLU activation function, which is the activation function of the network parameters to be estimated. and are weight parameters and bias terms, respectively, which can be obtained through CNN network training. The present invention selects a feature-level cross-reconstruction strategy to fuse features of different modalities, and the fusion process can be expressed as:

[0069]

[0070] in, represents the pixel-level fusion feature of the qth layer, which is the deep fusion feature I′. Represents the features of the cth pixel in the lth modality in the qth layer.

[0071] In order to generate the basic unit of semantic reasoning between different modalities, the initial semantic segmentation result of any single modality of hyperspectral, Lidar or SAR is used as the object of division of reasoning unit. The classification result of hyperspectral image is selected to generate reasoning unit, and the mean shift algorithm (MS) is used to divide the initial semantic segmentation result to obtain reasoning unit. The multimodal data features corresponding to the position of each reasoning unit are used as the data features of the subsequent reasoning unit, which are subsequently used to describe the representation of data features in the MRF model. The obtained reasoning unit is represented as R = {R s ,s∈S}, each reasoning unit R s Represents a vertex v in the graph model s In the present invention, each vertex corresponds to an area of ​​the image. The graph model in the present invention can be constructed as a graph G = {V, E}, where V = {R s , s∈S} represents the vertex set of the graph model, S is the position index set of each vertex, E={e s,t |s,t∈S} represents the edge set between vertices, and each edge e s,t Represents vertex v s and v t The spatial relationship of s and v t Adjacent, then e s,t = 1, if vertex v s and v t If they are not adjacent, then e s,t = 0. The MRF model in the present invention is based on a graph model consisting of the above vertices and edges.

[0072] Step 3: The initial semantic segmentation results of different modalities obtained in step 1 and the deep fusion features of multimodal data obtained in step 2 are respectively modeled as the label field and feature field in the MRF model to obtain a multimodal MRF model, which contains label fields of different modalities and feature fields of multimodal fusion.

[0073] The deep fusion features of multimodal image data and the initial semantic segmentation results of different modalities can be used as the feature field and label field in the MRF model, and then a multimodal MRF model can be established to achieve modeling of the multimodal data label field and fusion feature field.

[0074] Step 4: By self-learning and updating the interactive relationship between different modal semantics and the spatial contextual relationship within the same modality, the MRF model established in step 3 is iteratively solved to obtain the semantic segmentation result of multimodal data. The labeling result corresponding to the position of each inference unit is then used as the labeling result corresponding to the inference unit.

[0075] A self-learning strategy is used to obtain the interaction between two points in the energy function that describes the second-order potential function of the MRF model in the multi-layer logical model, and the modeling of spatial relationships within a single modality is achieved through conditional probability calculation. The reasoning between multimodal semantics is achieved by introducing the interaction relationship between different modal labels, the interaction relationship between labels is calculated through conditional probability, and the object-level MRF model is used to obtain the semantic segmentation results of multimodal data.

[0076] The method of using the object-level MRF model to model and obtain the semantic segmentation result map is: combining the initial segmentation results of different modalities with the multi-modal deep fusion features to establish a self-learning and updating multi-modal semantic reasoning method based on the MRF model. The specific steps are:

[0077] like Figure 2 As shown, the deep fusion feature is used as the feature field I = {I s |s∈S} and the labeling fields of different modalities l={l1,l2},l1,l2∈{1,2},and l1≠l2,that is,when l1=1,l2=2,on the contrary,when l1=2,l2=1. s Represents vertex R s The characteristic vector of s It is a multidimensional vector, the specific dimension is determined by the hyperspectral data I 1 and radar data I 2 The generated deep fusion feature I′ determines each tag variable is a random variable representing the mark at position s in the lth modality, corresponding to the initial semantic segmentation result Y 1 ,Y 2, corresponding to the initial segmentation results of hyperspectral and radar image data, respectively, Take values ​​from the category tag set Λ={1,2,…,K}, where K represents the total number of category tags. Where S={s} represents the location index set of image data. Let Represents a random field of markers A specific implementation of , then for the MRF model of multimodal data semantic reasoning proposed in the present invention, the objective function can be expressed as:

[0078]

[0079] Similarly, we can get:

[0080]

[0081] In the above formula, They represent the optimal realization of different modes, They represent the posterior probabilities of the labeling results of different modalities under the data feature conditions, Represent the feature likelihood functions of different modes respectively, described by mixed Gaussian distribution; They represent the prior probabilities of different modal labeling results respectively, and the establishment process is described in detail below; are the conditional probabilities of the current modality under the labeling of another modality; They respectively represent the feature information of multimodal image data based on the labeling results of different modalities, represents the intra-layer relationship of the l1th modal semantic layer, represents the intra-layer relationship of the l2th modal semantic layer, Respectively represent the semantic interaction information between different modalities, which can be expressed by E3(X l |X 3-l ) indicates that when l = 1, When l = 2, For different values ​​of l1, the corresponding objective function can be In order to obtain the optimal implementation of the labeling field corresponding to different modalities, the multimodal MRF semantic reasoning model proposed in the present invention designs a process of interactive reasoning using the features and semantic information between different modalities through conditional probability, aiming to realize information interaction between different modalities.

[0082] The specific process of objective function modeling and updating is as follows: For the multimodal feature field, that is, the modeling of the feature information of multimodal image data can be expressed as:

[0083]

[0084] After the initial division of the reasoning unit, it can be further expressed as R = {R s|s∈S}, s is the position node, when vertex v s The marker variable When h represents a value in the category label set Λ={1,2,…,K}, it is assumed that the feature vectors of the pixels contained in each over-segmented region as the inference unit are independent and identically distributed. In the present invention, Gaussian distribution is adopted. Then the likelihood function of the object feature vector is equivalent to the product of the likelihood functions of the feature vectors of the pixels contained in the object, that is, It can be expressed as:

[0085]

[0086] thereby:

[0087]

[0088] Among them, d is the number of bands of the fused data feature I′, μ h and Σ h I′ represents the mean and variance of the feature marked as h. s , I′ t represent the spectral values ​​at position s and position t respectively, Represents the probability of the feature distribution when the label at position s is h.

[0089] For the second term E2(X l ), which is used to represent the spatial contextual relationship of the semantics of a single modality image, and is expressed as:

[0090]

[0091] in,

[0092]

[0093] in, represents the spatial contextual relationship of position s in the lth layer of semantics, and the intra-layer relationship E2(X l ) is composed of the spatial contextual relationship of each position in the layer, represents the potential function of the double-point group at position s and its neighboring position t, N s ={t|t∈S,e s,t =1} represents the set of neighboring locations of location s, denote the lth layer mark at position s and position t respectively, e s,t represents the edge between position s and position t, e s,t =1 means positions s and t are adjacent, b s,t Indicates region R s and region R t The common boundary length is used to delineate the region R of the same mode.s and R t the strength of the spatial relationship between the marks, Represents the spatial relationship between category i and category j, which is defined as follows:

[0094]

[0095] in,

[0096]

[0097] m i,j represents the penalty coefficient, represents the probability that the l1 modal layer is labeled i when the l2 modal layer at position s is labeled j, represents the number of pixels marked as i in the l1 modal layer when the l2 modal layer at position s is marked as j, It represents the number of pixels marked as j in the l2 modal layer at position s. From the above formula, we can see that the penalty coefficient m i,j >1, such a coefficient can increase the penalty coefficient between categories with different marking results at the same position between different modalities, thereby promoting the marking of different modalities to take the same value.

[0098] It represents the weight of the spatial relationship between category i and category j in the labeling result of the lth layer, and is used to describe the penalty direction between different labels. Positive direction is used to encourage adjacent vertices to take the same label, and negative direction is used otherwise, as described in the following formula:

[0099]

[0100] Finally, we can further calculate

[0101] Semantic Interaction Information It is used to describe the spatial relationship between different modes. Since the same vertices are selected in the probability graph model defined in the present invention (as described in step 2), for any position s∈S, The relationship between different modal markers can be expressed as follows:

[0102]

[0103] Among them, α is the weight relationship between different modal categories, and its value is the average area of ​​the category:

[0104]

[0105] γ i,j Indicates the distance between markers. In the present invention, a discrete distance calculation method is adopted. Such a definition is more consistent with the definition of markers in remote sensing images. The formula is:

[0106]

[0107] Combining the above formula, we can get semantic interaction information

[0108] By substituting each component of the objective function in step 4 one by one, the marking result of the present invention can be obtained.

[0109] Through the above steps, the present invention firstly learns the multimodal data respectively through the deep learning method to obtain the initial semantic segmentation results of the multimodal data. Secondly, the image features are fused through the deep learning method, and the initial semantic segmentation results obtained in the first step are used to extract the reasoning units of each geographic object on the different modal fusion features. Finally, an MRF multimodal reasoning model is established based on the obtained reasoning units, as described in step two, and by combining the deep fusion features of the multimodal, the semantic reasoning of the same modality and different modalities is performed between the classification results of different modalities, as described in steps three and four.

[0110] In order to distinguish the multimodal data and semantic knowledge reasoning model proposed in the present invention from other multimodal data classification methods, the present invention implements experiments with different methods on two datasets, Houston 2013 and Berlin. The model proposed in the present invention introduces a CNN-based module to perform feature learning and fusion on multimodal data, thereby providing a framework based on Markov random field theory to realize semantic reasoning of different modalities. In order to test the effectiveness and robustness of the proposed MMDSI model, several multimodal algorithms with more applications were selected for quantitative and qualitative comparison. They are classification models of single modality (such as HS, LiDAR and SAR), denoted as model 1, model 2 using stacked features (directly using stacked multimodal data, such as hyperspectral and radar, hyperspectral and SAR), model 3, model 4, model 5, and the quantitative indicators are shown in Tables 1 and 2. In general, when using single modality data for classification, the classification results of model 1 on hyperspectral data are significantly better than those using only LiDAR data. Due to the characteristics of different modal data, it is difficult for the same single modality classification model to obtain good results on different modalities at the same time. Compared with single HSI data and LiDAR data, the method of stacking multimodal features shows better classification performance, which also shows the effectiveness of using multimodal remote sensing data. Model 1 is good at processing multispectral models, so the performance of model 2 by aligning multimodal training data is obviously consistent with the results of model 1 with stacked features. Model 3 constructs multi-layer structure extraction through stacked morphological contours and structural features, extracts spatial information from multimodal original images, and integrates the extracted spatial information using a low-rank model. Therefore, model 3 can better extract multimodal fusion features, and the classification results are also improved compared with the previous four methods. Model 4 based on deep learning can extract more discriminative feature representations and better capture the potential relationship between different modalities. It has achieved good performance in multimodal data fusion and classification. In addition, compared with other state-of-the-art methods, the classification results obtained by the model proposed in this invention can provide more realistic land cover and land use types and better spatial aggregation results. Since the shadow area is covered by clouds, for most methods, the objects in this range cannot be well identified. However, the model proposed in this invention overcomes the influence of clouds to a certain extent under the influence of Lidar data labeling. Figure 5As shown, the model proposed in the present invention is good at generating more realistic and smoother classification images and classification results with fewer noise pixels. The quantitative analysis results in Table 1 also verify the above conclusions. On the other hand, some quantitative analysis indicators are needed to help analyze the effect of segmentation or classification. The quantitative analysis indicators used in the present invention are Kappa coefficient, overall classification accuracy (OA), average classification accuracy (AA) and mean intersection over union (mIoU).

[0111] Table 1 Quantitative results of different methods on Houston 2013 data. (Unit: %)

[0112]

[0113] Note: The values ​​in bold are the highest values ​​of quantitative indicators.

[0114] Figure 6 The experimental results of different methods on the Berlin dataset are shown. Since the Berlin dataset contains more pixels to be classified and its training and testing ratio is much smaller than that of the Houston 2013 dataset, the classification difficulty on the Berlin dataset is much greater than the classification task of the Houston 2013 dataset mentioned above. At the same time, the classification results on this dataset show the same trend as the results of Houston 2013. In the classification task of single-modal data, since the GCN method is a method proposed for the recognition of HSI data, the classification results of HSI data are still due to the classification results of SAR images. But they each have their own advantages. For example, when classifying HSI images, the GCN method better identifies the residential areas in the upper half of the image, while when classifying SAR images, it better identifies the residential areas in the lower half of the image. The GCN model using multimodal stacking features and the CoSpace model that aligns multimodal training data perform similarly, and the classification results obtained are basically consistent in accuracy. The S2FL model and the CCR model are better than the first four methods in terms of results, especially in the identification of residential areas, low plants, industrial areas and water bodies, although there are still many misclassification phenomena, their results are better in different areas. The MMDSI model proposed in the present invention can combine the advantages of different modes due to the integration of deep fusion features and classification results of different modes. Especially in the identification of residential areas, the MMDSI model has achieved smoother and more accurate results. The quantitative analysis results in Table 2 also verify the above conclusions. Whether it is the overall accuracy or the recognition accuracy of each category, the MMDSI model has shown the best performance.

[0115] Table 2 Quantitative results of different methods on the Berlin dataset. (Unit: %)

[0116]

[0117] Note: The bold value is the highest value of the quantitative indicator.

[0118] By comparing the experiments of other different multimodal remote sensing data classification models on Houston 2013 and Berlin datasets, the MMDSI model proposed in the present invention achieved better results, which verified the effectiveness of the model proposed in the present invention.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal remote sensing image segmentation method based on MRF semantic reasoning, characterized in that: The steps are as follows: Step 1: Read different modal data of multimodal remote sensing images, use CNN to train each modal image data separately, and obtain the initial semantic segmentation results of different modal data; Step 2: Multimodal image data is deeply fused through a multimodal feature fusion classification network to obtain multimodal deep fusion features; multiple inference units are generated based on the initial semantic segmentation results of any single modality, and an MRF model is constructed based on the inference units; Step 3: The initial semantic segmentation results of different modalities obtained in step 1 and the deep fusion features of multimodal data obtained in step 2 are respectively used as the label field and feature field in the MRF model to model a multimodal MRF model; Step 4: By self-learning to update the interactive relationship between different modal semantics and the spatial contextual relationship within the same modality, the multimodal MRF model established in step 3 is iteratively solved to obtain the semantic segmentation result of multimodal data.

2. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 1 is characterized in that: The method for obtaining the initial semantic segmentation result in step 1 is as follows: different modal data I of the multimodal remote sensing image a =(I 1 , I 2 ) is m×n in size, and the image data I 1 , I 2 The number of spectral bands are d1, d2, and d1, d2∈N + , N + is a set of positive integers; CNN is used to image data I 1 , I 2 The initial semantic segmentation results of different modalities are obtained by training and learning respectively and recorded as Y = (Y 1 , Y 2 ), and Y (l) The label fields corresponding to different modalities of deep fusion features, l = {l1, l2}, l1, l2∈{1, 2}; The method for obtaining multi-modal deep fusion features is as follows: for image data of different modalities with N pixels in each band, the qth layer of features learned through the CNN network is represented as: in, represents the feature of the cth pixel in the lth mode in the qth layer, I l，c represents the feature of the cth pixel in the lth mode, p is the total number of layers of the CNN network; the function f(·) is the ReLU activation function, and They are weight parameters and bias terms respectively; The feature-level cross-reconstruction strategy is selected to fuse the features of different modalities. The fusion method is: in, Represents the pixel-level fusion feature of the qth layer, pixel-level fusion feature is the deep fusion feature I′, Represents the features of the cth pixel in the lth modality in the qth layer.

3. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 1 or 2, characterized in that: The method for generating multiple inference units is: using a mean shift algorithm to divide the initial semantic segmentation result of any single modality to obtain inference units, and the deep fusion feature corresponding to the position of each inference unit is used as the data feature of the inference unit; The MRF model is a graph model.

4. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 3 is characterized in that: The method for constructing the MRF model according to the inference unit is: the obtained inference unit is represented as R = {R s , s∈S}, each reasoning unit R s Represents a vertex v in the graph model s , each vertex corresponds to an area of ​​the image, and the graph model is constructed as a graph G = {V, E), where V = {R s , s∈S} represents the vertex set, S is the position index set of each vertex, E={e s，t |s, t∈S} represents the edge set between vertices, and edge e s，t Represents vertex v s and vertex v t The spatial relationship of s and vertex v t If the edge e is adjacent, s，t = 1, if vertex v s and vertex v t If they are not adjacent, then edge e s，t =0.

5. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 4 is characterized in that: The method for obtaining the semantic segmentation result of multimodal data is as follows: a self-learning strategy is used to obtain the interaction between two points in the energy function describing the second-order potential function of the MRF model in the multi-layer logical model, so as to realize the modeling of the spatial relationship within a single modality; the reasoning between multimodal semantics is realized by calculating the interaction relationship between tags through conditional probability, and the object-level MRF model is used to obtain the semantic segmentation result of multimodal data.

6. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 5, characterized in that: The method of using the object-level MRF model to obtain the semantic segmentation results of multimodal data is: The deep fusion feature is used as the feature field I = {I s |s∈S}, the labeling fields of different modalities l={l1, l2}, l1, l2∈{1, 2}, and l1≠l2, I s Represents vertex R s The eigenvector of s The dimension of is determined by the deep fusion feature I′; Tag variables is a random variable representing the mark at position s in the lth modality, corresponding to the initial semantic segmentation result Y 1 , Y 2 , label variable Take a value from the category label set Λ = {1, 2, ..., K}, where K represents the total number of semantic categories and S = {s} represents the location index set of the image data; Represents a random field of markers A specific implementation of , then for the MRF model of multimodal data semantic reasoning proposed in the present invention, the optimal implementation of different modalities is achieved through the objective function method: in, They represent the optimal realization of different modes, They represent the posterior probabilities of the labeling results of different modalities under the data feature conditions, Represent the feature likelihood functions of different modes respectively; They represent the prior probabilities of different modal labeling results respectively; are the conditional probabilities of the current modality under the labeling of another modality; They respectively represent the feature information of multimodal image data based on the labeling results of different modalities, represents the intra-layer relationship of the l1th modal semantic layer, represents the intra-layer relationship of the l2th modal semantic layer, They represent the semantic interaction information between different modalities respectively.

7. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 6 is characterized in that: For the multimodal feature field, the feature information of the multimodal image data is expressed as: After the division of the reasoning unit, it is further expressed as R = {R s |s∈S}, s is the position node, when vertex v s The marker variable When h represents a value in the category label set Λ = {1, 2, ..., K}, assuming that the feature vectors of the pixels contained in each over-segmented region as the inference unit are independent and identically distributed and obey the Gaussian distribution, then the likelihood function of the object feature vector is equivalent to the product of the likelihood functions of the feature vectors of the pixels contained in the object, that is, the conditional probability It is expressed as: So the characteristic information: Among them, d is the number of bands of deep fusion feature I′, μ h and∑ h Respectively represent the mean and variance of the feature marked as h; I′ s , I′ t represent the spectral values ​​at position s and position t respectively, Represents the probability of the feature distribution when the label at position s is h.

8. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 6 or 7, characterized in that: The intra-layer relationship E2(X l ) is calculated as: in, in, represents the spatial contextual relationship of position s in the lth modal semantic layer, and the intra-layer relationship E2(X l ) is determined by the spatial context of each position in the layer Composite, represents the potential function of the double-point group at position s and its neighboring position t, N s ={t|t∈S,e s，t =1} represents the set of neighboring locations of location s, Respectively represent the tags of the,th modal semantic layer at position s and position t, e s，t represents the edge between position s and position t, e s，t =1 means that position s is adjacent to position t, b s，t Indicates region R s and region R t The length of the common boundary of represents the spatial relationship between category i and category j, Represents the weight of the spatial relationship between category i and category j in the tagging result of the l-th modal semantic layer.

9. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 8, characterized in that: The spatial relationship between category i and category j is: in, m i，j represents the penalty coefficient, represents the probability that the l1th modal semantic layer is labeled as i when the l2th modal semantic layer at position s is labeled as category j, represents the number of pixels labeled as i in the l1th modal semantic layer when the l2th modal semantic layer at position s is labeled as category j, Represents the number of pixels labeled j at the l2th modality semantic layer at position s; The weight of the spatial relationship between category i and category j in the tagging result of the lth modal semantic layer for:

10. The multimodal remote sensing image segmentation method based on MRF semantic reasoning according to claim 1, characterized in that: For any position s∈S, the semantic interaction information The relationship between different modal markers is expressed as follows: Among them, α is the weight relationship between different modal categories, and its value is the average area of ​​the category: γ i，j Represents the distance between markers, and the discrete distance calculation method is:

Images