A Scene Understanding Method Based on Plot-Aware Dynamic Memory

Abstract

This invention discloses a scene understanding method based on land parcel perception dynamic memory. It studies a land use parcel perception memory network model combining a semantic perception structure of land use types based on parcel composition and a multi-scene dynamic memory prototype structure, enabling an understanding of the composition and distribution of land use parcels, which is helpful for generating national land space use patterns. The method includes the following steps: (1) extracting multi-layer convolutional features from remote sensing datasets; (2) constructing a semantic perception structure of land use types; (3) constructing a multi-scene dynamic memory prototype structure; (4) training the perception dynamic memory prototype network; and (5) predicting parcel categories in remote sensing scenes. Compared with existing methods, this remote sensing scene understanding method based on perception dynamic prototype memory can effectively address problems such as complex land features, variable rules for parcel combination, and poor overall parcel integrity, and can be used for generating national land space use patterns.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing, and specifically relates to a scene understanding method based on land parcel perception dynamic memory, which can be used for scene understanding of remote sensing land use. Background Technology

[0002] The purpose of land use scene understanding in remote sensing imagery is to identify land cover types in remote sensing image patches and to label land use categories such as residential land, highway land, and cultural and sports land, rather than simply classifying buildings and vegetation as pixel-level classifications. In recent years, with the development of remote sensing technology and the increasing demands for its applications, semantic understanding and scene recognition methods for high-resolution remote sensing imagery can be mainly classified into three categories: methods based on low-level visual features, mid-level visual representation, and high-level semantic scene understanding.

[0003] The idea behind scene understanding methods based on low-level visual features is to use feature operators to extract feature descriptions from low-level visual attributes of high-resolution remote sensing images, such as color, texture, and spectral values. These methods include color histograms, Gist descriptors, local binary patterns, texture descriptors, oriented gradient histograms, and scale-invariant feature transforms. These methods have good classification results for high-resolution remote sensing images with uniform spatial distribution and structural patterns, but they are not effective for scenes with uneven spatial distribution.

[0004] Scene understanding methods based on mid-level visual representation aim to construct mid-level features that represent the entire image by encoding local features. Commonly used models include: bag-of-visual-words, spatial pyramid matching, locally linearly constrained linear encoding, probabilistic latent semantic analysis, improved Fisher kernel, and locally clustered descriptor vectors. The improved bag-of-visual-words model can effectively improve the accuracy of scene recognition in remote sensing images. Furthermore, incorporating the semantic information contained in the mid-level features into land cover classification applications can significantly improve classification accuracy, and it is also applicable under small sample conditions. While this type of method significantly improves classification accuracy, it is still limited by low-level visual features and encoding methods, and cannot achieve optimal classification accuracy.

[0005] Scene understanding methods based on high-level semantic features primarily utilize deep learning to learn more abstract and discriminative features. These methods can be categorized into unsupervised and supervised feature learning. Unsupervised feature learning methods, such as k-means clustering, principal component analysis, and sparse coding, can automatically learn features from unlabeled images, effectively replacing methods based on manual features, but they cannot fully utilize the information in the data. Supervised feature learning methods based on deep learning optimize neural networks with given labels to achieve classification. The most typical model is the Convolutional Neural Network (CNN). CNNs are combined with other networks to handle more complex scenes, such as the CNN-GCN dual-stream network for high-resolution remote sensing image scene classification, which effectively fuses global and contextual features and uses a linear classifier for classification. Graph neural network models are used to establish high-level relationships between complex features and complete feature inference and information interaction. Graph model node information integration and update methods are used to infer relationships and interact with information between features to enhance feature representation. The improved Inception-V3 remote sensing image scene classification model employs a transfer learning strategy during training, fully utilizing existing models and knowledge to improve training efficiency.

[0006] Current methods for scene understanding include unsupervised feature learning, which is suitable for single scenes, and high-level feature methods based on CNNs, which typically contain global information in the image and can preserve spatial details. However, remote sensing images often contain multiple land cover scenes. Therefore, research should focus on how to utilize scene semantics to construct a current land cover scene map to address the more practical and challenging problem of understanding multiple land cover scenes in a single image.

[0007] In remote sensing imagery, a single land parcel often contains multiple types of land features, with complex feature coverage and varied pattern combinations, resulting in fragmentation and poor overall integrity. To address this issue, this study investigates a land use parcel perception and memory network model based on a combination of a semantic perception structure for land use types and a multi-scene dynamic memory prototype structure. This model aims to understand the composition and distribution of land use parcels, thereby facilitating the generation of national land spatial use patterns. Summary of the Invention

[0008] To address the shortcomings of existing methods, this invention proposes a scene understanding method based on dynamic memory of land parcel perception. By understanding the composition and distribution of land use parcels, this method generates the spatial use pattern of the national territory.

[0009] The technical solution of the present invention is as follows: a scene understanding method based on land parcel perception dynamic memory, comprising the following steps:

[0010] (1) Extract the multi-layer convolutional features of the remote sensing scene image, remove the last classifier of ResNet50, and use the remaining part as the feature extractor. The output of the remaining part is used as the features extracted from the image.

[0011] (2) Construct a semantic perception structure for land use type, and use convolution kernels of size 7×7, 5×5, 3×3 and 1×1 to fuse the multi-layer convolution features extracted in step (1) to balance the performance and accuracy of computation and obtain the convolutional semantic patch expression of remote sensing scene image.

[0012] (3) Construct a multi-scene dynamic memory prototype structure, store the semantic patches obtained in step (2) in an external memory, and design a learnable memory retrieval module to learn the semantic patches stored in the external memory. Based on the self-attention mechanism, use the fused features output after feature fusion as the query condition to retrieve semantic patches with high similarity to the input image features, which are used to identify the plot scenes in high-resolution remote sensing images.

[0013] (4) Train the land use parcel perception dynamic memory prototype network. Based on the land use type semantic perception structure composed of parcels and the multi-scenario dynamic memory prototype structure, the land use parcel perception memory network model is composed of the land use parcel perception memory network model. The Adam optimizer algorithm is used to train the perception dynamic memory prototype network, and the backpropagation algorithm is used to update the network parameters until the network converges.

[0014] (5) Predict the land parcel category of the remote sensing scene. Input the tested remote sensing scene into the trained perceptual dynamic memory prototype network to obtain the land parcel category of the remote sensing scene.

[0015] Furthermore, step (1) extracts multi-layer convolutional features, including the following steps:

[0016] First, a ResNet50 convolutional neural network is constructed using convolution, pooling, and fully connected operations. The ResNet50 is pre-trained using the ImageNet dataset. Then, remote sensing scene images are input into the pre-trained ResNet50 to extract multi-layer convolutional features. Finally, features are extracted using the convolution operation of the last stage of ResNet50, denoted as x′∈R. M×N×C Where M and N are the width and height of the convolutional feature x′, and C is the number of channels of the convolutional feature x′. ResNet50 is pre-trained on ImageNet. Passing an image through the ResNet50 network yields the output feature map, i.e.:

[0017] x′=Conv(I)

[0018] Where Conv represents the feature extraction process, I is the input remote sensing scene image, and x′ represents the extracted multi-layer convolutional features.

[0019] Furthermore, step (2) constructs a semantic perception structure for land use types, including the following steps:

[0020] ResNet was selected as the backbone network of the spatial pyramid. After inputting large-scale remote sensing images, convolutional features of different scales were extracted in each stage of the backbone network through a bottom-up process. The last set of feature maps of each ResNet stage was denoted as {C2, C3, C4, C5}. Then, the extracted convolutional features were upsampled to ensure that the dimension of the high-level convolutional features after processing was the same as the dimension of the low-level convolutional features from the bottom up. Subsequently, the corresponding elements of the high-level convolutional feature maps and the low-level convolutional feature maps passed by the lateral connections were weighted to obtain fused features with multi-scale information. The output feature maps of the perception extraction module were denoted as {P2, P3, P4, P5}, i.e.

[0021]

[0022] Where P5 = C5, considering pyramid pooling to preserve global information at different scales, and recognizing land type labels requires full consideration of global information, pyramid pooling is applied in the last layer of the backbone network to obtain the feature vector h, i.e.:

[0023]

[0024] Among them, Conv 1×1 This indicates a convolution operation with a kernel size of 1×1, performing Convolution on features extracted by ResNet at different scales. 1×1 Convolution operation; SPP represents a weighted summation, where the result of the convolution operation is weighted and summed with the output of the next level of perceptual extraction to obtain the output of the previous level. Considering pyramid pooling, global information is preserved at different scales, and recognizing land cover labels requires full consideration of global information; SPP represents the pyramid pooling process. h represents the output result of the last layer of perception extraction for the i-th input sample. i This represents the result of pyramid pooling of the i-th input, which is stored in external memory M.

[0025] Furthermore, step (3) constructs a multi-scenario dynamic memory prototype structure, including the following steps:

[0026] The multi-scene dynamic memory prototype structure projects the features extracted by the perception extraction module into a one-dimensional vector as the query condition Q(X). It uses the input question Q(X) to retrieve the key-value store, finding the key value of similar patches. The channel dimension of the key value is defined as L, and the channel dimension of the value value as U. The similarity calculation formula between the input image X and each patch is as follows:

[0027]

[0028] Among them, f Φ The projection function extracts the feature representation of X. Simultaneously, the external memory M is converted into keys K(M) and values ​​V(M). After softmax, an S-dimensional vector is output, representing the length S of the external memory storage. Each component represents the correlation probability between each patch and the query image. All values ​​are then weighted and summed.

[0029] z = R(X,M)·V(M)

[0030] The output feature z is obtained. Since memory retrieval is performed in a multi-head manner, the final feature can be expressed as:

[0031] z′=[z1,z2,…z H ]

[0032] Where H represents the number of heads, each head transforms the input image X and external memory storage M into a multi-head query Q. h (f Φ (X)), Key K h (M) and value V h (M) is used to generate the retrieved features. Finally, the output z' is fed into a fully convolutional network to infer land use parcels.

[0033] s=FCN(z′)

[0034] Where FCN represents a fully convolutional network, and s represents the output vector of the fully convolutional network.

[0035] Furthermore, step (4) trains the land use parcel perception dynamic memory prototype network, including the following steps:

[0036] The land use parcel perception memory network model is composed of a semantic perception structure of land use types based on land parcels and a multi-scenario dynamic memory prototype structure. The Adam optimizer algorithm is used to train the perception dynamic memory prototype network, and the backpropagation algorithm is used to update the network parameters until the network converges.

[0037] Furthermore, step (5) predicts the land parcel category in the remote sensing scene, including the following steps:

[0038] The remote sensing scene being tested is input into the trained land parcel-aware dynamic memory prototype network, and the scene category output by the fully convolutional network is used as the category of the land parcel.

[0039] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of the above-described scene understanding method based on land parcel perception dynamic memory.

[0040] The present invention also proposes a computer-readable storage medium storing a computer program that causes a computer to perform the steps of the above-described scene understanding method based on land parcel perception dynamic memory.

[0041] This invention utilizes CNNs to extract multi-layer semantic features from remote sensing images, and then constructs a land use parcel perception and memory network model based on a combination of a land use type semantic perception structure composed of land parcels and a multi-scene dynamic memory prototype structure. The model is trained until network convergence, and finally, it predicts the land use parcel category. Attached Figure Description

[0042] Figure 1 This is a flowchart of a scene understanding method based on land parcel perception dynamic memory according to the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the following description is provided in conjunction with the accompanying drawings. Figure 1 The present invention will now be described in further detail. (Refer to...) Figure 1 The present invention discloses a scene understanding method based on land parcel perception dynamic memory, the implementation steps of which are as follows:

[0044] Step 1: Extract multi-layer convolutional features.

[0045] (1a) First, a ResNet50 convolutional neural network is constructed using convolution, pooling, and fully connected operations. Specifically, the ResNet50 is pre-trained using the ImageNet dataset, and then remote sensing scene images are input into the pre-trained ResNet50 to extract multi-layer convolutional features. Features are extracted using the convolution operation of the last stage of ResNet50, denoted as x′∈R. M×N×C Where M and N are the width and height of the convolutional feature x′, and C is the number of channels of the convolutional feature x′. That is, ResNet50 is pre-trained on ImageNet, and passing an image through the ResNet50 network yields the output feature map, i.e.:

[0046] x′=Conv(I)

[0047] Where Conv represents the feature extraction process, I is the input image, and x′ represents the extracted features.

[0048] Step 2: Construct a semantic perception structure for land use types.

[0049] (2a) ResNet was selected as the backbone network of the spatial pyramid. After inputting large-scale remote sensing images, convolutional features of different scales were extracted in each stage of the backbone network through a bottom-up process. The last set of feature maps of each ResNet stage was denoted as {C2, C3, C4, C5}. Then, the extracted convolutional features were upsampled to ensure that the dimension of the high-level convolutional features after processing was the same as the dimension of the low-level convolutional features from the bottom up. Subsequently, the corresponding elements of the high-level convolutional feature maps and the low-level convolutional feature maps passed by the lateral connections were weighted to obtain fused features with multi-scale information. The output feature maps of the perception extraction module were denoted as {P2, P3, P4, P5}, i.e.

[0050]

[0051] Where P5 = C5, considering pyramid pooling to preserve global information at different scales, and recognizing land type labels requires full consideration of global information, pyramid pooling is applied in the last layer of the backbone network to obtain the feature vector h, i.e.:

[0052]

[0053] Among them, Conv 1×1 This indicates a convolution operation with a kernel size of 1×1, performing Convolution on features extracted by ResNet at different scales. 1×1 Convolution operation; SPP represents a weighted summation, where the result of the convolution operation is weighted and summed with the output of the next level of perceptual extraction to obtain the output of the previous level. Considering pyramid pooling, global information is preserved at different scales, and recognizing land cover labels requires full consideration of global information; SPP represents the pyramid pooling process. h represents the output result of the last layer of perception extraction for the i-th input sample. i This represents the result of pyramid pooling of the i-th input, which is stored in external memory M.

[0054] Step 3: Construct a multi-scenario dynamic memory prototype structure.

[0055] (3a) The multi-scene dynamic memory prototype structure projects the features extracted by the perception extraction module into a one-dimensional vector as the query condition Q(X). It uses the input question Q(X) to retrieve the key-value storage, finds the key value of the patch similar to the land parcel, defines the channel dimension of the key value as L, and the channel dimension of the value value as U. The similarity calculation formula between the input image X and each patch is as follows:

[0056]

[0057] Among them, f Φ The projection function extracts the feature representation of X. Simultaneously, the external memory M is converted into keys K(M) and values ​​V(M). After softmax, an S-dimensional vector is output, representing the length S of the external memory storage. Each component represents the correlation probability between each patch and the query image. All values ​​are then weighted and summed.

[0058] z = R(X,M)·V(M)

[0059] The output feature z is obtained. Since memory retrieval is performed in a multi-head manner, the final feature can be expressed as:

[0060] z′=[z1,z2,…z H ]

[0061] Where H represents the number of heads, each head transforms the input image X and external memory storage M into a multi-head query Q. h (f Φ (X)), Key K h (M) and value V h (M) is used to generate the retrieved features. Finally, the output z' is fed into a fully convolutional network to infer land use parcels.

[0062] s=FCN(z′)

[0063] Where FCN represents a fully convolutional network, and s represents the output vector of the fully convolutional network.

[0064] Step 4: Train the land use parcel perception dynamic memory prototype network.

[0065] The land use parcel perception memory network model is composed of a semantic perception structure of land use types based on land parcels and a multi-scenario dynamic memory prototype structure. The Adam optimizer algorithm is used to train the perception dynamic memory prototype network, and the backpropagation algorithm is used to update the network parameters until the network converges.

[0066] Step 5: Predict the land parcel category in the remote sensing scene.

[0067] The remote sensing scene being tested is input into the trained land parcel-aware dynamic memory prototype network, and the scene category output by the fully convolutional network is used as the category of the land parcel.

[0068] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of the above-described scene understanding method based on land parcel perception dynamic memory.

[0069] The present invention also proposes a computer-readable storage medium storing a computer program that causes a computer to perform the steps of the above-described scene understanding method based on land parcel perception dynamic memory.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit this application. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for scene understanding based on parcel-aware dynamic memory, characterized in that: Includes the following steps: (1) Extract the multi-layer convolutional features of the remote sensing scene image, remove the last classifier of ResNet50, and use the remaining part as the feature extractor. The output of the remaining part is used as the features extracted from the image. (2) Constructing the semantic perception structure of land use types, using convolution kernels with sizes of 7 7, 5 5, 3 3, 1 1 to fuse the multi-layer convolution features extracted in step (1), taking into account the performance and accuracy of the calculation, to obtain the convolution semantic map expression of the remote sensing scene image; Step (2) Constructing a semantic perception structure for land use types includes the following steps: The ResNet network is selected as the backbone network of the spatial pyramid, and after inputting a large-scale remote sensing image, different scale convolutional features are extracted in each stage of the backbone network through a bottom-up process. The last group of feature maps of each stage of the ResNet is denoted as , , , The extracted convolutional features are then up-sampled to ensure that the dimensions of the processed high-level convolutional features are the same as those of the low-level convolutional features extracted from the bottom-up process. Subsequently, the corresponding elements of the high-level convolutional feature maps and the low-level convolutional feature maps transmitted through lateral connection are weighted to obtain fusion features with multi-scale information. The output feature maps of the perception extraction module are denoted as , , , , that is in, = Considering that pyramid pooling preserves global information at different scales, and that identifying land use labels requires full consideration of global information, pyramid pooling is applied in the last layer of the backbone network to obtain feature vectors. ,Right now: in, This indicates a convolution operation with a kernel size of 1×1, applied to features extracted by ResNet at different scales. Convolution operation; This indicates a weighted summation, where the result of the convolution operation is weighted and summed with the output of the next level of perceptual extraction to obtain the output of the previous level. Considering pyramid pooling, global information is preserved at different scales, and recognizing land cover labels requires full consideration of global information. This represents the pyramid pooling process. Indicates the first i The output result of the last layer of perception extraction for each input sample Indicates the first i The input is processed by pyramid pooling, and the result is stored in external memory. M middle; (3) Construct a multi-scene dynamic memory prototype structure, store the semantic patches obtained in step (2) in an external memory, and design a learnable memory retrieval module to learn the semantic patches stored in the external memory. Based on the self-attention mechanism, use the fused features output after feature fusion as the query condition to retrieve semantic patches with high similarity to the input image features, which are used to identify the plot scenes in high-resolution remote sensing images. Step (3) Construct a multi-scenario dynamic memory prototype structure, including the following steps: The multi-scenario dynamic memory prototype structure projects the features extracted by the perception extraction module into a one-dimensional vector as query conditions. Q ( X ), using the input question Q ( X Retrieve the key-value store, find the key values ​​of similar land parcels, and define the channel dimension of the key value as... L The channel dimension of the value is U Input image X The formula for calculating the similarity with each patch is: in, As the projection function, extract the feature representation of X, and simultaneously, allocate external memory. M Convert to key Sum After softmax, the output is a S A dimensional vector, i.e., the length of external memory storage. S Each component represents the probability of each patch being related to the query image, and all values ​​are summed with weights. Obtain output features Since memory retrieval is performed in a multi-head manner, the final features obtained can be described as follows: in, H This indicates the number of heads, each head being processed by input image. X and external memory storage M Convert to multi-head query ,key Sum To generate the retrieved features, and finally, output. Feed the data into a fully convolutional network to infer land use parcels: in, Let represent a fully convolutional network, and s represent the output vector of the fully convolutional network; (4) Train the land use parcel perception dynamic memory prototype network. Based on the land use type semantic perception structure composed of parcels and the multi-scenario dynamic memory prototype structure, the land use parcel perception memory prototype network model is composed of the land use type semantic perception structure and the multi-scenario dynamic memory prototype structure. The Adam optimizer algorithm is used to train the perception dynamic memory prototype network, and the backpropagation algorithm is used to update the network parameters until the network converges. (5) Predict the land parcel category of the remote sensing scene. Input the tested remote sensing scene into the trained perceptual dynamic memory prototype network to obtain the land parcel category of the remote sensing scene.

2. The scene understanding method based on land parcel perception dynamic memory according to claim 1, characterized in that: Step (1) involves extracting multi-layer convolutional features, including the following steps: First, a ResNet50 convolutional neural network is built using convolution, pooling, and fully connected operations. The ResNet50 is pre-trained using the ImageNet dataset. Then, remote sensing scene images are input into the pre-trained ResNet50 to extract multi-layer convolutional features. Features are extracted using the convolution operation of the last stage of ResNet50, denoted as... Where M and N are convolutional features The width and height, C is the convolution feature. The number of channels, i.e., the number of channels of ResNet50 pre-trained on ImageNet, is used to pass the image through the ResNet50 network to obtain the output feature map, i.e.: ； Where Conv represents the feature extraction process, and I is the input remote sensing scene image. The expression represents the extracted multi-layer convolutional features.

3. The scene understanding method based on land parcel perception dynamic memory according to claim 1, characterized in that: Step (4) trains the land use parcel perception dynamic memory prototype network, including the following steps: The land use parcel perception memory network model is composed of a semantic perception structure of land use types based on land parcels and a multi-scenario dynamic memory prototype structure. The Adam optimizer algorithm is used to train the perception dynamic memory prototype network, and the backpropagation algorithm is used to update the network parameters until the network converges.

4. The scene understanding method based on land parcel perception dynamic memory according to claim 1, characterized in that: Step (5) predicts the land parcel category in the remote sensing scene, including the following steps: The remote sensing scene being tested is input into the trained land parcel-aware dynamic memory prototype network, and the scene category output by the fully convolutional network is used as the category of the land parcel.

5. An electronic device, characterized in that: The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor runs the computer program, it performs the steps of a scene understanding method based on land parcel-aware dynamic memory as described in any one of claims 1-4.

6. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to perform the steps of the scene understanding method based on land parcel perception dynamic memory disclosed in any one of claims 1-4.

Images