A city green plant identification method, device, medium and equipment

Abstract

The application discloses a kind of urban green plant identification method, device, medium and equipment, it is related to image recognition technical field, including obtaining remote sensing image to be identified;Deep feature in remote sensing image is extracted, obtain multiscale spectral feature in deep feature, deep feature and multiscale spectral feature are fused, generate multispectral feature map;Wherein, the deep feature includes color feature and texture feature;Strip feature and multiscale shape feature in remote sensing image are extracted, and strip feature and multiscale shape feature are fused, generate multi-shape feature map;By corresponding weight of multispectral feature map and multi-shape feature map is weighted fusion, determine the feature map after fusion;The resolution of the feature map after fusion is restored to original image size, then the feature map of recovery resolution is classified, obtains the final urban green classification result.

Description

A method, device, medium, and equipment for identifying urban green plants. Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, medium and device for identifying urban green plants. Background Technology

[0002] Urban green spaces are an important component of the urban ecosystem and have a significant impact on the urban environment. They play a vital role in many ways, such as reducing air pollution, conserving water resources, and acting as carbon sinks; increasing citizens' well-being by reducing stress and anxiety; and providing recreational spaces for residents. More importantly, different types and structures of urban green space distribution patterns have varying effects on mitigating urban thermal conditions. For example, the distribution structure of trees, shrubs, and grasslands differs significantly in its degree of mitigation. Furthermore, in some areas, the lack of reliable data on the distribution of different types of underground geological exploration teams poses a significant obstacle to effective policy-making and resource allocation. Therefore, fine-grained and precise extraction of trees, shrubs, and grasslands in urban green spaces can more effectively improve urban environmental quality and provide urban managers with a reference for more rationally allocating these three vegetation types in urban areas.

[0003] In recent years, to improve the completeness and accuracy of urban green space identification, high-resolution and ultra-high-resolution satellite imagery has been increasingly applied to urban green space classification. These high-resolution images provide complex details imperceptible to the naked eye, and the increased spatial resolution allows for the identification of more fragmented and scattered vegetation within cities, enhancing the completeness of urban green space identification. However, within the confined urban space, the spectral differences between different types of urban green spaces are relatively small. Furthermore, the limitations of high-resolution image bands mean that relying solely on high-resolution imagery cannot achieve accurate extraction of urban green spaces.

[0004] Meanwhile, to address the complex morphology of various vegetation types within cities, researchers have increasingly focused on designing multi-scale network models. Multi-scale networks can improve the recognition of features of different sizes. While multi-scale networks can significantly improve the accuracy of urban green space identification, in complex urban environments, green spaces are not always regular shapes, such as linear street trees or strip-shaped shrubs. Therefore, simply extracting regular information at different scales is far from sufficient, and consequently, it is difficult to achieve accurate extraction of urban green spaces. Summary of the Invention

[0005] This invention provides a method, apparatus, medium, and device for identifying urban green plants, to solve the aforementioned problems in the prior art, namely, how to accurately extract urban green spaces in the prior art. This invention provides a method for identifying urban green plants, which includes:

[0006] Acquire remote sensing images to be identified;

[0007] Deep features are extracted from remote sensing images, multi-scale spectral features are obtained from the deep features, and the deep features and multi-scale spectral features are fused to generate a multispectral feature map; wherein, the deep features include color features and texture features;

[0008] Extract strip features and multi-scale shape features from remote sensing images, and fuse the strip features and multi-scale shape features to generate a multi-shape feature map;

[0009] The fused feature map is determined by weighting and fusing the weights corresponding to the multispectral feature map and the multi-shape feature map.

[0010] The resolution of the fused feature map is restored to the original image size, and then the feature map with the restored resolution is classified to obtain the final urban green space classification result.

[0011] Optionally, the deep features, strip features, multispectral feature maps, and multi-shape feature maps are obtained through a multi-shape and spectral feature fusion encoder (MSSA-Encoder). The MSSA-Encoder specifically includes:

[0012] The system comprises a first branch, a second branch, a third branch, and a fourth branch. The first branch consists of two sequentially connected 3x3 DOConvs for extracting deep features from remote sensing images. The second and third branches are two parallel 1x3 and 3x1 DOConvs for extracting strip features from remote sensing images. The fourth branch is a dilated spatial convolution pooling pyramid module consisting of a 1x1 convolution and three 3x3 convolutions with different dilation rates, for extracting multi-scale spectral features and multi-scale shape features from deep features and remote sensing images, respectively.

[0013] Optionally, the weights corresponding to the multispectral feature map and the multi-shape feature map are weighted and fused by the feature fusion guidance module FFG. The feature fusion guidance module FFG includes a first branch and a second branch in parallel. The first branch selects the optimal shape feature from the multi-shape and multispectral information, and the second branch obtains the fusion feature of shallow spatial and spectral information.

[0014] The shape features are dynamically aggregated with the fusion features of shallow spatial and spectral information. The specific calculation formula is as follows:

[0015]

[0016] in, For the total weight, and These are the initial weights of the feature map. and These are the dynamic weights that the network assigns to the two feature maps during the training process.

[0017] Optionally, the remote sensing images are preprocessed Gaofen-2 images and Sentinel-2 images; wherein, the preprocessing specifically includes radiometric correction, atmospheric correction and orthorectification processing.

[0018] This invention provides an urban green plant identification device, comprising:

[0019] The acquisition module is used to acquire the remote sensing images to be identified;

[0020] The multispectral feature map acquisition module is used to extract deep features from remote sensing images, obtain multi-scale spectral features from the deep features, and fuse the deep features with the multi-scale spectral features to generate a multispectral feature map; wherein, the deep features include color features and texture features;

[0021] The multi-shape feature map acquisition module is used to extract strip features and multi-scale shape features from remote sensing images, and to fuse the strip features and multi-scale shape features to generate a multi-shape feature map.

[0022] The fusion module is used to determine the fused feature map by weighted fusion of the weights corresponding to the multispectral feature map and the multi-shape feature map;

[0023] The classification module is used to restore the resolution of the fused feature map to the original image size, and then classify the feature map with the restored resolution to obtain the final urban green space classification result.

[0024] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described urban green plant identification method.

[0025] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned urban green plant identification method.

[0026] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a method for identifying urban green plants. This method deeply integrates shape perception and spectral information perception technologies into a classification network to construct a multi-shape and spectral feature fusion encoder. Specifically, by using dynamic directional convolution kernels for convolution operations, it can accurately extract deep features and strip features of the image of the test area, such as vegetation type and land feature boundaries. By using pyramid convolution to extract multi-scale features from the image, including multi-scale shape features and multi-scale spectral features, and then by fusing multi-scale spectral features with deep features, a multi-spectral feature map is output, which can enhance the ability to identify subtle spectral differences between different vegetation types. At the same time, by fusing multi-scale shape features and strip features, the extraction accuracy of complex-shaped land features can be effectively improved. In addition, by multiplying the multi-spectral feature map and the multi-shape feature map with their corresponding weights and then adding them together to obtain the fused feature map, a deep fusion of shape and spectral features is achieved, providing highly discriminative feature input for the subsequent decoder, thereby significantly improving the accuracy of urban green space classification. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0028] Figure 1 is a flowchart of an urban green plant identification method provided by an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of the overall structure of the MSSA-Encoder, a multi-shape and spectral feature fusion encoder provided in an embodiment of the present invention.

[0030] Figure 3 is a schematic diagram of the feature fusion guidance module FFG structure provided in an embodiment of the present invention;

[0031] Figure 4 is a schematic diagram of the classification results of different deep learning models provided in the embodiments of the present invention;

[0032] Figure 5 is a schematic diagram of the computer equipment used in the urban green plant identification method provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0035] Figure 1 is a flowchart of an urban green plant identification method provided by an embodiment of the present invention. As shown in Figure 1, this embodiment illustrates an urban green plant identification method, including:

[0036] S1: Acquire the remote sensing image to be identified.

[0037] For example, the remote sensing images can be Gaofen-2 and Sentinel-2 remote sensing images. Radiometric correction, atmospheric correction, and orthorectification are performed on the acquired Gaofen-2 and Sentinel-2 remote sensing images. This results in a 1m resolution Gaofen-2 image and a 10m resolution Sentinel-2 image.

[0038] For example, the three red-edge bands of the Sentinel-2 image can be extracted and resampled to obtain three red-edge bands with a resolution of 1m.

[0039] After acquiring the 1m resolution red-edge band and the panchromatic band of the high-resolution image, the integrated Gram-Schmidt method was used in ENVI for image fusion to obtain the fused red-edge image. This fused image was then overlaid with the corrected Gaofen-2 image to obtain the final experimental image, which serves as a green vegetation image dataset. This dataset can contain seven bands: red, green, blue, infrared, and three near-infrared bands. All high-resolution remote sensing images in the dataset were segmented into 128*128 image patches before use. To improve model stability, image augmentation was performed using methods such as mirroring, rotation, and adding salt-and-pepper noise.

[0040] S2: Extract deep features from remote sensing images, obtain multi-scale spectral features from the deep features, and fuse the deep features with the multi-scale spectral features to generate a multispectral feature map; wherein, the deep features include color features and texture features.

[0041] Optionally, the deep features, strip features, multispectral feature maps, and multi-shape feature maps are obtained through a multi-shape and spectral feature fusion encoder (MSSA-Encoder). The MSSA-Encoder specifically includes:

[0042] The system comprises a first branch, a second branch, a third branch, and a fourth branch. The first branch consists of two sequentially connected 3x3 DOConvs for extracting deep features from remote sensing images. The second and third branches are two parallel 1x3 and 3x1 DOConvs for extracting strip features from remote sensing images. The fourth branch is a dilated spatial convolution pooling pyramid module consisting of a 1x1 convolution and three 3x3 convolutions with different dilation rates, for extracting multi-scale spectral features and multi-scale shape features from deep features and remote sensing images, respectively.

[0043] S3: Extract strip features and multi-scale shape features from remote sensing images, and fuse the strip features and multi-scale shape features to generate a multi-shape feature map.

[0044] S4: Multiply the multispectral feature map and the multishape feature map with their corresponding weights and then add them together to determine the fused feature map.

[0045] For example, the feature fusion guidance module FFG performs weighted fusion on the weights corresponding to the multispectral feature map and the multi-shape feature map. The feature fusion guidance module FFG includes a first branch and a second branch in parallel. The first branch selects the optimal shape feature from the multi-shape feature map, and the second branch obtains the global spatial spectral feature after the fusion of the shallow space and the multispectral feature map.

[0046] S5: Restore the resolution of the fused feature map to the original image size, then classify the feature map with the restored resolution to obtain the final urban green space classification result.

[0047] For example, before classifying and recognizing the remote sensing images of green plants to be identified, it is necessary to set the model parameters. Specifically, this may include selecting the original high-resolution image to be classified, determining the total number of categories S, determining the image segmentation size img_size, and the number of training samples; then, determining the number of downsampling layers and upsampling layers in the encoding and decoding parts of the deep learning model; then, determining the network learning rate, the number of optimization iterations, and the model optimizer Adam; finally, the number of one-dimensional vector elements output by the feature classifier can be set according to the determined total number of categories.

[0048] For example, based on the set model parameters, a semantic segmentation network MSSFNet is configured, and the training data prepared in S1, namely the green plant image dataset, is used to complete the model training. This network is characterized by the following components:

[0049] (1) Input layer, used to receive remote sensing images as input;

[0050] (2) Multi-shape and spectral feature fusion encoder: used to combine multi-scale depth features with different receptive fields and different shapes, and transmit the upper-layer features to the lower layer through the downsampling layer to further mine the multi-scale information of the features.

[0051] (3) Feature fusion guidance module: weighted fusion of multi-shape and multi-spectral information obtained from the encoder, thereby guiding the decoder to make full use of its small shape and spectral differences.

[0052] (4) Decoder: It is used to restore the resolution of the feature map to the size of the original image. The decoder contains four upsampling layers.

[0053] (5) Classifier module: used to obtain the final soft probability map through convolution, and then obtain the remote sensing image classification map.

[0054] The specific structure of the multi-shape and spectral feature fusion encoder is shown in Figure 2. In the multi-scale feature fusion encoder, ordinary convolutional kernels are first replaced with DOConvs to form the local shape information enhancement module. DOConv integrates an adaptive channel focusing mechanism to dynamically adjust channel weights, improving the model's accuracy. Subsequently, this module is designed to consist of four branches to improve the model's ability to represent features of different sizes and shapes. The first branch consists of two 3x3 DOConvs for further deep feature extraction; the second and third branches are two parallel 1x3 and 3x1 DOConvs, with two parallel strip convolutions obtaining strip feature-enhanced semantic information. This is extremely helpful for extracting strip-shaped vegetation information within a narrow urban area. The fourth branch is an upgraded pyramid convolution module, a dilated spatial convolution pooling pyramid module, which can specifically include: a 1x1 convolution and three 3x3 convolutions with different dilation rates, which can be selected as 6, 12, and 18, respectively. This module can extract green areas of various sizes and regular shapes within the urban area and fuse their features at different scales. Finally, the features obtained from strip convolution and pyramid convolution are superimposed and the maximum value is taken, which obtains features at different scales while preserving features of different shapes. Subsequently, the fused multi-shape features are input into the transformer encoder to focus on their different spectral features.

[0055] For example, the specific structure of the feature fusion guidance module is shown in Figure 3. The feature fusion guidance module includes two branches. The first branch is enhanced shape features, which extract the optimal shape features from the encoder's multi-scale fusion block; the second branch is global spatial spectral features, which are extracted from the last layer of each encoder layer, representing the fusion of shallow spatial and spectral information. After obtaining the two, they are dynamically aggregated together to form enhanced shape spectral features, which are used to guide the decoding process, specifically:

[0056]

[0057] In the formula, For the total weight, and These are the initial weights of the feature map. and These are the dynamic weights that the network assigns to the two feature maps during the training process.

[0058] For example, the acquired remote sensing image data can first be preprocessed, such as through radiometric correction, atmospheric correction, and orthorectification, to lay the foundation for subsequent analysis. Next, shape perception and spectral information perception technologies are integrated into the classification network. When processing remote sensing images, the former captures the complex and varied shape features and distribution patterns of vegetation such as trees, shrubs, and grasslands, while the latter identifies subtle spectral differences between different vegetation types, thus obtaining rich vegetation information. Then, the network transmits the extracted shape and spectral information to a feature fusion guide decoder. This decoder fuses and reconstructs the spectral spatial features of trees, shrubs, and grasslands, extracting more representative features. Finally, based on these reconstructed features, the model can classify each pixel in the remote sensing image, assigning it to the corresponding vegetation category, thereby achieving remote sensing semantic segmentation of different types of vegetation and completing the task of accurate identification and segmentation of vegetation in complex urban environments.

[0059] In the comparative experiment, the deep learning network MSSFNet in this invention was quantitatively compared with mainstream deep learning remote sensing image classification algorithms. Table 1 shows the accuracy comparison results of the method of this invention and commonly used deep learning models such as UNet, Deeplabv3+, and PSPNet for different types of vegetation, including overall accuracy (OA), average pixel accuracy (MPA), average intersection-over-union ratio (MIOU), and intersection-over-union ratio (IOU) and pixel accuracy (PA) for each land cover.

[0060] Table 1. Accuracy Comparison Results of Different Deep Learning Models

[0061]

[0062] Figure 4 shows a comparison of the extraction instances of the classification methods on the dataset, demonstrating that the deep learning network proposed in this invention performs best on fused images.

[0063] The above describes one or more embodiments of the urban green plant identification method provided in this specification. Based on the same idea, this specification also provides a corresponding urban green plant identification device, including:

[0064] The acquisition module is used to acquire the remote sensing images to be identified;

[0065] The multispectral feature map acquisition module is used to extract deep features from remote sensing images, obtain multi-scale spectral features from the deep features, and fuse the deep features with the multi-scale spectral features to generate a multispectral feature map; wherein, the deep features include color features and texture features;

[0066] The multi-shape feature map acquisition module is used to extract strip features and multi-scale shape features from remote sensing images, and to fuse the strip features and multi-scale shape features to generate a multi-shape feature map.

[0067] The fusion module is used to multiply the multispectral feature map and the multi-shape feature map with their corresponding weights and then add them together to determine the fused feature map;

[0068] The classification module is used to restore the resolution of the fused feature map to the original image size, and then classify the feature map with the restored resolution to obtain the final urban green space classification result.

[0069] Specific limitations regarding urban green plant identification devices can be found in the limitations of urban green plant identification methods described above, and will not be repeated here. Each module in the aforementioned urban green plant identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0070] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the urban green plant identification method provided above.

[0071] The present invention also provides a structural schematic diagram of the computer device shown in Figure 5. As shown in Figure 5, at the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the urban green plant identification method provided in the above embodiments.

[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A method for identifying urban green plants, characterized in that, include: The process involves: acquiring a remote sensing image to be identified; extracting deep features from the remote sensing image, obtaining multi-scale spectral features from the deep features, and fusing the deep features with the multi-scale spectral features to generate a multispectral feature map; wherein the deep features include color features and texture features; extracting strip features and multi-scale shape features from the remote sensing image, and fusing the strip features with the multi-scale shape features to generate a multi-shape feature map; acquiring the deep features, strip features, multi-scale spectral features, and multi-scale shape features through a multi-shape and spectral feature fusion encoder (MSSA-Encoder), wherein the multi-shape and spectral feature fusion encoder (MSSA-Encoder) specifically includes: a first branch, a second branch, a third branch, and a fourth branch; wherein... The first branch includes two sequentially connected 3x3 DOConvs for extracting deep features from remote sensing images; the second and third branches are two parallel 1x3 and 3x1 DOConvs for extracting strip features from remote sensing images; the fourth branch is a dilated spatial convolution pooling pyramid module including a 1x1 convolution and three 3x3 convolutions with different dilation rates, for extracting multi-scale spectral features and multi-scale shape features from deep features and remote sensing images, respectively; the multi-spectral feature map and the multi-shape feature map are weighted and fused to determine the fused feature map; the resolution of the fused feature map is restored to the original image size, and then the restored resolution feature map is classified to obtain the final urban green space classification result.

2. The urban green plant identification method as described in claim 1, characterized in that, The multispectral feature map and the multi-shape feature map are weighted and fused using the Feature Fusion Guidance Module (FFG), which includes a parallel first branch and a second branch. The first branch selects the optimal shape feature from the multi-shape feature map, and the second branch obtains the global spatial spectral feature after fusing the shallow spatial and multispectral feature maps. The shape feature and the global spatial spectral feature are then dynamically aggregated, with the specific calculation formula as follows: in, For the total weight, and These are the initial weights for shape features and global spatial spectral features, respectively. and These are the dynamic weights assigned to shape features and global spatial spectral features by the network during training.

3. The urban green plant identification method as described in claim 1, characterized in that, The remote sensing images are preprocessed Gaofen-2 and Sentinel-2 images; the preprocessing specifically includes radiometric correction, atmospheric correction and orthorectification.

4. A device for identifying urban green plants, characterized in that, include: The system comprises the following modules: an acquisition module for acquiring remote sensing images to be identified; a multispectral feature map acquisition module for extracting deep features from the remote sensing images, acquiring multi-scale spectral features from the deep features, and fusing the deep features with the multi-scale spectral features to generate a multispectral feature map; wherein the deep features include color features and texture features; a multi-shape feature map acquisition module for extracting strip features and multi-scale shape features from the remote sensing images, and fusing the strip features with the multi-scale shape features to generate a multi-shape feature map; and a multi-shape and spectral feature fusion encoder (MSSA-Encoder) for acquiring the deep features, strip features, multi-scale spectral features, and multi-scale shape features, wherein the multi-shape and spectral feature fusion encoder (MSSA-Encoder) specifically includes: a first branch, a second branch, and a third branch. The system comprises a first branch and a fourth branch; wherein the first branch includes two sequentially connected 3x3 DOConvs for extracting deep features from remote sensing images; the second and third branches are two parallel 1x3 and 3x1 DOConvs for extracting strip features from remote sensing images; the fourth branch is a dilated spatial convolution pooling pyramid module including a 1x1 convolution and three 3x3 convolutions with different dilation rates, for extracting multi-scale spectral features and multi-scale shape features from deep features and remote sensing images, respectively; a fusion module is used to perform weighted fusion of multi-spectral feature maps and multi-shape feature maps to determine the fused feature map; and a classification module is used to restore the resolution of the fused feature map to the original image size, and then classify the restored resolution feature map to obtain the final urban green space classification result.

5. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the urban green plant identification method according to any one of claims 1-3.

6. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the urban green plant identification method according to any one of claims 1-3.

Images