Urban land use scene classification method based on multimodal data of street view image matching and remote sensing image

Abstract

The present invention proposes a multimodal data urban land use scene classification method based on matching street view images with remote sensing images, designs a cross-view image matching method based on geometric transformation, and realizes spatial position matching of multi-view street view images and remote sensing images; uses a pre-trained large model with strong generalization ability to construct an urban land use scene classification model based on high-resolution remote sensing images and street view images, and extracts deep features of different scales and different viewpoints; designs a global-local feature fusion module to aggregate features of different levels through self-attention; designs a gate-controlled recurrent fusion module to realize the interaction of street view images with different shooting times and shooting perspectives; uses a channel attention mechanism to allocate remote sensing image features and street view image features according to different weights, optimizes their contribution ratio in different classification tasks, and designs a dynamic gradient modulation strategy to adaptively fuse different modal information to achieve accurate classification of urban land use remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a multimodal data urban land use scene classification method based on matching street view images with remote sensing images. Background Art

[0002] Land use is the ongoing or periodic management and economic activity carried out by humans based on the natural conditions of the land and societal needs. This process reflects the interaction and impact between human socioeconomic activities and surface resources. Accurately identifying the categories of various urban areas and understanding basic information on the current status of urban land use can provide fundamental data support for sustainable urban development planning. Furthermore, automatically identifying urban land use categories facilitates the delineation of urban functional zones within the national geographic conditions monitoring project.

[0003] Remote sensing technology has become the main means of obtaining urban geographic information due to its wide observation range, strong timeliness and short cycle characteristics. High-resolution remote sensing images have the ability to represent spatial details and can be effectively used for accurate monitoring and classification of urban surface changes. However, the spectral, texture and shape characteristics of different construction land uses (such as medium and low-density residential areas and commercial land) are complex and the feature differentiation is weak. It is difficult to effectively achieve accurate urban land use classification by relying solely on remote sensing images. Therefore, how to accurately identify land use categories is an urgent problem to be solved. In fact, multi-view street view images provide rich details of real-world landscapes, which can supplement the problem of insufficient functional features of remote sensing images and improve the accuracy of urban land use scene classification based on remote sensing images. Therefore, this method constructs a dual-branch neural network architecture by matching remote sensing images with multi-view street view images. At the same time, through feature fusion modules and dynamic gradient modulation strategies, accurate and automatic classification of urban land use is achieved. Summary of the Invention

[0004] The purpose of the present invention is to propose a multimodal data urban land use scene classification method based on street view image matching remote sensing image. By matching remote sensing image with multi-view street view image, a urban land use scene classification model based on high-resolution remote sensing image and multi-view street view image is constructed, so as to carry out the classification of urban land use scenes in a large range in a refined manner, overcoming the problem of too few functional features in a single remote sensing image in traditional methods.

[0005] To achieve the above-mentioned purpose, the technical solution of the present invention is: a multimodal data urban land use scene classification method based on street view image matching remote sensing image, comprising the following steps:

[0006] Step S1: Obtain high-resolution remote sensing images and road network data of the study area; perform preprocessing operations on the remote sensing images; filter and streamline the road network data, set equally spaced coordinate points, and obtain multi-view street view images for each coordinate point;

[0007] Step S2: Design a cross-view image matching method based on geometric transformation to perform spatial position matching on multi-view street view images and remote sensing images;

[0008] Step S3: Based on a dual-branch neural network architecture, a pre-trained large model feature extractor is constructed to extract multi-level semantic features of the remote sensing image after spatial position matching and deep semantic features of the multi-view street view image;

[0009] Step S4: Construct a multimodal feature fusion and enhancement module; construct a global-local feature fusion module based on the multi-level semantic features of the extracted remote sensing image to improve the model's understanding of the entire remote sensing image and obtain enhanced features; design a gate-controlled recurrent fusion module based on the deep semantic features extracted from the multi-view street view image to perform spatiotemporal interaction on the bi-temporal semantic features and obtain fused features; generate the final features for classification based on the enhanced features and fused features and perform classification;

[0010] Step S5: Design an adaptive loss function to address the optimization imbalance problem in the model. Design a dynamic gradient modulation strategy to adaptively optimize the training process of each modality by monitoring the difference in the contribution of different modalities to the learning objective.

[0011] Step S6: Create a multimodal urban land use classification sample set covering remote sensing images and street view images, and use the sample set to train and predict the model, as well as optimize and improve the model.

[0012] Furthermore, the preprocessing operation of the remote sensing image in step S1 is specifically: obtaining a high-resolution remote sensing image of the study area, and performing preprocessing operations on the image, specifically including radiation correction, geometric correction, image cropping and multispectral-panchromatic fusion operations.

[0013] Furthermore, the road network data is filtered and simplified in step S1, coordinate points are set at equal intervals, and a multi-view street view image of each coordinate point is obtained, specifically:

[0014] Download Open Street Map road network data and filter and clean it, specifically removing road networks with chaotic topological relationships, mountain road networks, and road networks within residential areas. Set equally spaced coordinate points at preset intervals. Based on these coordinate points, use the Baidu Maps API to obtain street view images of each point at a viewing angle of -0 degrees in front, -90 degrees to the right, -180 degrees behind, and -270 degrees to the left.

[0015] Furthermore, step S2 specifically includes the following steps:

[0016] Step S21: For the remote sensing image pre-processed in step S1, a projection transformation method based on pixel resolution is used to convert the remote sensing image from geographic coordinates to pixel coordinates. The calculation formula is as follows:

[0017]

[0018] Where C a and C b They represent the geographic coordinates of the upper left corner of the image; H and W represent the pixel resolution in the image; and Represents the pixel coordinates of pixel i; and The pixel column and row number representing the pixel coordinates;

[0019] Step S22: Based on the remote sensing image processed in step S21, two adjacent points on the street are selected. The coordinates of one point are recorded as (x1, y1) and the coordinates of the other point are recorded as (x2, y2). The center point is used as the vehicle shooting point, and the direction from the center point to the other point is used as the vehicle driving direction. The angle between the vehicle driving direction and the row of the pixel coordinate system is calculated. The calculation formula is as follows:

[0020]

[0021] The result calculated by atan2 is the radian of rotation, which is multiplied by Convert to angle to get the required rotation angle A;

[0022] Step S23: Based on the remote sensing image processed in step S21 and the rotation angle calculated in step S22, the remote sensing image is rotated using an affine transformation matrix so that the vehicle's travel direction is parallel to the rows of the pixel coordinate system to avoid data redundancy caused by black edges generated by subsequent cropping. The affine transformation matrix X is shown below:

[0023]

[0024] Step S24: The rotation-corrected remote sensing image is divided into four equal blocks in a clockwise direction based on the center point determined in step S22. The multi-perspective street view image captured at the shooting point is subdivided into eight equal areas to achieve spatial position matching between the multi-perspective street view image and the remote sensing image. The street view image is captured in the four directions of the vehicle (front, rear, left, and right), and the street view image in each shooting direction is subdivided into two equal areas. Each remote sensing image block corresponds to an equal area of ​​the street view image in two adjacent shooting directions, and the correspondence is determined by the viewing angle characteristics of the street view and remote sensing images.

[0025] Furthermore, step S3 specifically includes the following steps:

[0026] Step S31: In the remote sensing image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract multi-level semantic features of the remote sensing image;

[0027] Step S32: In the multi-view street view image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract deep semantic features of the multi-view street view image.

[0028] Furthermore, step S4 specifically includes the following steps:

[0029] Step S41: Design a global-local feature fusion module for the multi-level semantic features extracted in S31. The global-local feature fusion module performs the following operations:

[0030] Add the corresponding elements of the extracted low-level features and high-level features. The calculation formula is as follows:

[0031] F R =σ(F C )+σ(F C )·δ(σ(F B ))

[0032] Where F R Represents the linear combination of features, F C and F B They represent the high-level semantic features and low-level semantic features of the remote sensing image block, σ is a 1×1 convolutional layer, and δ is a sigmoid function;

[0033] After linear combination, a lightweight network based on self-attention is used to establish the global context dependency of features at different levels. Specifically, three parallel 1×1 convolutional layers are used to process F R , generate query Q, key K, value V, and generate weight W through softmax function. The calculation formula is as follows:

[0034] W=Softmax(Q·K)

[0035] Finally, the enhanced features are generated and the calculation formula is as follows:

[0036]

[0037] Step S42: Design a gate-controlled loop fusion module based on the deep semantic features of the multi-view street view image extracted in step S32. The gate-controlled loop fusion module performs the following operations:

[0038] The deep semantic features of multi-view street view images are spliced ​​in the channel dimension. The calculation formula is as follows:

[0039] F fwd1 =Cat(F d ,F e )

[0040] F fwd2 =Cat(F e ,F d )

[0041] Where, F e and F d Represents the features of street view images from different perspectives matched with remote sensing image patches, F fwd1 and F fwd2 Represents two features after splicing in the channel dimension; F fwd1 and F fwd2 Input gate control unit to obtain the fused feature F out , the calculation formula is as follows:

[0042] F out =GRU(F fwd1 ,F fwd2 )

[0043] Step S43: The features obtained by processing S41 and S42 are spliced ​​in the channel dimension, and the channel attention mechanism is used to adjust the weights of each channel feature to generate the final features for classification. The final features pass through the fully connected layer to obtain the category.

[0044] Furthermore, the adaptive loss function designed in step S5 is specifically as follows:

[0045] For the model constructed in steps S3 and S4, the cross entropy function is used as the loss function of the model to update the weights of the network and reduce the error between the actual and predicted results. The calculation formula is as follows:

[0046]

[0047] Where N represents the batch size of training, C represents the number of categories, and y j,c The indicator variable representing the true label of the j-th sample in the c-th category, Represents the predicted probability of the jth sample for category c, which serves as an optimizer for urban scene classification.

[0048] Furthermore, the dynamic gradient modulation strategy designed in step S5 is specifically as follows:

[0049] SGD is used as the optimizer and the initial learning rate is set for the optimization strategy of land use scenario classification. The SGD optimizer formula is as follows:

[0050]

[0051] In the formula represents the estimated value of the gradient, η represents the learning rate, Represents the model parameters at step t; calculate the contribution of different modes to the learning objective, the calculation formula is as follows:

[0052]

[0053] Where, and Represents the weighted sum of the contributions of multi-view street view images and remote sensing images to classification; Is an indicator function, when k = y j When , its value is 1, indicating that the predicted category is the same as the true category, otherwise the value is 0; j is the corresponding sample, and represents the weight in the fully connected layer, which is used to weight street view and remote sensing features; t is the index of the training optimization process, which is incremented each time the parameters are updated; and represents the features extracted from the street view image and remote sensing image branches; b is the bias term; in order to adjust the gradient contribution of different modalities in the learning process, an unbalanced ratio is designed. and adjustment factor To ensure that the contributions of multi-view street view images and remote sensing images to classification are balanced, the formula is as follows:

[0054]

[0055] In the formula, α is a hyperparameter that controls the degree of modulation, u represents the selected mode, and finally Integrated into the SGD optimizer, the formula is as follows:

[0056]

[0057] Furthermore, the categories of the classification sample set in step S6 include low- and medium-density residential areas, high-density residential areas, undeveloped land, commercial land, and industrial factories.

[0058] Furthermore, step S6 specifically includes the following steps:

[0059] Step S61: Select a commonly used labeling method, manually interpret the remote sensing image category through visual interpretation, and combine it with the matching method designed in S2 to spatially match the street view image and the remote sensing image to complete the sample set production;

[0060] Step S62: Based on the sample set obtained in S61, set hyperparameters including the number of iterations and the learning rate to train the model, and store the weight file with the best effect in the validation set;

[0061] Step S63: Based on the weight file obtained in step S62, the images in the test set are predicted, and color indexing is used to convert them into RGB images, and affine information and projection information are added to generate the final prediction results.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] First, a geometric method for matching multi-view street view images with remote sensing images was designed. This method matches buildings in remote sensing images with buildings in multi-view street view images. By learning the functional features of buildings, the method aids land use classification in remote sensing images, improving classification accuracy. Secondly, the pre-trained large-scale encoder model used in the feature extraction branch, pre-trained on large-scale image data, demonstrates strong generalization capabilities for handling a variety of complex and diverse image tasks. Furthermore, a global-local feature fusion module is designed to aggregate features at different levels through a self-attention mechanism. For multi-view street view images, a gated recurrent fusion module is designed to perform spatiotemporal interaction on bi-temporal semantic features. Finally, a dynamic gradient modulation strategy is designed to adaptively control the optimization of each modality. This method provides technical support for urban land use classification and is of great significance for understanding urban development patterns, assessing the impact of human activities on urban development, and providing scientific guidance for urban land planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the process of the present invention;

[0065] Figure 2 A diagrammatic illustration of the matching of street view images and remote sensing images according to the present invention;

[0066] Figure 3 This is a scene classification result diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following is combined with Figure 1-3 , the technical solution of the present invention is described in detail.

[0068] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0069] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0070] The present invention proposes a multimodal data urban land use scene classification method based on street view image matching and remote sensing image matching, which specifically includes the following steps:

[0071] Step S1: Obtain high-resolution remote sensing images and road network data of the study area, and perform preprocessing operations on the remote sensing images; filter and streamline the road network data, and set equally spaced coordinate points. Based on the coordinate points, obtain multi-perspective street view images through the Baidu Map API interface.

[0072] Step S2: Design a cross-view image matching method based on geometric transformation to perform spatial position matching on multi-view street view images and remote sensing images.

[0073] Step S3: Based on the dual-branch neural network architecture, a pre-trained large model feature extractor is constructed to extract the multi-level semantic features of the remote sensing image after spatial position matching and the deep semantic features of the multi-view street view image.

[0074] Step S4: Construct a multimodal feature fusion and enhancement module. Based on the multi-level semantic features of the extracted remote sensing images, a global-local feature fusion module is constructed to improve the model's understanding of the entire remote sensing image and obtain enhanced features. Based on the deep semantic features extracted from the multi-view street view images, a gate-controlled recurrent fusion module is designed to perform spatiotemporal interaction on the bi-temporal semantic features and obtain fused features. The final features for classification are generated based on the enhanced and fused features, and classification is performed.

[0075] Step S5: Design an adaptive loss function. To address the optimization imbalance problem in the model, design a dynamic gradient modulation strategy to adaptively optimize the training process of each modality by monitoring the differences in the contributions of different modalities to the learning objectives.

[0076] Step S6: Create a multimodal urban land use classification sample set covering remote sensing images and street view images, including low- and medium-density residential areas, high-density residential areas, undeveloped land, commercial land, and industrial factories. Use the sample set to train and predict the model, as well as optimize and improve the model.

[0077] In this embodiment, step S1 specifically includes the following steps:

[0078] Step S11: Obtain high-resolution remote sensing images of the study area and perform preprocessing operations on the images, including radiation correction, geometric correction, image cropping and multispectral-panchromatic fusion operations.

[0079] Step S12: Download the Open Street Map road network data and filter and clean it, specifically removing topologically chaotic road networks, mountain roads, and residential roads. Coordinate points are set at 100-meter intervals. Based on these coordinate points, street view images are obtained from the Baidu Maps API at each point from 0 degrees (directly forward), 90 degrees (right), 180 degrees (directly behind), and 270 degrees (left).

[0080] In this embodiment, step S2 specifically includes the following steps:

[0081] Step S21: For the remote sensing image pre-processed in step S1, a projection transformation method based on pixel resolution is used to convert the remote sensing image from geographic coordinates to pixel coordinates. The calculation formula is as follows:

[0082]

[0083] Where C a and C b They represent the geographic coordinates of the upper left corner of the image; H and W represent the pixel resolution in the image; and Represents the pixel coordinates of pixel i; and The pixel column and row number representing the pixel coordinates;

[0084] Step S22: Based on the processed remote sensing image, select two adjacent points on the street. Take one point as the center and record the coordinates of the center point as (x1, t1). The other point is recorded as (x2, y2). The line connecting the two points can be regarded as the vehicle's travel direction, and the center point can be regarded as the vehicle's shooting point. Calculate the angle between the travel direction and the row of the pixel coordinate system. The calculation formula is as follows:

[0085]

[0086] The result calculated by atan2 is the radian of rotation, which is multiplied by Convert to an angle to get the desired rotation angle, usually between -180 and 180 degrees.

[0087] Step S23: Based on the processed remote sensing image and the calculated rotation angle, the image is rotated to the direction of vehicle travel parallel to the rows of the pixel coordinate system using an affine transformation matrix to avoid data redundancy caused by black edges generated by subsequent cropping. The affine transformation matrix is ​​shown below:

[0088]

[0089] Step S24: Divide the rotation-corrected remote sensing image into four equal blocks in a clockwise direction based on the center point determined in step S22. The multi-perspective street view image taken at the shooting point is further subdivided into eight equally divided areas. Each remote sensing image block corresponds to a part of each of the two adjacent street view perspectives, and this correspondence is determined by the perspective characteristics of the street view and the remote sensing image. Specifically, the street view image is taken in the front, rear, left, and right directions of the vehicle, and the perspective in each direction slightly covers the area in the adjacent direction. For example, when shooting directly in front, the perspective includes part of the left front and right front areas at the same time. Similarly, the remote sensing image is acquired from a bird's-eye view and has a larger coverage area. When the remote sensing image is divided into four equal parts, each part will cover the ground area in the left and right directions of the vehicle. For example, the remote sensing image part in the left front covers the ground on the left and in front at the same time. (See Figure 2 )

[0090] In this embodiment, step S3 specifically includes the following steps:

[0091] Step S31: In the remote sensing image feature extraction branch, the encoder of a pre-trained large model (e.g., Fast Sam) is selected as the remote sensing image feature extraction backbone network to extract multi-level semantic features of the remote sensing image;

[0092] Step S32: In the multi-view street view image feature extraction branch, the encoder of a pre-trained large model (such as Fast Sam) is selected as the multi-view street view image feature extraction backbone network to extract deep semantic features of the multi-view street view image.

[0093] In this embodiment, step S4 specifically includes the following steps:

[0094] Step S41: Design a global-local feature fusion module based on the multi-level semantic features extracted in S31. Specifically, the attention mechanism is used to refine high-level features with low-level semantic features, enabling the model to recognize the macro context of the image while maintaining attention to important details, thereby achieving a better balance between fine classification and scene understanding. The main approach is as follows:

[0095] Add the corresponding elements of the extracted low-level features and high-level features. The calculation formula is as follows:

[0096] F R =σ(F C )+σ(F C )·δ(σ(F B ))

[0097] Where F RRepresents the linear combination of features, F C and F B Represent the high-level semantic features and low-level semantic features of the remote sensing image block, σ is a 1×1 convolution layer, and δ is a sigmoid function. After combination, a lightweight network based on self-attention is used to establish the global context dependency relationship of features at different levels. Specifically, three parallel 1×1 convolution layers are used to process F R , generate Query (Q), Key (K), Value (V), and generate weight W through the softmax function. The calculation formula is as follows:

[0098] W=Softmax(Q·K)

[0099] Finally, the enhanced features are generated and the calculation formula is as follows:

[0100]

[0101] Step S42: Design a gate-controlled loop fusion module based on the deep semantic features of the multi-view street view images extracted in step S32. Specifically, the street view images were captured with time differences and some perspectives overlap. The gate-controlled loop feature fusion module can combine the depth of historical and current information to perform spatiotemporal interaction on bi-temporal semantic features. The main steps are as follows:

[0102] The deep semantic features of multi-view street view images are spliced ​​in the channel dimension. The calculation formula is as follows:

[0103] F fwd1 =Cat(F d ,F e )

[0104] F fwd2 =Cat(F e ,F d )

[0105] Where, F e and F d Represents the features of street view images from different perspectives matched with remote sensing image patches, F fwd1 and F fwd2 Represents the two features after concatenation in the channel dimension. fwd1 and F fwd2 Input gate control unit to obtain the fused feature F out , the calculation formula is as follows:

[0106] F out =GRU(F fwd1 ,F fwd2 )

[0107] Step S43: The features obtained by processing S41 and S42 are spliced ​​in the channel dimension. Considering the different contributions of remote sensing images and multi-view street view images to the final classification, the channel attention mechanism is used to adjust the weights of each channel feature to generate the final features for classification. The final features are passed through the fully connected layer to obtain the category.

[0108] In this embodiment, step S5 specifically includes the following steps:

[0109] Step S51: For the model constructed in steps S3 and S4, a loss function is designed to update the network weights to reduce the error between the actual and predicted results. Specifically, the cross entropy function is used as the loss function of the model, and the calculation formula is as follows:

[0110]

[0111] Where N represents the batch size of training, C represents the number of categories, and y j,c The indicator variable representing the true label of the j-th sample in the c-th category, represents the predicted probability of the jth sample for category c, which serves as an optimizer for urban scene classification;

[0112] Step S52: SGD is used as the optimizer and the initial learning rate is set for the optimization strategy of land use scene classification. Considering the optimization imbalance problem of remote sensing images and multi-view street view images during the training process, a dynamic gradient modulation strategy is designed. The main approach is as follows:

[0113] The SGD optimizer formula is as follows:

[0114]

[0115] In the formula represents the estimated value of the gradient, η represents the learning rate, Represents the model parameters at step t. The contribution of different modes to the learning objective is calculated as follows:

[0116]

[0117] Where, and Represents the weighted sum of the contributions of multi-view street view images and remote sensing images to classification; Is an indicator function, when k = y j When , its value is 1, indicating that the predicted category is the same as the true category, otherwise the value is 0; j is the corresponding sample, and represents the weight in the fully connected layer, which is used to weight street view and remote sensing features; t is the index of the training optimization process, which is incremented each time the parameters are updated; and represents the features extracted from the street view image and remote sensing image branches; b is the bias term. In order to adjust the gradient contribution of different modalities in the learning process, an unbalanced ratio is designed. and adjustment factor This ensures that their contributions to the classification are balanced. The formula is as follows:

[0118]

[0119] Where α is a hyperparameter that controls the degree of modulation, and u represents the selected mode, which is generally the mode that dominates the training. Integrated into the SGD optimizer, the formula is as follows:

[0120]

[0121] In this embodiment, step S6 specifically includes the following steps:

[0122] Step S61: Select a commonly used labeling method, manually interpret the remote sensing image category through visual interpretation, and combine it with the matching method designed in S2 to spatially match the street view image and the remote sensing image to complete the sample set production;

[0123] Step S62: Based on the sample set obtained in S61, set reasonable hyperparameters such as the number of iterations and learning rate to train the model, and store the weight file with the best effect in the validation set;

[0124] Step S63: Based on the weight file obtained in step S62, the images in the test set are predicted, and color indexing is used to convert them into RGB images, and affine information and projection information are added to generate the final prediction results.

[0125] In this real-world example, the urban area of ​​XX City, XX Province, was used as the research object. 2020 street view imagery and remote sensing imagery of the urban area were used to classify urban scenes using red, green, and blue bands. This example used 1,000 256×256 pixel remote sensing images and 2,000 512×400 street view images and corresponding annotated data for model training. The scenes were categorized into five categories: low- and medium-density residential areas, high-density residential areas, commercial land, industrial plants, and undeveloped land. Figure 3 For detailed classification results, Figure 3 The results include the classification of commercial land using street view as an aid and the classification of commercial land without using street view.

[0126] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A multimodal urban land use scene classification method based on street view image matching and remote sensing imagery includes the following steps: Step S1: Obtain high-resolution remote sensing images and road network data of the study area; perform preprocessing operations on the remote sensing images; filter and streamline the road network data, set equally spaced coordinate points, and obtain multi-view street view images for each coordinate point; Step S2: Design a cross-view image matching method based on geometric transformation to perform spatial position matching on multi-view street view images and remote sensing images; Step S3: Based on a dual-branch neural network architecture, a pre-trained large model feature extractor is constructed to extract multi-level semantic features of the remote sensing image after spatial position matching and deep semantic features of the multi-view street view image; Step S4: Construct a multimodal feature fusion and enhancement module; construct a global-local feature fusion module based on the multi-level semantic features of the extracted remote sensing image to improve the model's understanding of the entire remote sensing image and obtain enhanced features; Based on the deep semantic features extracted from multi-view street view images, a gate-controlled recurrent fusion module is designed to perform spatiotemporal interaction on the bi-temporal semantic features and obtain fused features. Generate final features for classification based on enhanced features and fused features and perform classification; Step S5: Design an adaptive loss function to address the optimization imbalance problem in the model. Design a dynamic gradient modulation strategy to adaptively optimize the training process of each modality by monitoring the difference in the contribution of different modalities to the learning objective. Step S6: Create a multimodal urban land use classification sample set covering remote sensing images and street view images, and use the sample set to train and predict the model, as well as optimize and improve the model.

2. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: The preprocessing operation of the remote sensing image described in step S1 is specifically: obtaining a high-resolution remote sensing image of the study area and performing preprocessing operations on the image, including radiation correction, geometric correction, image cropping and multispectral-panchromatic fusion operations.

3. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: The road network data is filtered and simplified in step S1, coordinate points are set at equal intervals, and a multi-view street view image of each coordinate point is obtained, specifically: Download Open Street Map road network data and filter and clean it, specifically removing road networks with chaotic topological relationships, mountain road networks, and road networks within residential areas. Set equally spaced coordinate points at preset intervals. Based on these coordinate points, use the Baidu Maps API to obtain street view images of each point at a viewing angle of -0 degrees in front, -90 degrees to the right, -180 degrees behind, and -270 degrees to the left.

4. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1, wherein step S2 specifically comprises the following steps: Step S21: For the remote sensing image pre-processed in step S1, a projection transformation method based on pixel resolution is used to convert the remote sensing image from geographic coordinates to pixel coordinates. The calculation formula is as follows: Where C a and C b They represent the geographic coordinates of the upper left corner of the image; H and W represent the pixel resolution in the image; and Represents the pixel coordinates of pixel i; and The pixel column and row number representing the pixel coordinates; Step S22: Based on the remote sensing image processed in step S21, two adjacent points on the street are selected. The coordinates of one point are recorded as (x1, y1) and the coordinates of the other point are recorded as (x2, y2). The center point is used as the vehicle shooting point, and the direction from the center point to the other point is used as the vehicle driving direction. The angle between the vehicle driving direction and the row of the pixel coordinate system is calculated. The calculation formula is as follows: The result calculated by atan2 is the radian of rotation, which is multiplied by Convert to angle to get the required rotation angle A; Step S23: Based on the remote sensing image processed in step S21 and the rotation angle calculated in step S22, the remote sensing image is rotated using an affine transformation matrix so that the vehicle's travel direction is parallel to the rows of the pixel coordinate system to avoid data redundancy caused by black edges generated by subsequent cropping. The affine transformation matrix X is shown below: Step S24: The rotation-corrected remote sensing image is divided into four equal blocks in a clockwise direction based on the center point determined in step S22. The multi-perspective street view image captured at the shooting point is subdivided into eight equal areas to achieve spatial position matching between the multi-perspective street view image and the remote sensing image. The street view image is captured in the four directions of the vehicle (front, rear, left, and right), and the street view image in each shooting direction is subdivided into two equal areas. Each remote sensing image block corresponds to an equal area of ​​the street view image in two adjacent shooting directions, and the correspondence is determined by the viewing angle characteristics of the street view and remote sensing images.

5. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: Step S3 specifically includes the following steps: Step S31: In the remote sensing image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract multi-level semantic features of the remote sensing image; Step S32: In the multi-view street view image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract deep semantic features of the multi-view street view image.

6. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: Step S4 specifically includes the following steps: Step S41: Design a global-local feature fusion module for the multi-level semantic features extracted in S31. The global-local feature fusion module performs the following operations: Add the corresponding elements of the extracted low-level features and high-level features. The calculation formula is as follows: F R =σ(F C )+σ(F C )·δ(σ(F B )) Where F R Represents the linear combination of features, F C and F B They represent the high-level semantic features and low-level semantic features of the remote sensing image block, σ is a 1×1 convolutional layer, and δ is a sigmoid function; After linear combination, a lightweight network based on self-attention is used to establish the global context dependency of features at different levels. Specifically, three parallel 1×1 convolutional layers are used to process F R , generate query Q, key K, value V, and generate weight W through softmax function. The calculation formula is as follows: W=Softmax(Q·K) Finally, the enhanced features are generated and the calculation formula is as follows: Step S42: Design a gate-controlled loop fusion module based on the deep semantic features of the multi-view street view image extracted in step S32. The gate-controlled loop fusion module performs the following operations: The deep semantic features of multi-view street view images are spliced ​​in the channel dimension. The calculation formula is as follows: F fwd1 =Cat(F d ,F e ) F fwd2 =Cat(F e ,F d ) Where, F e and F d Represents the features of street view images from different perspectives matched with remote sensing image patches, F fwd1 and F fwd2 Represents two features after splicing in the channel dimension; F fwd1 and F fwd2 Input gate control unit to obtain the fused feature F out , the calculation formula is as follows: F out =GRU(F fwd1 ,F fwd2 ) Step S43: The features obtained by processing S41 and S42 are spliced ​​in the channel dimension, and the channel attention mechanism is used to adjust the weights of each channel feature to generate the final features for classification. The final features pass through the fully connected layer to obtain the category.

7. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: The adaptive loss function designed in step S5 is as follows: For the model constructed in steps S3 and S4, the cross entropy function is used as the loss function of the model to update the weights of the network and reduce the error between the actual and predicted results. The calculation formula is as follows: Where N represents the batch size of training, C represents the number of categories, and y j,c The indicator variable representing the true label of the j-th sample in the c-th category, Represents the predicted probability of the jth sample for category c, which serves as an optimizer for urban scene classification.

8. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: The dynamic gradient modulation strategy designed in step S5 is specifically as follows: SGD is used as the optimizer and the initial learning rate is set for the optimization strategy of land use scenario classification. The SGD optimizer formula is as follows: In the formula represents the estimated value of the gradient, η represents the learning rate, Represents the model parameters at step t; calculate the contribution of different modes to the learning objective, the calculation formula is as follows: Where, and Represents the weighted sum of the contributions of multi-view street view images and remote sensing images to classification; Is an indicator function, when k = y j When , its value is 1, indicating that the predicted category is the same as the true category, otherwise the value is 0; j is the corresponding sample, and Represents the weights in the fully connected layer, used to weight street view and remote sensing features; t is the index of the optimization process of training, which is incremented each time the parameters are updated; and Represents the features extracted from the street view image and remote sensing image branches; b is the bias term; in order to adjust the gradient contribution of different modes in the learning process, the imbalance ratio is designed and adjustment factor To ensure that the contributions of multi-view street view images and remote sensing images to classification are balanced, the formula is as follows: In the formula, α is a hyperparameter that controls the degree of modulation, u represents the selected mode, and finally Integrated into the SGD optimizer, the formula is as follows:

9. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1 is characterized in that: The categories of the classification sample set in step S6 include low- and medium-density residential areas, high-density residential areas, undeveloped land, commercial land, and industrial factories.

10. The multimodal data urban land use scene classification method based on street view image matching and remote sensing image according to claim 1, characterized in that: Step S6 specifically includes the following steps: Step S61: Select a commonly used labeling method, manually interpret the remote sensing image category through visual interpretation, and combine it with the matching method designed in S2 to spatially match the street view image and the remote sensing image to complete the sample set production; Step S62: Based on the sample set obtained in S61, set hyperparameters including the number of iterations and the learning rate to train the model, and store the weight file with the best effect in the validation set; Step S63: Based on the weight file obtained in step S62, the images in the test set are predicted, and color indexing is used to convert them into RGB images, and affine information and projection information are added to generate the final prediction results.

Images