Multi-modal data urban land utilization scene classification method based on street scene image matching remote sensing image

Abstract

The invention provides a multi-modal data urban land utilization scene classification method based on street scene image matching remote sensing image, designs a cross-view image matching method based on geometric transformation, and realizes spatial position matching of a multi-view street scene image and a remote sensing image; the method comprises the following steps: constructing an urban land utilization scene classification model based on a high-resolution remote sensing image and a streetscape image by utilizing a pre-trained large model with relatively strong generalization ability, and extracting depth features of different scales and different visual angles; designing a global-local feature fusion module, and aggregating features of different levels through self-attention; a gate control cycle fusion module is designed to realize interaction of streetscape images at different shooting time and shooting view angles; through a channel attention mechanism, remote sensing image features and streetscape image features are distributed according to different weights, contribution proportions of the features in different classification tasks are optimized, a dynamic gradient modulation strategy is designed, and different modal information is adaptively fused, so that accurate classification of urban land utilization remote sensing images is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a multimodal data urban land use scene classification method based on matching street view images with remote sensing images. Background Art

[0002] Land use is the continuous or regular management and economic activities carried out by humans based on the natural conditions of the land and social needs. This process reflects the interaction and impact between human social and economic activities and surface resources. Urban land use is an important manifestation of the urbanization process. my country's urbanization is rapid, with an urbanization rate of over 65%. However, the ensuing contradictions in land resource supply and demand, imbalanced utilization structure, and ecological and environmental pressures have brought severe challenges to urban sustainable development. Therefore, accurately obtaining the categories of various areas in the city and mastering the basic information on the current status of urban land use can provide basic data support for urban sustainable development planning. In addition, automatically identifying the category information of urban land use is helpful in carrying out the work of dividing urban functional areas in the national geographic conditions monitoring project.

[0003] Remote sensing technology has become the main means of obtaining urban geographic information due to its wide observation range, strong timeliness and short cycle characteristics. High-resolution remote sensing images have the ability to represent spatial details and can be effectively used for accurate monitoring and classification of urban surface changes. However, the spectral, texture and shape characteristics between different construction lands (such as medium and low-density residential areas and commercial lands) are complex and the feature distinction is weak. It is difficult to effectively achieve accurate urban land use classification by relying solely on remote sensing images. Therefore, how to accurately identify land use categories is an urgent problem to be solved. In fact, multi-perspective street view images provide rich details of real-world landscapes, which can supplement the problem of insufficient functional features of remote sensing images and improve the classification accuracy of urban land use scenes based on remote sensing images. Therefore, this method matches remote sensing images with multi-perspective street view images, constructs a dual-branch neural network architecture, and achieves accurate and automatic classification of urban land use through feature fusion modules and dynamic gradient modulation strategies. Summary of the invention

[0004] The purpose of the present invention is to propose a multimodal data urban land use scene classification method based on street view image matching remote sensing image, and to construct an urban land use scene classification model based on high-resolution remote sensing image and multi-view street view image by matching remote sensing image with multi-view street view image, so as to carry out refined classification of urban land use scenes in a large range, thus overcoming the problem of too few functional features in a single remote sensing image in traditional methods.

[0005] To achieve the above object, the technical solution of the present invention is: a multimodal data urban land use scene classification method based on street view image matching remote sensing image, comprising the following steps:

[0006] Step S1: Obtain high-resolution remote sensing images and road network data of the study area; perform preprocessing operations on the remote sensing images; filter and streamline the road network data, set equally spaced coordinate points, and obtain multi-view street view images of each coordinate point;

[0007] Step S2: Design a cross-view image matching method based on geometric transformation to perform spatial position matching on multi-view street view images and remote sensing images;

[0008] Step S3: Based on the dual-branch neural network architecture, a pre-trained large model feature extractor is constructed to extract the multi-level semantic features of the remote sensing image after spatial position matching and the deep semantic features of the multi-view street view image;

[0009] Step S4: construct a multimodal feature fusion and enhancement module; construct a global-local feature fusion module for the extracted multi-level semantic features of the remote sensing image, improve the model's cognition of the overall remote sensing image and obtain enhanced features; design a gate-controlled loop fusion module for the extracted deep semantic features of the multi-view street view image, perform spatiotemporal interaction on the dual-phase semantic features and obtain fusion features; generate the final features for classification based on the enhanced features and fusion features and perform classification;

[0010] Step S5: Design an adaptive loss function to address the optimization imbalance problem in the model and design a dynamic gradient modulation strategy to adaptively optimize the training process of each modality by monitoring the difference in the contribution of different modalities to the learning objectives;

[0011] Step S6: Create a multimodal urban land use classification sample set covering remote sensing images and street view images, and use the sample set to train and predict the model, as well as optimize and improve the model.

[0012] Furthermore, the preprocessing operation of the remote sensing image in step S1 is specifically: obtaining a high-resolution remote sensing image of the study area, and performing preprocessing operations on the image, including radiation correction, geometric correction, image cropping and multispectral-panchromatic fusion operations.

[0013] Furthermore, the road network data is screened and simplified in step S1, coordinate points are set at equal intervals, and a multi-view street view image of each coordinate point is obtained, specifically:

[0014] Download Open Street Map road network data, filter and clean the road network data, including filtering and removing road networks with chaotic topological relationships, mountain road networks, and road networks within residential areas; set equally spaced coordinate points at intervals of preset lengths, and based on the coordinate points, obtain street view images of each point at the perspectives of -0 degrees in front, -90 degrees to the right, -180 degrees behind, and -270 degrees to the left through the Baidu Map API interface.

[0015] Furthermore, step S2 specifically includes the following steps:

[0016] Step S21: For the remote sensing image preprocessed in S1, a projection transformation method based on pixel resolution is used to convert the remote sensing image from geographic coordinates to pixel coordinates. The calculation formula is as follows:

[0017]

[0018] Where C a and C b They represent the geographic coordinates of the upper left corner of the image; H and W represent the pixel resolution in the image; and Represents the pixel coordinates of pixel i; and The pixel column and row number representing the pixel coordinates;

[0019] Step S22: Based on the remote sensing image processed in step S21, two adjacent points on the street are selected, with one point as the center, the coordinates of the center point are recorded as (x1, y1), and the coordinates of the other point are recorded as (x2, y2). The center point is used as the vehicle shooting point, and the direction from the center point to the other point is used as the vehicle driving direction. The angle between the vehicle driving direction and the row of the pixel coordinate system is calculated. The calculation formula is as follows:

[0020]

[0021] The result calculated by atan2 is the radian of rotation, which can be obtained by multiplying Convert to angle to get the required rotation angle A;

[0022] Step S23: Based on the remote sensing image processed in step S21 and the rotation angle calculated in step S22, the remote sensing image is rotated to a point where the vehicle's driving direction is parallel to the row of the pixel coordinate system using an affine transformation matrix to avoid data redundancy caused by black edges generated by subsequent cropping; the affine transformation matrix X is shown below:

[0023]

[0024] Step S24: the remote sensing image that has been rotationally corrected is divided into four equal blocks in a clockwise direction based on the center point determined in step S22; the multi-perspective street view image taken at the shooting point is subdivided into eight equal areas to achieve spatial position matching between the multi-perspective street view image and the remote sensing image; wherein the street view image is shot in four directions, namely, the front, rear, left and right directions of the vehicle, and the street view image in each shooting direction is subdivided into two equal areas; each remote sensing image block corresponds to one equal area of ​​the street view images in two adjacent shooting directions, and the corresponding relationship is determined by the viewing angle characteristics of the street view and the remote sensing image.

[0025] Furthermore, step S3 specifically includes the following steps:

[0026] Step S31: In the remote sensing image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract multi-level semantic features of the remote sensing image;

[0027] Step S32: In the multi-view street view image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract deep semantic features of the multi-view street view image.

[0028] Furthermore, step S4 specifically includes the following steps:

[0029] Step S41: Design a global-local feature fusion module for the multi-level semantic features extracted in S31. The global-local feature fusion module performs the following operations:

[0030] Add the corresponding elements of the extracted low-level features and high-level features. The calculation formula is as follows:

[0031] F R =σ(F C )+σ(F C )·δ(σ(F B ))

[0032] Where F R Represents the linear combination of features, F C and F B They represent the high-level semantic features and low-level semantic features of the remote sensing image block, σ is a 1×1 convolutional layer, and δ is a sigmoid function;

[0033] After linear combination, a lightweight network based on self-attention is used to establish the global contextual dependency of features at different levels. Specifically, three parallel 1×1 convolutional layers are used to process F R , generate query Q, key K, value V, and generate weight W through softmax function. The calculation formula is as follows:

[0034] W = Softmax(Q·K)

[0035] Finally, the enhanced features are generated, and the calculation formula is as follows:

[0036]

[0037] Step S42: Design a gate control loop fusion module for the deep semantic features of the multi-view street view image extracted in step S32. The gate control loop fusion module performs the following operations:

[0038] The deep semantic features of multi-view street view images are spliced ​​in the channel dimension. The calculation formula is as follows:

[0039] F fwd1 =Cat(F d ,F e )

[0040] F fwd2 =Cat(F e ,F d )

[0041] In the formula, F e and F d Represents the features of street view images from different perspectives matched with remote sensing image patches, F fwd1 and F fwd2 Represents two features after concatenation in the channel dimension; F fwd1 and F fwd2 Input gate control unit to obtain the fused feature F out , the calculation formula is as follows:

[0042] F out =GRU(F fwd1 ,F fwd2 )

[0043] Step S43: The features obtained by processing S41 and S42 are concatenated in the channel dimension, and the channel attention mechanism is used to adjust the weights of the features of each channel to generate the final features for classification. The final features pass through the fully connected layer to obtain the category.

[0044] Furthermore, the adaptive loss function is designed in step S5 as follows:

[0045] For the model constructed in steps S3 and S4, the cross entropy function is used as the loss function of the model to update the weights of the network and reduce the error between the actual and predicted results. The calculation formula is as follows:

[0046]

[0047] Where N represents the batch size of training, C represents the number of categories, and y j,c represents the indicator variable of the true label of the jth sample in the cth category, Represents the predicted probability of the jth sample for category c, which serves as an optimizer for urban scene classification.

[0048] Furthermore, the dynamic gradient modulation strategy designed in step S5 is specifically as follows:

[0049] SGD is used as the optimizer and the initial learning rate is set for the optimization strategy of land use scenario classification. The SGD optimizer formula is as follows:

[0050]

[0051] In the formula represents the estimated value of the gradient, η represents the learning rate, Represents the model parameters at step t; calculate the contribution of different modes to the learning objective, the calculation formula is as follows:

[0052]

[0053] In the formula, and Represents the weighted sum of the contributions of multi-view street view images and remote sensing images to classification; is an indicator function, when k = y j When , its value is 1, indicating that the predicted category is the same as the true category, otherwise the value is 0; j is the corresponding sample, and represents the weight in the fully connected layer, which is used to weight street view and remote sensing features; t is the index of the training optimization process, which is incremented each time the parameters are updated; and represents the features extracted from the street view image and remote sensing image branches; b is the bias term; in order to adjust the gradient contribution of different modalities in the learning process, an unbalanced ratio is designed. and adjustment factor To ensure that the contribution of multi-view street view images and remote sensing images to classification is balanced, the formula is as follows:

[0054]

[0055] In the formula, α is a hyperparameter that controls the degree of modulation, u represents the selected mode, and finally Integrated into the SGD optimizer, the formula is as follows:

[0056]

[0057] Furthermore, the categories of the classified sample set in step S6 include medium and low density residential areas, high density residential areas, undeveloped land, commercial land and industrial plants.

[0058] Furthermore, step S6 specifically includes the following steps:

[0059] Step S61: Select a commonly used labeling method, interpret the remote sensing image category by manual visual interpretation, and combine the matching method designed in S2 to match the street view image and the remote sensing image in spatial position, and complete the sample set production;

[0060] Step S62: Based on the sample set obtained in S61, set hyperparameters including the number of iterations and the learning rate to train the model, and store the weight file with the best effect in the validation set;

[0061] Step S63: Based on the weight file obtained in step S62, the images in the test set are predicted, and the color index is used to convert them into RGB images, and affine information and projection information are added to generate the final prediction results.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] Firstly, a geometric method for matching multi-view street view images with remote sensing images is designed to match buildings in remote sensing images with buildings in multi-view street view images. The functional features of buildings are learned to assist remote sensing images in land use classification and improve the classification accuracy. Secondly, the encoder of the pre-trained large model used in the feature extraction branch has a strong generalization ability when processing various complex and diverse image tasks based on large-scale image data. In addition, for semantic features at different levels, a global-local feature fusion module is designed to aggregate features at different levels through a self-attention mechanism; for multi-view street view images, a gate-controlled recurrent fusion module is designed to perform spatiotemporal interaction on dual-phase semantic features; finally, a dynamic gradient modulation strategy is designed to adaptively control the optimization of each modality. It provides technical support for urban land use classification, which is of great significance for understanding the laws of urban development, evaluating the impact of human activities on urban development, and providing scientific guidance for urban land planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the process of the present invention;

[0065] Figure 2 A diagrammatic illustration of the matching of a street view image and a remote sensing image according to the present invention;

[0066] Figure 3 This is a scene classification result diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following is combined with Figure 1-3 , the technical solution of the present invention is specifically described.

[0068] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0069] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0070] The present invention proposes a multimodal data urban land use scene classification method based on street view image matching remote sensing image, which specifically includes the following steps:

[0071] Step S1: Obtain high-resolution remote sensing images and road network data of the study area, and preprocess the remote sensing images; filter and streamline the road network data, and set equally spaced coordinate points. Based on the coordinate points, obtain multi-perspective street view images through the Baidu Map API interface.

[0072] Step S2: Design a cross-view image matching method based on geometric transformation to perform spatial position matching on multi-view street view images and remote sensing images.

[0073] Step S3: Based on the dual-branch neural network architecture, a pre-trained large model feature extractor is constructed to extract the multi-level semantic features of the remote sensing images after spatial position matching and the deep semantic features of the multi-view street view images.

[0074] Step S4: Construct a multimodal feature fusion and enhancement module. For the multi-level semantic features of the extracted remote sensing images, construct a global-local feature fusion module to improve the model's cognition of the overall remote sensing image and obtain enhanced features. For the deep semantic features of the extracted multi-view street view images, design a gate-controlled loop fusion module to perform spatiotemporal interaction on the dual-phase semantic features and obtain fusion features. Generate the final features for classification based on the enhanced features and fusion features and perform classification.

[0075] Step S5: Design an adaptive loss function. To address the optimization imbalance problem in the model, design a dynamic gradient modulation strategy to adaptively optimize the training process of each modality by monitoring the differences in the contributions of different modalities to the learning objectives.

[0076] Step S6: Prepare a multimodal urban land use classification sample set covering remote sensing images and street view images, including low- and medium-density residential areas, high-density residential areas, undeveloped land, commercial land, and industrial factories, and use the sample set to train and predict the model, as well as optimize and improve the model.

[0077] In this embodiment, step S1 specifically includes the following steps:

[0078] Step S11: Obtain high-resolution remote sensing images of the study area and perform preprocessing operations on the images, including radiation correction, geometric correction, image cropping and multispectral-panchromatic fusion operations.

[0079] Step S12: Download the Open Street Map road network data, filter and clean the road network data, including filtering and removing road networks with chaotic topological relationships, mountain road networks, and road networks within residential areas. Set equally spaced coordinate points at intervals of 100 meters, and based on the coordinate points, obtain street view images of each point at 0 degrees (directly in front), 90 degrees (right side), 180 degrees (directly behind), and 270 degrees (left side) through the Baidu Map API interface.

[0080] In this embodiment, step S2 specifically includes the following steps:

[0081] Step S21: For the remote sensing image preprocessed in S1, a projection transformation method based on pixel resolution is used to convert the remote sensing image from geographic coordinates to pixel coordinates. The calculation formula is as follows:

[0082]

[0083] Where C a and C b They represent the geographic coordinates of the upper left corner of the image; H and W represent the pixel resolution in the image; and Represents the pixel coordinates of pixel i; and The pixel column and row number representing the pixel coordinates;

[0084] Step S22: Based on the processed remote sensing image, select two adjacent points on the street, take one point as the center, record the coordinates of the center point as (x1, t1), and record the other point as (x2, y2). The line connecting the two points can be regarded as the driving direction of the vehicle, and the center point can be regarded as the vehicle shooting point. Calculate the angle between the driving direction and the row of the pixel coordinate system. The calculation formula is as follows:

[0085]

[0086] The result calculated by atan2 is the radian of rotation, which can be obtained by multiplying Convert to an angle to get the desired rotation angle, usually between -180 and 180 degrees.

[0087] Step S23: Based on the processed remote sensing image and the calculated rotation angle, the image is rotated to be parallel to the vehicle's driving direction and the row of the pixel coordinate system using an affine transformation matrix to avoid data redundancy caused by black edges generated by subsequent cropping. The affine transformation matrix is ​​shown below:

[0088]

[0089] Step S24: Divide the rotation-corrected remote sensing image into four equal blocks in a clockwise direction based on the center point determined in step S22. The multi-view street view image captured at the shooting point is further subdivided into eight equally divided areas. Each remote sensing image block corresponds to a part of each of two adjacent street view perspectives, and this correspondence is determined by the perspective characteristics of the street view and the remote sensing image. Specifically, the street view image is shot in the front, rear, left, and right directions of the vehicle, and the perspective in each direction slightly covers the area in the adjacent direction. For example, when shooting directly in front, the perspective also includes parts of the left front and right front areas. Similarly, the remote sensing image is acquired by looking down and has a larger coverage area. When the remote sensing image is divided into four equal parts, each part will cover the ground area on the left and right sides of the vehicle. For example, the remote sensing image portion in the left front covers the ground on the left and in front at the same time. (See Figure 2 )

[0090] In this embodiment, step S3 specifically includes the following steps:

[0091] Step S31: In the remote sensing image feature extraction branch, select the encoder of the pre-trained large model (such as Fast Sam) as the remote sensing image feature extraction backbone network to extract multi-level semantic features of the remote sensing image;

[0092] Step S32: In the multi-view street view image feature extraction branch, select the encoder of a pre-trained large model (such as Fast Sam) as the multi-view street view image feature extraction backbone network to extract deep semantic features of the multi-view street view images.

[0093] In this embodiment, step S4 specifically includes the following steps:

[0094] Step S41: Design a global-local feature fusion module for the multi-level semantic features extracted in S31. Specifically, use the attention mechanism to refine high-level features with low-level semantic features, so that the model can maintain attention to important details while recognizing the macro background of the image, thereby achieving a better balance between fine classification and scene understanding. The main approach is as follows:

[0095] Add the corresponding elements of the extracted low-level features and high-level features. The calculation formula is as follows:

[0096] F R =σ(F C )+σ(F C )·δ(σ(F B ))

[0097] Where F RRepresents the linear combination of features, F C and F B They represent the high-level semantic features and low-level semantic features of the remote sensing image block, σ is a 1×1 convolution layer, and δ is a sigmoid function. After being combined, a lightweight network based on self-attention is used to establish the global contextual dependency of features at different levels. Specifically, three parallel 1×1 convolution layers are used to process F R , generate Query (Q), Key (K), Value (V), and generate weight W through the softmax function. The calculation formula is as follows:

[0098] W = Softmax(Q·K)

[0099] Finally, the enhanced features are generated, and the calculation formula is as follows:

[0100]

[0101] Step S42: Design a gate control loop fusion module for the deep semantic features of the multi-view street view images extracted in step S32. Specifically, there is a time difference in the shooting of the street view images, and some of the perspectives overlap. The gate control loop feature fusion module can combine the depth of historical and current information to perform spatiotemporal interaction on the dual-phase semantic features. The main steps are as follows:

[0102] The deep semantic features of multi-view street view images are spliced ​​in the channel dimension. The calculation formula is as follows:

[0103] F fwd1 =Cat(F d ,F e )

[0104] F fwd2 =Cat(F e ,F d )

[0105] In the formula, F e and F d Represents the features of street view images from different perspectives matched with remote sensing image patches, F fwd1 and F fwd2 Represents two features after concatenation in the channel dimension. fwd1 and F fwd2 Input gate control unit to obtain the fused feature F out , the calculation formula is as follows:

[0106] F out =GRU(F fwd1 ,F fwd2 )

[0107] Step S43: The features obtained by processing S41 and S42 are concatenated in the channel dimension. Considering the different contributions of remote sensing images and multi-view street view images to the final classification, the channel attention mechanism is used to adjust the weights of each channel feature to generate the final features for classification. The final features are passed through the fully connected layer to obtain the category.

[0108] In this embodiment, step S5 specifically includes the following steps:

[0109] Step S51: For the model constructed in steps S3 and S4, a loss function is designed to update the weights of the network to reduce the error between the actual and predicted results. Specifically, the cross entropy function is used as the loss function of the model, and the calculation formula is as follows:

[0110]

[0111] Where N represents the batch size of training, C represents the number of categories, and y j,c represents the indicator variable of the true label of the jth sample in the cth category, represents the predicted probability of the jth sample for category c, which serves as an optimizer for urban scene classification;

[0112] Step S52: Use SGD as the optimizer and set the initial learning rate for the optimization strategy of land use scene classification. Considering the problem of imbalanced optimization in the training process of remote sensing images and multi-view street view images, a dynamic gradient modulation strategy is designed. The main steps are as follows:

[0113] The SGD optimizer formula is as follows:

[0114]

[0115] In the formula represents the estimated value of the gradient, η represents the learning rate, Represents the model parameters at step t. The contribution of different modes to the learning objectives is calculated as follows:

[0116]

[0117] In the formula, and Represents the weighted sum of the contributions of multi-view street view images and remote sensing images to classification; is an indicator function, when k = y j When , its value is 1, indicating that the predicted category is the same as the true category, otherwise the value is 0; j is the corresponding sample, and represents the weight in the fully connected layer, which is used to weight street view and remote sensing features; t is the index of the training optimization process, which is incremented each time the parameters are updated; and represents the features extracted from the street view image and remote sensing image branches; b is the bias term. In order to adjust the gradient contribution of different modalities in the learning process, an unbalanced ratio is designed. and adjustment factor This ensures that their contributions to the classification are balanced. The formula is as follows:

[0118]

[0119] Where α is a hyperparameter that controls the degree of modulation, and u represents the selected mode, which is generally the mode that dominates the training. Integrated into the SGD optimizer, the formula is as follows:

[0120]

[0121] In this embodiment, step S6 specifically includes the following steps:

[0122] Step S61: Select a commonly used labeling method, interpret the remote sensing image category by manual visual interpretation, and combine the matching method designed in S2 to match the street view image and the remote sensing image in spatial position, and complete the sample set production;

[0123] Step S62: Based on the sample set obtained in S61, set reasonable hyperparameters such as the number of iterations and learning rate to train the model, and store the weight file with the best effect in the validation set;

[0124] Step S63: Based on the weight file obtained in step S62, the images in the test set are predicted, and the color index is used to convert them into RGB images, and affine information and projection information are added to generate the final prediction results.

[0125] In this real example, the main urban area of ​​XX City, XX Province is used as the research object, and the street view images and remote sensing images of the main urban area in 2020 are used to classify the urban scenes, using red, green and blue bands. This example uses 1,000 remote sensing images with a pixel size of 256×256 and 2,000 street view images with a pixel size of 512×400 and corresponding annotation data for model training, which are divided into 5 categories of scenes, including low- and medium-density residential areas, high-density residential areas, commercial land, industrial factories, and undeveloped land. Figure 3 For detailed classification results, Figure 3 The results include the classification of commercial land using street view as an aid and the classification of commercial land without using street view.

[0126] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A multimodal data urban land use scene classification method based on street view image matching remote sensing image, comprising the following steps: Step S1: Obtain high-resolution remote sensing images and road network data of the study area; perform preprocessing operations on the remote sensing images; filter and streamline the road network data, set equally spaced coordinate points, and obtain multi-view street view images of each coordinate point; Step S2: Design a cross-view image matching method based on geometric transformation to perform spatial position matching on multi-view street view images and remote sensing images; Step S3: Based on the dual-branch neural network architecture, a pre-trained large model feature extractor is constructed to extract the multi-level semantic features of the remote sensing image after spatial position matching and the deep semantic features of the multi-view street view image; Step S4: construct a multimodal feature fusion and enhancement module; construct a global-local feature fusion module for the multi-level semantic features of the extracted remote sensing image, improve the model's cognition of the overall remote sensing image and obtain enhanced features; According to the deep semantic features extracted from multi-view street view images, a gate-controlled recurrent fusion module is designed to perform spatiotemporal interaction on the bi-temporal semantic features and obtain fused features. Generate final features for classification based on enhanced features and fused features and perform classification; Step S5: Design an adaptive loss function to address the optimization imbalance problem in the model and design a dynamic gradient modulation strategy to adaptively optimize the training process of each modality by monitoring the difference in the contribution of different modalities to the learning objectives; Step S6: Create a multimodal urban land use classification sample set covering remote sensing images and street view images, and use the sample set to train and predict the model, as well as optimize and improve the model.

2. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1 is characterized in that: The preprocessing operation of the remote sensing image described in step S1 is specifically: obtaining a high-resolution remote sensing image of the study area, and performing preprocessing operations on the image, including radiation correction, geometric correction, image cropping and multispectral-panchromatic fusion operations.

3. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1 is characterized in that: The step S1 is to filter and simplify the road network data, set equally spaced coordinate points, and obtain a multi-view street view image of each coordinate point, specifically: Download Open Street Map road network data, filter and clean the road network data, including filtering and removing road networks with chaotic topological relationships, mountain road networks, and road networks within residential areas; set equally spaced coordinate points at intervals of preset lengths, and based on the coordinate points, obtain street view images of each point at the perspectives of -0 degrees in front, -90 degrees to the right, -180 degrees behind, and -270 degrees to the left through the Baidu Map API interface.

4. According to the multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1, step S2 specifically comprises the following steps: Step S21: For the remote sensing image preprocessed in S1, a projection transformation method based on pixel resolution is used to convert the remote sensing image from geographic coordinates to pixel coordinates. The calculation formula is as follows: Where C a and C b They represent the geographic coordinates of the upper left corner of the image; H and W represent the pixel resolution in the image; and Represents the pixel coordinates of pixel i; and The pixel column and row number representing the pixel coordinates; Step S22: Based on the remote sensing image processed in step S21, two adjacent points on the street are selected, with one point as the center, the coordinates of the center point are recorded as (x1, y1), and the coordinates of the other point are recorded as (x2, y2). The center point is used as the vehicle shooting point, and the direction from the center point to the other point is used as the vehicle driving direction. The angle between the vehicle driving direction and the row of the pixel coordinate system is calculated. The calculation formula is as follows: The result calculated by atan2 is the radian of rotation, which can be obtained by multiplying Convert to angle to get the required rotation angle A; Step S23: Based on the remote sensing image processed in step S21 and the rotation angle calculated in step S22, the remote sensing image is rotated to a point where the vehicle's driving direction is parallel to the row of the pixel coordinate system using an affine transformation matrix to avoid data redundancy caused by black edges generated by subsequent cropping; the affine transformation matrix X is shown below: Step S24: the remote sensing image that has been rotationally corrected is divided into four equal blocks in a clockwise direction based on the center point determined in step S22; the multi-perspective street view image taken at the shooting point is subdivided into eight equal areas to achieve spatial position matching between the multi-perspective street view image and the remote sensing image; wherein the street view image is shot in four directions, namely, the front, rear, left and right directions of the vehicle, and the street view image in each shooting direction is subdivided into two equal areas; each remote sensing image block corresponds to one equal area of ​​the street view images in two adjacent shooting directions, and the corresponding relationship is determined by the viewing angle characteristics of the street view and the remote sensing image.

5. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31: In the remote sensing image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract multi-level semantic features of the remote sensing image; Step S32: In the multi-view street view image feature extraction branch, the encoder of the pre-trained large model is selected as the backbone network to extract deep semantic features of the multi-view street view image.

6. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1 is characterized in that: Step S4 specifically includes the following steps: Step S41: Design a global-local feature fusion module for the multi-level semantic features extracted in S31. The global-local feature fusion module performs the following operations: Add the corresponding elements of the extracted low-level features and high-level features. The calculation formula is as follows: F R =σ(F C )+σ(F C )·δ(σ(F B )) Where F R Represents the linear combination of features, F C and F B They represent the high-level semantic features and low-level semantic features of the remote sensing image block, σ is a 1×1 convolutional layer, and δ is a sigmoid function; After linear combination, a lightweight network based on self-attention is used to establish the global contextual dependency of features at different levels. Specifically, three parallel 1×1 convolutional layers are used to process F R , generate query Q, key K, value V, and generate weight W through softmax function. The calculation formula is as follows: W = Softmax(Q·K) Finally, the enhanced features are generated, and the calculation formula is as follows: Step S42: Design a gate control loop fusion module for the deep semantic features of the multi-view street view image extracted in step S32. The gate control loop fusion module performs the following operations: The deep semantic features of multi-view street view images are spliced ​​in the channel dimension. The calculation formula is as follows: F fwd1 =Cat(F d ,F e ) F fwd2 =Cat(F e ,F d ) In the formula, F e and F d Represents the features of street view images from different perspectives matched with remote sensing image patches, F fwd1 and F fwd2 Represents two features after concatenation in the channel dimension; F fwd1 and F fwd2 Input gate control unit to obtain the fused feature F out , the calculation formula is as follows: F out =GRU(F fwd1 ,F fwd2 ) Step S43: The features obtained by processing S41 and S42 are concatenated in the channel dimension, and the channel attention mechanism is used to adjust the weights of the features of each channel to generate the final features for classification. The final features pass through the fully connected layer to obtain the category.

7. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1 is characterized in that: The adaptive loss function designed in step S5 is as follows: For the model constructed in steps S3 and S4, the cross entropy function is used as the loss function of the model to update the weights of the network and reduce the error between the actual and predicted results. The calculation formula is as follows: Where N represents the batch size of training, C represents the number of categories, and y j,c represents the indicator variable of the true label of the jth sample in the cth category, Represents the predicted probability of the jth sample for category c, which serves as an optimizer for urban scene classification.

8. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1 is characterized in that: The dynamic gradient modulation strategy designed in step S5 is specifically as follows: SGD is used as the optimizer and the initial learning rate is set for the optimization strategy of land use scenario classification. The SGD optimizer formula is as follows: In the formula represents the estimated value of the gradient, η represents the learning rate, Represents the model parameters at step t; calculate the contribution of different modes to the learning objective, the calculation formula is as follows: In the formula, and Represents the weighted sum of the contributions of multi-view street view images and remote sensing images to classification; is an indicator function, when k = y j When , its value is 1, indicating that the predicted category is the same as the true category, otherwise the value is 0; j is the corresponding sample, and Represents the weights in the fully connected layer, used to weight street view and remote sensing features; t is the index of the optimization process of training, which is incremented each time the parameters are updated; and Represents the features extracted from the street view image and remote sensing image branches; b is the bias term; in order to adjust the gradient contribution of different modes in the learning process, the imbalance ratio is designed and adjustment factor To ensure that the contribution of multi-view street view images and remote sensing images to classification is balanced, the formula is as follows: In the formula, α is a hyperparameter that controls the degree of modulation, u represents the selected mode, and finally Integrated into the SGD optimizer, the formula is as follows:

9. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1, characterized in that: The categories of the classification sample set in step S6 include medium and low density residential areas, high density residential areas, undeveloped land, commercial land and industrial factories.

10. The multimodal data urban land use scene classification method based on street view image matching remote sensing image according to claim 1, characterized in that: Step S6 specifically includes the following steps: Step S61: Select a commonly used labeling method, interpret the remote sensing image category by manual visual interpretation, and combine the matching method designed in S2 to match the street view image and the remote sensing image in spatial position, and complete the sample set production; Step S62: Based on the sample set obtained in S61, set hyperparameters including the number of iterations and the learning rate to train the model, and store the weight file with the best effect in the validation set; Step S63: Based on the weight file obtained in step S62, the images in the test set are predicted, and the color index is used to convert them into RGB images, and affine information and projection information are added to generate the final prediction results.

Images