A satellite image socio-economic prediction method based on POI anchoring and visual enhancement
By employing POI anchoring and visual enhancement methods, the stability and accuracy of socioeconomic forecasting based on satellite imagery have been improved. This has resolved the forecasting instability issue in scenarios with heterogeneous image quality, and enabled more reliable forecasting of socioeconomic indicators.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JILIN UNIVERSITY
- Filing Date
- 2026-06-17
- Publication Date
- 2026-07-21
Smart Images
Figure CN122435480A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary fields of computer vision, multimodal learning, and urban computing, specifically to a method for socioeconomic prediction based on satellite imagery using POI anchoring and visual enhancement. Background Technology
[0002] With the development of remote sensing technology and deep learning, using satellite imagery to predict urban socioeconomic indicators (such as income levels, education levels, and unemployment rates) has become an efficient alternative to traditional population censuses. Existing technologies are mainly divided into two categories:
[0003] Single-modal vision methods: These methods utilize convolutional neural networks (such as ResNet (residual network)) or ViT (visual transformer) to directly extract features from satellite images for regression prediction.
[0004] Visual-language multimodal methods: Represented by UrbanCLIP (Urban Contrastive Language-Image Pretraining), these methods utilize pre-trained large language models (LLMs) to generate text descriptions based on satellite images. By contrastive learning, visual and textual features are aligned, thereby introducing semantic information to enhance predictive capabilities.
[0005] Despite some progress in existing methods, serious “prediction instability” still exists in practical applications, which manifests as semantic instability: when remote sensing images are obscured by clouds or fog, have reduced resolution, or have sensor differences, text descriptions generated directly from the images are prone to semantic ambiguity or functional misjudgment, resulting in unstable prediction results.
[0006] Insufficient robustness in representation: Existing methods focus on improving the accuracy of single predictions and are not specifically designed for heterogeneous image quality scenes, resulting in significant differences in features obtained for the same region under different image conditions; Insufficient utilization of scale information: Existing technologies often use single-scale or simple fusion methods to extract image features, making it difficult to simultaneously take into account both macroscopic spatial structure and local detail information. Summary of the Invention
[0007] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.
[0008] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:
[0009] S1: Multimodal data collection and preprocessing: First, define the urban area set, collect satellite image data, POI data, and label socioeconomic indicators;
[0010] S2: Construct a semantic stability branch and use POI data to guide a multimodal large model to generate text.
[0011] S3: Constructing the visual stability branch:
[0012] Satellite-customized data augmentation: for original images Perform randomized data augmentation operations;
[0013] Multi-scale feature extraction: ResNet-18 was used as the backbone network to process the original images separately. The enhanced view extracts the feature maps from the output of the four residual blocks, forming a bottom-up path;
[0014] Feature pyramid fusion: Constructing top-down paths and lateral connections;
[0015] Global feature aggregation: Perform global average pooling on the feature maps of each layer of the pyramid, and concatenate the results to obtain a unified image vector representation;
[0016] Projection mapping: Mapping the original image features and the enhanced image features separately using two independent projection heads;
[0017] S4: Model pre-training to construct a composite loss function Jointly optimize image and text alignment and image Figure 1 To the point of being compatible;
[0018] S5: Fine-tuning of downstream socioeconomic forecasting tasks: Supervised fine-tuning of the parameters of the MLP predictor is performed only using the gradient descent algorithm.
[0019] As a preferred embodiment of the satellite image socioeconomic prediction method based on POI anchoring and visual enhancement described in this invention, the urban area set in S1 is defined as follows: For each urban area ;
[0020] The method for acquiring the satellite image data is as follows: acquiring regional... The corresponding satellite image is represented as ,in and These represent the height and width of the image, respectively.
[0021] The method for obtaining the POI data is as follows: Obtain the region Collect objective points of interest data within the area, count the names of each POI category and their frequency within the area, and construct a structured POI list dictionary. ;
[0022] The method for obtaining the labeling of socioeconomic indicators is as follows: obtaining the regional... The corresponding real socioeconomic statistical survey data are logarithmically normalized and smoothed using the following formula: After processing As a supervisory label during the fine-tuning phase.
[0023] As a preferred embodiment of the satellite image socioeconomic forecasting method based on POI anchoring and visual enhancement described in this invention, the specific method of S2 includes:
[0024] Multimodal suggestion generation: Combine satellite images acquired in S1 with the statistically analyzed list of POIs. Simultaneously used as input;
[0025] Text description generation: Using the list of POIs as context anchors, text descriptions of the regions are generated using LMM. The process can be formalized as follows:
[0026]
[0027] in, It provides factual information that is decoupled from visual input;
[0028] Text feature extraction: The generated text is processed using a pre-trained text encoder, BERT. The text is encoded using the BERT encoder to obtain the original text representation vector. and through the projection head Mapping to the feature space yields the final text embedding vector. :
[0029]
[0030]
[0031] During this process, the parameters of the BERT text encoder are kept frozen to take advantage of its general language understanding capabilities.
[0032] As a preferred embodiment of the satellite image socioeconomic prediction method based on POI anchoring and visual enhancement described in this invention, the specific steps for constructing the visual stability branch in step S3 include:
[0033] S3.1: Satellite-customized data augmentation of the original image Perform the following data augmentation operations sequentially or in random combinations: random saturation adjustment, Gaussian noise injection, random erasure, and random rotation, generating two different augmented views, denoted as follows: and ;
[0034] S3.2: Multi-scale feature extraction: ResNet-18 is used as the backbone network to process the original images separately. First Enhanced View Second Enhanced View ;
[0035] For any input image X∈ The feature maps output from its four residual blocks are extracted to form a bottom-up feature pyramid hierarchy, denoted as . ,in:
[0036] The feature map output by the first residual block has a resolution of 1 / 4 of the original image and contains rich local detail information.
[0037] The feature map output by the second residual block has a resolution of 1 / 8 of the original image and contains semantic information at a medium scale.
[0038] The feature map output by the third residual block has a resolution of 1 / 16 of the original image and contains higher-level regional structure information.
[0039] The feature map output by the fourth residual block has a resolution of 1 / 32 of the original image, but it has the strongest global semantic information.
[0040] S3.3: Feature Pyramid Fusion: Constructing top-down paths and lateral connections, fusing features at each level. The specific calculation process is as follows: First, for the top-level features... Apply 1×1 convolution, To obtain a unified number of channels :
[0041]
[0042] Then, for hierarchical indexes Decrease from 4 to 2, performing upsampling and lateral connections sequentially:
[0043]
[0044] in, This represents a bilinear upsampling operation, used to upsample high-level feature maps (...). The size is enlarged to match the lower-level feature map. The dimensions are consistent; finally, the fused feature maps are... A 3×3 convolution is applied to eliminate upsampling aliasing, resulting in the final pyramid feature map. ;
[0045] ;
[0046] The final pyramid feature atlas was obtained. ;
[0047] S3.4: Global Feature Aggregation: Feature maps of each layer of the pyramid , , , Perform global average pooling separately to obtain pooling vectors. These four vectors are then concatenated along the channel dimension to obtain a unified image vector representation. ;
[0048] Following the same process, the two enhanced views were processed separately. and After processing, the corresponding result is obtained. and ;
[0049]
[0050] Where AP represents global average pooling. This indicates splicing along the channel dimension;
[0051] S3.5: Projection Mapping: After obtaining the original image vector representation Compared with the enhanced image representation and Then, by constructing two independent projection heads: the first projection head Second projection head The vector representation of the original image The first projection head is input and mapped to the shared feature space to obtain the final original image embedding vector. :
[0052]
[0053] Representing the two enhanced views as vectors and The input to the second projection head is mapped to the same shared feature space to obtain the final enhanced image embedding vector. and :
[0054] .
[0055] As a preferred embodiment of the satellite image socioeconomic prediction method based on POI anchoring and visual enhancement described in this invention, the specific steps of constructing the composite loss function and performing joint optimization in step S4 include:
[0056] Constructing a composite loss function for model pre-training This function is the image-text contrast loss. Image-to-image contrast loss Weighted sum:
[0057] Composite loss function
[0058] in and These are weight hyperparameters; This represents the contrast loss between the text representation and the original image; This indicates the contrast loss between two enhanced views;
[0059] The method for calculating the loss between image and text comparison is as follows:
[0060] Set the batch size for training to N, meaning each batch contains N samples from different regions;
[0061] For the i-th sample in the batch, i = 1, 2, …, N, its original image embedding vector is denoted as . Its text embedding vector is denoted as ;
[0062] First, calculate the one-way contrast loss from the image to the text direction. :
[0063]
[0064] Then, the one-way contrast loss from the text to the image direction is calculated. :
[0065]
[0066] Finally, the loss in comparison between text and images. Take the average of the two unidirectional losses mentioned above:
[0067]
[0068] Where exp is an abbreviation for the natural exponential function. Represents cosine similarity. For temperature parameters;
[0069] The calculation process for image-to-image contrast loss is as follows:
[0070] For the two augmented view embedding vectors of the i-th sample in the batch, denoted as follows: and ;
[0071] First, calculate the unidirectional contrast loss from the first augmented view to the second augmented view:
[0072]
[0073] Then, the unidirectional contrast loss in the direction from the second augmented view to the first augmented view is calculated:
[0074]
[0075] Finally, image-image contrast loss Take the average of the two unidirectional losses mentioned above:
[0076]
[0077] in, The cosine similarity function is represented, tau is the preset temperature parameter, the first term in the denominator sums all N second augmented view samples in the batch, and the second term sums all first augmented view samples in the batch except for its own index i.
[0078] During model pre-training, the composite loss function is minimized. Jointly optimize image and text alignment and image Figure 1 To the point of being responsive.
[0079] As a preferred embodiment of the satellite image socioeconomic prediction method based on POI anchoring and visual enhancement described in this invention, the specific method of S5 is as follows: after pre-training convergence, the parameters of the visual encoder trained in S3 are retained and its network parameters are solidified. For the target urban area to be predicted, its remote sensing image is input to the frozen visual encoder to extract a high-dimensional feature vector that is robust to environmental disturbances and contains multi-scale information. A lightweight multilayer perceptron is connected to the back end of the visual encoder as a regression predictor, and the mean squared error loss is calculated using the specific task labels collected in S1.
[0080] Compared with existing technologies, the beneficial effects of this invention are as follows: By introducing a POI anchoring mechanism and a specific visual enhancement strategy, this invention overcomes the shortcomings of existing technologies in predicting failure when image quality changes, achieving significant progress. In the presence of image quality degradation, the generated regional semantic description is more stable and reliable, avoiding prediction bias caused by semantic misleading. Under different image perturbation conditions, the fluctuation range of the prediction results for the same region is significantly reduced, and the stability improvement can reach more than 50%. While maintaining prediction stability, the overall prediction accuracy is improved simultaneously, and the performance is more significant on complex socio-economic indicators. Attached Figure Description
[0081] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:
[0082] Figure 1 This is a framework diagram of a satellite image socioeconomic forecasting method based on POI anchoring and visual enhancement according to the present invention.
[0083] Figure 2 This is a flowchart illustrating the overall process of a satellite image socioeconomic forecasting method based on POI anchoring and visual enhancement, as described in this invention.
[0084] Figure 3 This is a comparison chart of the true and predicted values of a satellite image socioeconomic prediction method based on POI anchoring and visual enhancement according to the present invention. Detailed Implementation
[0085] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0086] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.
[0087] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0088] This invention aims to enhance the model's ability to resist environmental interference in both semantic and visual dimensions by constructing a multi-layered robust learning mechanism. Specifically, in terms of semantic stability, a POI-anchored text generation strategy is proposed. Unlike traditional methods that rely solely on visual information to generate descriptions, this strategy introduces objectively existing point-of-interest (POI) distribution data within urban areas as "semantic anchors" into the generation process of a multimodal large model. Leveraging the strong indicative nature of POI data on regional functions, it forces the model to generate high-quality text descriptions that are factually accurate and rich in socioeconomic attributes, unaffected by image visual quality degradation. This provides reliable semantic supervision signals for subsequent training. In terms of visual stability, a customized enhancement (STA) module adapted to satellite remote sensing characteristics is designed. This module simulates imaging interference in the real environment through operations such as random saturation adjustment, noise injection, and rotation erasure. Graph-to-graph contrast learning forces the model to eliminate irrelevant interference such as lighting and season, focusing on learning visual representations that reflect the essential structure of the city. Furthermore, this invention integrates a Feature Pyramid Network (FPN). Through a multi-scale feature fusion mechanism, it ensures that the model can simultaneously capture macroscopic regional layout and microscopic building textures, avoiding the loss of scale-sensitive information due to image quality degradation. Finally, by jointly optimizing the image-text contrast loss and the visual enhancement contrast loss, visual features and high-quality semantic features are aligned in the latent space, achieving a socioeconomic prediction model that is highly robust to changes in input image quality. This effectively overcomes the challenge of performance fluctuations in complex real-world scenarios encountered by existing technologies. The overall framework consists of two stages: pre-training and fine-tuning, with the structure as follows: Figure 1 As shown.
[0089] This invention reduces the model's sensitivity to satellite image quality, achieving stable and accurate predictions. It primarily enhances model robustness from two levels: semantic and visual. Semantically, it uses points of interest as anchors to assist the large language model in generating reliable region descriptions, avoiding generation illusions caused by low-quality images and improving the consistency between region representation and actual function. Visually, it employs a robustness-centric training strategy, extracting inherent structural invariance from the enhanced multi-scale image features. Validated in real-world urban scenarios, this invention demonstrates significant advantages over existing methods and effectively reduces performance fluctuations when images are disturbed.
[0090] For details, please refer to Figure 1 (In the image, POI retrieval means obtaining the set of POIs (Points of Interest, such as schools, banks, restaurants, gas stations, hospitals, supermarkets, etc.) within the coverage area of the satellite image; P2, P3, P4, and P5 in the image represent the final pyramid feature set; in the image...) , These represent two satellite image vectors after enhancement in the i-th region of a total batch size of N; in the figure , These represent the original satellite image vector of the i-th region with a batch size of N and the corresponding text embedding vector of the i-th region, respectively; the FPN visual encoder in the figure represents the global feature aggregation process; the visual encoder in the fine-tuning part of stage 2 in the figure refers to the process of feature pyramid fusion. A satellite image socioeconomic prediction method based on POI (point of interest) anchoring and visual enhancement includes the following steps:
[0091] Step 1: Multimodal data collection and preprocessing:
[0092] First, define the urban region set as... For each city area Collect the following three types of data.
[0093] Satellite imagery data: acquisition and regional The corresponding satellite image is represented as ,in and Representing the height and width of the image respectively (e.g.) (pixels).
[0094] POI Data: Obtaining Regions Objective Points of Interest (POI) data within the specified area. Statistical analysis of the names of each POI category and their frequency within the area, constructing a structured POI list dictionary. For example, extract the specific quantities for different categories such as catering, schools, medical care, and entertainment.
[0095] Socioeconomic indicator labeling: obtaining regional data The corresponding real socioeconomic statistical survey data (such as education level index, average income level, unemployment rate, crime rate, etc.). To eliminate the adverse effects of long-tail distribution and extreme values on the neural network training gradient, the original statistical values are logarithmically normalized and smoothed. The specific transformation formula is as follows: After processing As a supervisory label during the fine-tuning phase.
[0096] Step 2: Construct the semantic stability branch:
[0097] To obtain high-quality semantic descriptions unaffected by image quality, POI data is used to guide a large multimodal model (LMM) to generate text:
[0098] Multimodal cue generation: This involves generating satellite images from step 1. and the statistical list of POIs It also serves as input.
[0099] Text description generation: Using the list of Points of Interest (POIs) as contextual anchors, text descriptions of regions are generated using an LMM (Multimodal Large Language Model). The process can be formalized as follows:
[0100]
[0101] in, Provides factual information decoupled from visual input, ensuring the generated description It contains accurate information on functional zoning and human activities.
[0102] Text feature extraction: The generated text is processed using a pre-trained text encoder, BERT. The text is encoded using the BERT encoder to obtain the original text representation vector. and through the projection head Mapping to the feature space yields the final text embedding vector. :
[0103]
[0104]
[0105] During this process, the parameters of the BERT text encoder are kept frozen to take advantage of its general language understanding capabilities.
[0106] Step 3: Construct the visual stability branch:
[0107] To extract multi-scale visual features that are robust to ambient noise, the following steps are performed:
[0108] Satellite Custom Data Augmentation (STA): Augmentation of raw images Perform randomized data augmentation operations, including random saturation adjustment, Gaussian noise injection, random erasure, and random rotation, to generate two different augmented views, denoted as follows: and .
[0109] Multi-scale feature extraction (Backbone): Using ResNet-18 as the backbone network, the original images are processed separately. And its enhanced view. Extract feature maps from the output of the four residual blocks (layer1 to layer4).
[0110] Recorded as This forms a bottom-up path.
[0111] C2: The feature map output by the first residual block (layer1), with a high resolution (1 / 4 of the original image) and rich local detail information (such as building edges and textures).
[0112] C3: The feature map output by the second residual block (layer2) has a resolution of 1 / 8 of the original image and contains medium-scale semantic information (such as building clusters).
[0113] C4: The feature map output by the third residual block (layer3), with a resolution of 1 / 16 of the original image, contains higher-level regional structure information (such as street layout).
[0114] C5: The feature map output by the fourth residual block (layer 4) has the lowest resolution (1 / 32 of the original image) but the strongest global semantic information (such as the type of urban functional area).
[0115] Feature Pyramid Fusion (FPN): Constructs top-down paths and lateral connections. First, the top-level feature C5 is processed, then upsampled layer by layer and fused with features from lower layers. The calculation process is as follows:
[0116]
[0117]
[0118] In the formula: express The convolution operation is used to adjust the number of channels; This represents the feature map after convolution and channel adjustment; This indicates an upsampling operation (such as bilinear interpolation) used to enlarge the size of high-level feature maps to match that of low-level feature maps. Figure 1 To;
[0119] in The values are 4, 3, and 2. To eliminate the aliasing effect caused by upsampling, the fused feature map is subjected to... Convolution is performed to obtain the final pyramid feature map.
[0120]
[0121] Global Feature Aggregation: Perform global average pooling (AP) on the feature maps of each layer of the pyramid, and concatenate the results to obtain a unified image vector representation.
[0122]
[0123] Where AP represents global average pooling. This indicates splicing along the channel dimension;
[0124] Similarly, the above process is performed on the enhanced view to obtain... and .
[0125] Projection mapping: via two independent projection heads and Map the original image features and the enhanced image features separately:
[0126]
[0127] .
[0128] Step 4: Model pre-training (joint contrastive learning optimization) to construct a composite loss function Jointly optimize image and text alignment and image Figure 1 Consistency, the formula is as follows:
[0129]
[0130] in and These are weight hyperparameters; This represents the contrast loss between the text representation and the original image; This represents the contrast loss between two enhanced views.
[0131] Image and text comparison of losses ): Aiming to bring visual embedding closer With text embedding The distance. For batch size The calculation formula is:
[0132]
[0133] Among them, one-way loss (such as) ) is defined as:
[0134]
[0135] In the formula, exp is an abbreviation for the natural exponential function. Represents cosine similarity. This refers to the temperature parameter.
[0136] Image-image contrast loss ( ): Designed to enable two enhanced views and The characteristics remain consistent. The calculation formula is:
[0137]
[0138] in:
[0139]
[0140] Step 5: Fine-tuning of downstream socio-economic forecasting tasks
[0141] After pre-training converges, the visual encoder parameters trained in step 3 (including the ResNet-18 backbone and FPN network) are retained and their network parameters are frozen. For the target urban area to be predicted, its remote sensing image is input to the frozen visual encoder to extract a high-dimensional feature vector that is robust to environmental disturbances and contains multi-scale information. A lightweight multilayer perceptron (MLP) is integrated as a regression predictor at the back end of the visual encoder. The mean squared error loss is calculated using the specific task labels collected in step 1 (such as logarithmized crime rates and income data). A gradient descent algorithm is employed to perform supervised fine-tuning of the MLP predictor parameters only, achieving accurate mapping regression from image features to continuous socioeconomic indicators.
[0142] Experimental results:
[0143] To verify the feasibility and effectiveness of the above model in real-world business scenarios, this embodiment uses New York City as the application target area for a full-process instantiation application. Grid division and data placement: Based on the "Census Block Group" defined by the U.S. Census Bureau as the basic spatial prediction unit, irregular boundary segmentation of the region is achieved.
[0144] The specific task of obtaining high-resolution (4.7 meters / pixel) data covering the entire city of New York is defined as follows: four core indicators in urban governance are selected for prediction, including: crime rate prediction (Crime data obtained from NYC Open Data), education development indicators (Edu data provided by SafeGraph), economic level indicators (Income data provided by NYC Open Data), and employment stability indicators (Unemployment rate data provided by LAUS).
[0145] Fine-tuning strategy deployment: The acquired labeled New York dataset was strictly divided into training, validation, and test sets in a 6:2:2 ratio. The training set was used to perform supervised fine-tuning of the MLP backend of the model, and blind testing was conducted on the test set. The significant performance improvements achieved on the two prediction tasks are shown in Table 1.
[0146] Table 1: Performance comparison of different methods on the two tasks
[0147]
[0148] The coefficient of determination (COD) represents the proportion of variation in the actual values explained by the model's predictions. Its value typically ranges from 0 to 1; a value closer to 1 indicates a better model fit (more accurate prediction). The root mean square error (RMSE) measures the average deviation between the predicted and actual values, and is more sensitive to larger errors. A smaller value is better, and the unit is the same as the original label. The mean absolute error (MAE) represents the average absolute difference between the predicted and actual values, reflecting the typical degree of prediction deviation. A smaller value is better, as it is more robust to outliers than RMSE.
[0149] Table 2: Comparison of robust stability of various models under multiple image degradation scenarios
[0150]
[0151] As shown in Table 2, stability tests were conducted, testing four indicators under three types of image degradation scenarios: original image, added noise, and added fog. The robustness performance was measured by the attenuation magnitude of the coefficient of determination. As can be seen from Table 2, the existing models experienced a significant decline in indicators due to image degradation, while the performance of the present invention showed the least attenuation, verifying that POI anchoring and visual enhancement can effectively improve prediction stability.
[0152] This research addresses the critical challenge of insufficient predictive stability in socioeconomic forecasting using satellite imagery, a problem stemming from the inconsistent quality of visual data in real-world scenarios. To resolve this, we propose an innovative framework that systematically builds model robustness across multiple levels. Future development can expand this invention to incorporate data from more modalities, such as street view imagery, road networks, and building outlines, to enrich the dimensions and meaning of urban regional representation.
[0153] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. A socioeconomic forecasting method based on satellite imagery with POI anchoring and visual enhancement, characterized in that, Includes the following steps: S1: Multimodal data collection and preprocessing: First, define the urban area set, collect satellite image data, POI data, and label socioeconomic indicators; S2: Construct a semantic stability branch and use POI data to guide a multimodal large model to generate text. S3: Constructing the visual stability branch: Satellite-customized data augmentation: for original images Perform randomized data augmentation operations; Multi-scale feature extraction: ResNet-18 was used as the backbone network to process the original images separately. The enhanced view extracts the feature maps from the output of the four residual blocks, forming a bottom-up path; Feature pyramid fusion: Constructing top-down paths and lateral connections; Global feature aggregation: Perform global average pooling on the feature maps of each layer of the pyramid, and concatenate the results to obtain a unified image vector representation; Projection mapping: Mapping the original image features and the enhanced image features separately using two independent projection heads; S4: Model pre-training to construct a composite loss function Jointly optimize text-image alignment and image-image consistency; S5: Fine-tuning of downstream socioeconomic forecasting tasks: Supervised fine-tuning of the parameters of the MLP predictor is performed only using the gradient descent algorithm.
2. The method for socioeconomic forecasting based on satellite imagery with POI anchoring and visual enhancement as described in claim 1, characterized in that, The urban area set in S1 is defined as follows: For each urban area ; The method for acquiring the satellite image data is as follows: acquiring regional... The corresponding satellite image is represented as ,in and These represent the height and width of the image, respectively. The method for obtaining the POI data is as follows: Obtain the region Collect objective points of interest data within the area, count the names of each POI category and their frequency within the area, and construct a structured POI list dictionary. ; The method for obtaining the labeling of socioeconomic indicators is as follows: obtaining the regional... The corresponding real socioeconomic statistical survey data are logarithmically normalized and smoothed using the following formula: After processing As a supervisory label during the fine-tuning phase.
3. The method for socioeconomic forecasting based on satellite imagery with POI anchoring and visual enhancement as described in claim 1, characterized in that, The specific method of S2 includes: Multimodal suggestion generation: Combine satellite images acquired in S1 with the statistically analyzed list of POIs. Simultaneously used as input; Text description generation: Using the list of POIs as context anchors, text descriptions of the regions are generated using LMM. The process can be formalized as follows: in, It provides factual information that is decoupled from visual input; Text feature extraction: The generated text is processed using a pre-trained text encoder, BERT. The text is encoded using the BERT encoder to obtain the original text representation vector. and through the projection head Mapping to the feature space yields the final text embedding vector. : During this process, the parameters of the BERT text encoder are kept frozen to take advantage of its general language understanding capabilities.
4. The method for socioeconomic forecasting based on satellite imagery with POI anchoring and visual enhancement as described in claim 1, characterized in that, The specific steps for constructing the visual stability branch in S3 include: S3.1: Satellite-customized data augmentation of the original image Perform the following data augmentation operations sequentially or in random combinations: random saturation adjustment, Gaussian noise injection, random erasure, and random rotation, generating two different augmented views, denoted as follows: and ; S3.2: Multi-scale feature extraction: ResNet-18 is used as the backbone network to process the original images separately. First Enhanced View Second Enhanced View ; For any input image X∈ The feature maps output from its four residual blocks are extracted to form a bottom-up feature pyramid hierarchy, denoted as . ,in: The feature map output by the first residual block has a resolution of 1 / 4 of the original image and contains rich local detail information. The feature map output by the second residual block has a resolution of 1 / 8 of the original image and contains semantic information at a medium scale. The feature map output by the third residual block has a resolution of 1 / 16 of the original image and contains higher-level regional structure information. The feature map output by the fourth residual block has a resolution of 1 / 32 of the original image, but it has the strongest global semantic information. S3.3: Feature Pyramid Fusion: Constructing top-down paths and lateral connections, fusing features at each level. The specific calculation process is as follows: First, for the top-level features... Apply 1×1 convolution, To obtain a unified number of channels : Then, for hierarchical indexes Decrease from 4 to 2, performing upsampling and lateral connections sequentially: in, This represents a bilinear upsampling operation, used to upsample high-level feature maps (...). The size is enlarged to match the lower-level feature map. The dimensions are consistent; finally, the fused feature maps are... A 3×3 convolution is applied to eliminate upsampling aliasing, resulting in the final pyramid feature map. ; ； The final pyramid feature atlas was obtained. ; S3.4: Global Feature Aggregation: Feature maps of each layer of the pyramid , , , Perform global average pooling separately to obtain pooling vectors. These four vectors are then concatenated along the channel dimension to obtain a unified image vector representation. ; Following the same process, the two enhanced views were processed separately. and After processing, the corresponding result is obtained. and ; Where AP represents global average pooling. This indicates splicing along the channel dimension; S3.5: Projection Mapping: After obtaining the original image vector representation Compared with the enhanced image representation and Then, by constructing two independent projection heads: the first projection head Second projection head The vector representation of the original image The first projection head is input and mapped to the shared feature space to obtain the final original image embedding vector. : Representing the two enhanced views as vectors and The input to the second projection head is mapped to the same shared feature space to obtain the final enhanced image embedding vector. and : 。 5. The method for socioeconomic forecasting based on satellite imagery with POI anchoring and visual enhancement according to claim 1, characterized in that, The specific steps for constructing the composite loss function and performing joint optimization in S4 include: Constructing a composite loss function for model pre-training This function is the image-text contrast loss. Image-to-image contrast loss Weighted sum: Composite loss function in and These are weight hyperparameters; This represents the contrast loss between the text representation and the original image; This indicates the contrast loss between two enhanced views; The method for calculating the loss between image and text comparison is as follows: Set the batch size for training to N, meaning each batch contains N samples from different regions; For the i-th sample in the batch, i = 1, 2, …, N, its original image embedding vector is denoted as . Its text embedding vector is denoted as ; First, calculate the one-way contrast loss from the image to the text direction. : Then, the one-way contrast loss from the text to the image direction is calculated. : Finally, the loss in comparison between text and images. Take the average of the two unidirectional losses mentioned above: Where exp is an abbreviation for the natural exponential function. Represents cosine similarity. For temperature parameters; The calculation process for image-to-image contrast loss is as follows: For the two augmented view embedding vectors of the i-th sample in the batch, denoted as follows: and ; First, calculate the unidirectional contrast loss from the first augmented view to the second augmented view: Then, the unidirectional contrast loss in the direction from the second augmented view to the first augmented view is calculated: Finally, image-image contrast loss Take the average of the two unidirectional losses mentioned above: in, The cosine similarity function is represented, tau is the preset temperature parameter, the first term in the denominator sums all N second augmented view samples in the batch, and the second term sums all first augmented view samples in the batch except for its own index i. During model pre-training, the composite loss function is minimized. We jointly optimize the alignment of text and images and the consistency between images.
6. The method for socioeconomic forecasting based on satellite imagery with POI anchoring and visual enhancement as described in claim 1, characterized in that, The specific method of S5 is as follows: after pre-training convergence, the visual encoder parameters trained in S3 are retained and their network parameters are solidified. For the target urban area to be predicted, its remote sensing image is input to the frozen visual encoder to extract a high-dimensional feature vector that is robust to environmental disturbances and contains multi-scale information. A lightweight multilayer perceptron is connected to the back end of the visual encoder as a regression predictor, and the mean squared error loss is calculated using the specific task labels collected in S1.