A method for obtaining urban development level based on neural network and night light remote sensing technology

Abstract

The present invention discloses a method for obtaining the level of urban development based on neural network and night light remote sensing technology. In addition to geographical factors, this method comprehensively considers indicators of various factors of urban development, and collects 12 types of data such as network index, distance from super cities, and Forbes business ranking score. In this method, the BP neural network with repeated iterations and self-learning ability builds a model that is more universal and scientific. In this method, the scrapy crawler technology can be used to quickly and extensively crawl data, effectively making up for the lag of statistical data. In addition to using NPP / VIIRS data, night light remote sensing data maps of Landsat8 images, Luojia-1 images, and Gaofen-1 images are also collected. The images are processed by ARCMAP and other software to obtain the total amount of night lights (NTL) in the city, avoiding the problem of a single data source in the remote sensing field, and having higher fault tolerance.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method for obtaining a city development level based on a neural network and night light remote sensing technology. Background Art

[0002] For the collection and analysis of urban characteristic indicators, most existing technical methods tend to focus on the collection of ground information, such as population, urban area, etc. For example, in Wu Fengmin et al. Analysis of urban spatial development and change in Chongqing based on NPP / VIIRS data [J]. Geospatial Information, 2021, 19(05): 45-50, the types of statistical data collected include geographic location information such as urban area, population, and NPP / VIIRS fitting data, but lack reference indicators of corresponding political, economic, social, and commercial factors of the city, and the collected indicators are not comprehensive enough. In addition, when analyzing the built-up area of ​​Chongqing, a simple threshold method is used for extraction, which lacks a corresponding training process. In Xu Lihua. Study on the spatiotemporal evolution of urban expansion in the Yangtze River Delta urban agglomeration based on NPP-VIIRS data [D]. Shanghai Normal University, 2019., the reference data source is NPP / VIIRS data, which involves a single data source in the field of remote sensing. Summary of the invention

[0003] In view of the problems existing in the existing methods, the present invention introduces a method for obtaining the level of urban development based on neural network and night light remote sensing technology.

[0004] The present invention adopts the following technical solution:

[0005] A method for obtaining the level of urban development based on neural network and night light remote sensing technology, comprising the following steps:

[0006] Step 1: Use the Scrapy crawler framework to obtain the characteristic indicators of the city to be evaluated and other cities with Forbes Business City Ranking scores, obtain the NPP-VIIRS night light data of the aforementioned cities through night light remote sensing images, and generate Excel tables;

[0007] Step 2: Preprocess the characteristic indicators and night light data obtained in step 1 through Python program;

[0008] Step 3: Using the characteristic indicators and night light data of the cities with Forbes Business City Ranking scores processed by the program in step 2, the BP neural network model is trained;

[0009] Step 4: Input the characteristic indicators of the city to be evaluated into the trained model in step 3 to obtain the Forbes Business City Ranking score of the city to be evaluated.

[0010] Preferably, in step 1, the Scrapy crawler framework is used to crawl various characteristic indicator data of the city, specifically including:

[0011] Step 11: Enter the command Scrapy Startproject project_name to create a scrapy project and generate various py files required by the Scrapy crawler in the project file directory, including items.py, piplines.py, settings.py, etc.;

[0012] Step 12: Enter the command scrapy genspider spider_name http: / / example / com to generate the crawler file;

[0013] Step 13: Run the scrapy shell http: / / example.com command in the terminal command window; this command will get the Response object and Request object corresponding to the website and enter the Python execution environment;

[0014] Step 14: Based on step 113, the Response object obtained can be used to parse and extract page data, and the fetch method can be used to obtain the Response and Request objects of the new page;

[0015] Step 15: Save the extracted data locally, delete unnecessary content, and clean the data; summarize the collected data of various characteristic indicators of the city into an Excel table for subsequent analysis;

[0016] Preferably, in step 1, obtaining city night light data using night light remote sensing images specifically includes:

[0017] Step 16: Use Landsat8, Gaofen-1, NPP / VIIRS night light data, Luojia-1 night light data and relevant national statistical yearbook data as data sources to collect the total night light (NTL) index;

[0018] Step 17: Remove the effects of stray light, moonlight, and accidental events such as fires by using high-quality data collected by each satellite to perform an annual synthesis;

[0019] Step 18: Repeat step 17 until the collection of indicators for each city is completed.

[0020] The step 2 comprises:

[0021] Step 21: normalize the numerical data in each characteristic indicator data obtained;

[0022] Step 22: Extract the string indicator data from each feature indicator data by calling the pd.get_dummies() function, convert it into numerical values ​​of 0 and 1 and make it have a size relationship.

[0023] Preferably, the step 3 comprises:

[0024] Step 31: Determine the number of network input layer nodes, hidden layer nodes, and output layer nodes according to the system input and output sequence, give the learning rate and neuron excitation function, and implement BP neural network initialization. Introduce the activation function to increase the nonlinear characteristics of the neural network model.

[0025] Step 32: Construct a BP neural network; use the characteristic indicators and night light data of the cities with Forbes business city ranking scores processed in step 2 as a training set, use the BP neural network to continuously learn the training set data generation process, select the Forbes business city ranking score as the result, repeatedly train, compare the Forbes business city ranking score calculated by the model with the actual Forbes business city ranking score, and select the model with the highest accuracy as the final city characteristic indicator analysis model.

[0026] The beneficial effects of the present invention are:

[0027] In addition to geographical factors, this method comprehensively considers various factors and indicators of urban development, and collects 12 types of data such as network index, distance from super cities, and Forbes business ranking score. The BP neural network in this method is repeatedly iterated and has self-learning ability, and the model built is more universal and scientific. The scrapy crawler technology used in this method can quickly and extensively crawl data, effectively making up for the lag of statistical data. In addition to using NPP / VIIRS data, night light remote sensing data maps of Landsat8 images, Luojia-1 images, and Gaofen-1 images are also collected. The images are processed by ARCMAP and other software to obtain the total amount of night lights (NTL) in the city, avoiding the problem of a single data source in the remote sensing field and having higher fault tolerance. DETAILED DESCRIPTION

[0028] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0029] The method of the present invention specifically comprises the following steps:

[0030] Step 1: Use the Scrapy crawler framework to obtain the characteristic indicators of 120 cities respectively, and use night light remote sensing images to obtain the NPP-VIIRS night light data of the above 120 cities.

[0031] The specific steps include:

[0032] Step 11: Use the Scrapy crawler framework to crawl various characteristic indicators of 120 cities including Nanjing, Jiangsu, Suzhou, Jiangsu, and Zhenjiang, Jiangsu. In this embodiment, the various characteristic indicators of the city mainly include: whether it is coastal, whether it is along the river, the distance from the super city, the average altitude, the network index, the population, the average annual rainfall, the city area, the Forbes business city ranking score, the Forbes ranking, the 30 cities with the most innovation in 2018 Forbes, and the 30 cities with the highest operating costs in 2017 Forbes. The specific method includes:

[0033] Step 111: Build the Scrapy framework: Enter the command Scrapy Startproject project_name to create a scrapy project and generate various py files required by the Scrapy crawler in the project file directory, including items.py, piplines.py, settings.py, etc.

[0034] Step 112: Enter the command scrapy genspider spider_name http: / / example / com to generate a crawler file. Here, http: / / example / com is the specific website for the required data. In this solution, the main website for obtaining data is:

[0035] https: / / www.forbeschina.com / lists / 1163, https: / / www.gscloud.cn / sources / index?pid=263, https: / / www.swguancha.com / home / ranking-list?code=gdp&tabName=city, etc.

[0036] Step 113: Run the scrapy shell http: / / example.com command in the terminal command window. This command will get the Response object and Request object corresponding to the website and enter the Python execution environment.

[0037] Step 114: Based on step 113, the Response object obtained is used to parse and extract the page data, and the fetch method is used to obtain the Response and Request objects of the new page.

[0038] Step 115: Save the extracted data locally. Summarize the collected urban raw data information into an Excel table for subsequent analysis.

[0039] Step 12: Use night light remote sensing images to obtain NPP-VIIRS night light data for the city

[0040] Step 121: Use Landsat8, Gaofen-1, NPP-VIIRS night light data, Luojia-1 night light data and relevant national statistical yearbook data as data sources.

[0041] Step 122: Remove the effects of stray light, moonlight, and accidental events such as fires by performing an annual synthesis using high-quality data collected by each satellite.

[0042] Step 123: Repeat step 122 until the collection of the total night light (NTL) index of each city is completed, see the NTL provincial proportion in Table 1. The NTL provincial proportion is the specific value obtained after the normalization of the total night light (NTL) index of the city, and the two refer to the same type of data.

[0043] In this embodiment, the characteristic index data of Nanjing, Suzhou and Zhenjiang obtained by the method described in step 1 above are shown in Table 1 below:

[0044] Table 1

[0045]

[0046] Note: The data not filled in the chart are missing data. Since the neural network has a certain fault tolerance, it has little impact on the calculation.

[0047] Step 2: Process the data through Python program, which includes the following steps:

[0048] Step 21: Normalize the obtained numerical data. The numerical data in this solution include: distance from super cities, average altitude, network index, population, average annual rainfall, urban area, Forbes business city ranking score, Forbes ranking, 2018 Forbes' 30 most innovative cities, 2017 Forbes' 30 cities with the highest operating costs, and NTL's share of the province, a total of 11 indicators.

[0049] Step 211: Traverse all data under the same type of numerical indicators and find the maximum value X in the indicator data max With the minimum value Xmin .

[0050] Step 212: normalize each numerical indicator data. The normalization formula is as follows:

[0051]

[0052] Where X is the original indicator data, X norm The normalized data are shown in Tables 2 and 3.

[0053] Table 2

[0054]

[0055] Table 3

[0056]

[0057] Note: The data not filled in the chart are missing data. Since the neural network has a certain fault tolerance, it has little impact on the calculation.

[0058] Step 23: Extract the features of the string indicator data in the city data and convert them into numerical indicator data. Extract the features of string data, such as "whether it is coastal", etc., by calling the pd.get_dummies() function, and convert them into numerical values ​​of 0 and 1 so that they have a size relationship. In this embodiment, the character data include: whether it is coastal, whether it is along the river. Through this step, the character data can be converted into numerical data with a relative size relationship, as shown in Table 4.

[0059] Table 4

[0060]

[0061]

[0062] Step 3: Use the various characteristic indicators of the city processed by the procedure in step 2 to train the BP neural network model.

[0063] Specifically include:

[0064] Step 31: Determine the number of network input layer nodes, hidden layer nodes, and output layer nodes according to the system input and output sequence, give the learning rate and neuron excitation function, and initialize the BP neural network. Introduce the Sigmoid activation function to increase the nonlinear characteristics of the neural network model.

[0065] Step 32: The characteristic indicators of the 120 cities collected in step 1 are divided into a training set and a test set at a ratio of 1:1, that is, the characteristic indicators of 60 cities are used as the training set, and the characteristic indicators of the other 60 cities are used as the test set. The BP neural network is used to continuously learn the training set data generation process, and the Forbes business city ranking score is selected as the result. After repeated training, the Forbes business city ranking score calculated by the model is compared with the actual Forbes business city ranking score, and the model with the highest accuracy is selected as the final city characteristic indicator analysis model.

[0066] Step 4: Based on step 3, use the Sklearn algorithm to define a score parameter custom evaluation standard to evaluate the performance of the model; specifically:

[0067] Step 41: After the neural network training is completed, use the Sklearn algorithm to define a score parameter to evaluate the performance of the model.

[0068] Step 42: Verify the accuracy of the neural network model. Determine the coefficient of determination score and apply it to urban data prediction.

[0069] Note: The determination coefficient is an important statistic that reflects the goodness of model fit. It is the ratio of the regression sum of squares to the total sum of squares. It ranges from 0 to 1 and has no unit. Its numerical value reflects the relative degree of regression contribution, that is, the percentage of the total variation of the dependent variable that can be explained by the regression relationship.

[0070] The calculation method of determining the coefficient score is as follows: Calculate the mean μ, standard deviation σ of the overall data of 12 types of data in 60 cities, and the actual observed value x of the 12 types of indicators in each city, then

[0071] In this embodiment, the determination coefficient score is used as a standard to measure the accuracy of the model. The standard deviation of the result calculated by the model and the actual Forbes business city ranking score (the standard deviation of the prediction model is finally obtained by calculating the mean, variance, and average variance). In this embodiment, the score value calculated is 0.72.

[0072] Step 5 includes: using the trained neural network model to evaluate urban development status.

[0073] Step 51: Input the 12 types of data indicators of the 60 city characteristics used as the test set in step 32 into the trained BP neural network model.

[0074] Step 52: The neural network predicts that the Forbes business city ranking scores of Nanjing, Jiangsu, Suzhou, Jiangsu, and Zhenjiang, Jiangsu are 0.5401, 0.5174, and 0.2747, respectively. Compared with the actual Forbes business city ranking scores of the three cities, the actual Forbes business city ranking scores of the three cities are: Nanjing, Jiangsu: 0.5479; Suzhou, Jiangsu: 0.5175; Zhenjiang, Jiangsu: 0.2738. By comparing the existing data, it can be seen that there is a slight error between the calculated results and the actual Forbes business city ranking scores, but the score parameter of the model obtained by combining the above training is 0.72, that is, the accuracy is high and the error is small.

[0075] The Forbes Business City Ranking score predicted by the model can reflect the overall development level of a city. Nanjing's Forbes Business City Ranking score is high, which confirms that Nanjing belongs to the new first-tier city; Suzhou's Forbes Business City Ranking score is high, which confirms that Suzhou belongs to the new first-tier city; and Zhenjiang's Forbes Business City Ranking score is not high enough, which confirms that Zhenjiang belongs to the second-tier city. And by comparing the calculation results of different cities, it can reflect the size of the gap between different cities to a certain extent. Comparing Suzhou and Nanjing, the calculation results of the Forbes Business City Ranking score are not much different, and the actual development levels of the two are similar; while comparing Nanjing and Zhenjiang, it is not difficult to find that the Forbes Business City Ranking score is quite different, which also confirms that in the actual city level classification, Zhenjiang belongs to the second-tier city and Nanjing belongs to the new first-tier city, and there is a certain gap in the development level between the two. In addition, for cities that lack Forbes Business City Ranking scores, this model can also be used to calculate the predicted Forbes Business City Ranking score, so as to make a series of analyses and comparisons on the development status of the city.

[0076] This method uses the Scrapy crawler framework as a collector of general search engine web pages, which is convenient for obtaining more data sources. Compared with traditional methods for studying urban construction, its data collection efficiency is high, timeliness is high, and it does not require a lot of manpower and material resources. Remote sensing imaging technology is used to expand geospatial data. By collecting NPP-VIIRS night light data, the missing spatial information in traditional methods is supplemented with higher accuracy. In addition, when facing large data processing problems, compared with traditional machine learning algorithms, the powerful computing power of neural networks can process more massive data and save more time.

Claims

1. A method for obtaining the level of urban development based on neural network and night light remote sensing technology, characterized in that: The steps include: Step 1: Use the Scrapy crawler framework to obtain the characteristic indicators of the city to be evaluated and other cities with Forbes Business City Ranking scores, obtain the NPP-VIIRS night light data of the aforementioned cities through night light remote sensing images, and generate Excel tables; Step 2: Preprocess the characteristic indicators and night light data obtained in step 1 through Python program; Step 3: Using the characteristic indicators and night light data of the cities with Forbes Business City Ranking scores processed by the program in step 2, the BP neural network model is trained; Step 4: Input the characteristic indicators of the city to be evaluated into the trained model in step 3 to obtain the Forbes Business City Ranking score of the city to be evaluated.

2. The method for obtaining the level of urban development based on neural network and night light remote sensing technology according to claim 1 is characterized in that: In step 1, the Scrapy crawler framework is used to crawl various characteristic indicator data of the city, including: Step 11: Enter the command Scrapy Startproject project_name to create a scrapy project and generate various py files required by the Scrapy crawler in the project file directory, including items.py, piplines.py and settings.py; Step 12: Enter the command scrapy genspider spider_name http: / / example / com to generate the crawler file; Step 13: Run the scrapy shell http: / / example.com command in the terminal command window; this command will get the Response object and Request object corresponding to the website and enter the Python execution environment; Step 14: Based on step 113, use the Response object obtained to parse and extract the page data, and use the fetch method to obtain the Response and Request objects of the new page; Step 15: Save the extracted data locally, delete unnecessary content, and perform data cleaning; summarize the collected city characteristic indicator data into an Excel table.

3. The method for obtaining the level of urban development based on neural network and night light remote sensing technology according to claim 1 is characterized in that: The step 1 uses night light remote sensing images to obtain urban night light data, specifically including: Step 16: Use Landsat8, Gaofen-1, NPP / VIIRS night light data, Luojia-1 night light data and relevant national statistical yearbook data as data sources to collect the total night light (NTL) index; Step 17: Remove the effects of stray light, moonlight, and fire incidents by using high-quality data collected by each satellite to perform an annual synthesis; Step 18: Repeat step 17 until the collection of indicators for each city is completed.

4. The method for obtaining the level of urban development based on neural network and night light remote sensing technology according to claim 3 is characterized in that: The step 2 specifically includes: Step 21: normalize the numerical data in each characteristic indicator data obtained; Step 22: Extract the string indicator data from each feature indicator data by calling the pd.get_dummies() function, convert it into numerical values ​​of 0 and 1 and make it have a size relationship.

Images