Data processing method based on big data urban indicator knowledge graph

Abstract

The embodiment of the present invention discloses a data processing method based on a big data city indicator knowledge graph, including: obtaining the city macro big data of multiple historical years in the target area to generate an original indicator set; selecting the first city indicator with the highest priority from the original indicator set, and determining each second city indicator with the first city indicator as the pre-indicator, thereby generating an available indicator set; extracting multiple indicator combinations in the available indicator set one by one, respectively learning the data relationship between each indicator combination and the first city indicator in the same year, and determining the best indicator combination and the best data model; returning to the operation of selecting the first city indicator until all city indicators in the original indicator set are selected; and constructing a knowledge graph with each city indicator as a node according to the best indicator combination and the best data model corresponding to each city indicator. This embodiment clarifies the complex data relationship between a large number of city indicators in a quantitative form.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data mining technology, and in particular to a data processing method based on a big data city indicator knowledge graph. Background Art

[0002] Urban macro-static data mining refers to the application of data mining methods to urban macro-static data to reveal the characteristics and laws of urban data and provide data support for urban construction and decision-making.

[0003] The current mainstream urban macro-static data mining methods include the following:

[0004] Data visualization: Visualize the city’s macro-static data in the form of charts, maps, etc., so as to intuitively understand and analyze the city’s situation.

[0005] Various statistical analysis methods: Analyze the macro-static data of the city through statistical methods to reveal the correlation and patterns between urban data.

[0006] Various data mining algorithms: Use data mining algorithms to conduct in-depth mining of urban macro-static data to discover patterns, associations and laws hidden in urban data.

[0007] The common problem of these technical means is that the urban data characteristics and data patterns revealed are fragmented and small in structure, involving only a single city indicator or a small number of city indicators, and cannot systematically reflect the overall characteristics and complex relationships of a large number of urban data. For example, only a few factors related to urban infrastructure can be obtained, and when it comes to multiple factors such as energy, traffic, and transportation, the data relationship between a large number of urban indicators cannot be reflected. Summary of the invention

[0008] The embodiment of the present invention provides a data processing method based on a big data city indicator knowledge graph, constructs a city-wide indicator knowledge graph based on the city's macro big data, and clarifies the complex data relationships between a large number of city indicators such as energy, traffic, and transportation in a quantitative form.

[0009] In a first aspect, an embodiment of the present invention provides a data processing method based on a big data city indicator knowledge graph, comprising:

[0010] S110, obtaining urban macro big data of multiple historical years in the target area, and sorting multiple urban indicators in the urban macro big data according to priority to jointly form an original indicator set; wherein the multiple urban indicators include at least one of the following: highway diesel fuel consumption, highway freight turnover, number of traffic accidents, number of fires, total water consumption and sulfur dioxide emissions;

[0011] S120: Select a first city indicator with the highest priority from city indicators that have not been selected in the original indicator set, and determine each second city indicator with the first city indicator as a pre-indicator; and form an available indicator set for calculating the first city indicator from other indicators in the original indicator set except the first city indicator and each second city indicator;

[0012] S130, extracting multiple indicator combinations from the available indicator set one by one, building a sample library based on the city macro big data, learning the data relationship between each indicator combination and the first city indicator in the same year, and determining the best indicator combination and the best data model for calculating the first city indicator according to the learning error; returning to the operation of selecting the first city indicator with the highest priority from the city indicators that have not been selected in the original indicator set, until all the city indicators in the original indicator set are learned and trained;

[0013] S140. According to the optimal indicator combination and the optimal data model corresponding to each city indicator, a knowledge graph with each city indicator as a node is constructed, and the city data of the target area in future years is calculated according to the knowledge graph.

[0014] In a second aspect, an embodiment of the present invention provides an electronic device, including:

[0015] one or more processors;

[0016] a memory for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method based on the big data city indicator knowledge graph described in any embodiment.

[0018] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the data processing method based on the big data city indicator knowledge graph described in any embodiment is implemented.

[0019] Based on the urban macro data of the target area, the embodiment of the present invention systematically extracts the quantitative relationship between a large number of urban indicators such as highway diesel fuel consumption, highway freight turnover, number of traffic accidents, number of fires, total water consumption and sulfur dioxide emissions, and adds the quantitative relationship to the knowledge graph to form an overall or thematic relationship knowledge graph and urban system for energy, traffic, transportation and other topics, which are used to calculate the future urban data of the target area in terms of energy, traffic, transportation and other aspects. In particular, when processing a large amount of urban indicator data, by excluding the second city indicator with the current first city indicator as the pre-indicator, the risk of cyclic call and infinite calculation between a large number of urban indicators is avoided, and the convergence and robustness of the knowledge graph system are guaranteed. At the same time, in the urban data measurement based on the knowledge graph, the overall update of urban macro data is realized step by step by some urban indicators, giving full play to the systematic role of the knowledge graph on urban energy, traffic, transportation and other data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of a data processing method based on a big data city indicator knowledge graph provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of a decision tree regression model for calculating highway freight turnover provided by an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a decision tree regression model for calculating the number of traffic accidents provided by an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the distribution of decision tree regression models in the random forest regression model for calculating coke consumption provided by an embodiment of the present invention, wherein Figure 4 (a) is a schematic diagram of the depth distribution of each decision tree regression model. Figure 4 (b) is a schematic diagram of the distribution of leaf node numbers of each decision tree regression model;

[0025] Figure 5 yes Figure 4 A diagram of a decision tree regression model in the random forest regression model of ;

[0026] Figure 6is a schematic diagram of a representative decision tree model in a random forest regression model for calculating crude oil consumption provided by an embodiment of the present invention;

[0027] Figure 7 It is a partial schematic diagram of a knowledge graph under a graph structure provided by an embodiment of the present invention;

[0028] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0030] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0031] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] Figure 1 1 is a flow chart of a data processing method based on a big data city indicator knowledge graph provided by an embodiment of the present invention. The method is applicable to data mining of a large amount of city indicator data and is executed by an electronic device. Figure 1 As shown, the method specifically includes:

[0033] S110, obtaining urban macro big data of multiple historical years in the target area, and sorting multiple urban indicators in the urban macro big data according to priority to jointly form an original indicator set.

[0034] The target area is the area to be studied, which may be a city, a province or a larger area. This embodiment processes a large amount of urban indicator data of the target area, quantifies the data rules therein, and calculates the future urban data of the target area according to the quantified rules.

[0035] Urban macro big data refers to static statistical data of the target area, involving a large number of urban indicators, and may include at least one of the following categories of urban indicators: energy (such as diesel consumption, electricity consumption, nitrogen oxide emissions, etc.), transportation (such as highway freight turnover, postal business volume, express delivery volume, etc.), traffic (such as the number of traffic accidents, the number of public transportation vehicles, etc.), population (such as population size, life expectancy, etc.). It should be noted that in actual applications, any available urban indicator data can be used for calculation, and this embodiment does not limit the specific name of the urban indicator. Optionally, the static macro data of the target area (such as a city yearbook) can be downloaded from the Internet, and the data values ​​of each urban indicator can be crawled and extracted from it as the data source of the entire method.

[0036] In practical applications, the available city indicators usually include hundreds or even thousands. In this embodiment, these city indicators are sorted according to priority, and all the sorted city indicators together constitute the original ordered available indicator set, referred to as the original indicator set. It is worth mentioning that the elements in this set are only indicator names, and no specific data needs to be stored.

[0037] S120. Select a first city indicator with the highest priority from city indicators that have not been selected in the original indicator set, and determine each second city indicator with the first city indicator as a pre-indicator; and use other indicators in the original indicator set except the first city indicator and each second city indicator to form an available indicator set for calculating the first city indicator.

[0038] The priority here refers to the priority of the city indicator, which can be pre-set as needed. In this step, a single city indicator is selected in turn according to the priority as the object of subsequent learning of data rules. In the subsequent operation, it will be further clarified which city indicators can be used to accurately calculate the city indicator. If these city indicators are calculated based on the city indicator with the highest priority, there will be a situation where indicator d is calculated through indicators a, b and c, but indicator a is calculated through indicators d and e. At this time, the city indicators call each other, which will cause the data processing to loop infinitely. This embodiment can avoid this situation. Specifically, the city indicator with the highest priority selected each time is called the first city indicator, and the city indicator with the first city indicator as the pre-indicator is the second city indicator. This step excludes the first city indicator and the second city indicator from the original indicator set, and the remaining city indicators together constitute an available indicator set that can be used as a pre-variable of the first city indicator, thereby avoiding mutual calls between the first city indicator and the second city indicator.

[0039] S130, extracting multiple indicator combinations in the available indicator set one by one, building a sample library based on the city macro big data, learning the data relationship between each indicator combination and the first city indicator in the same year, and determining the best indicator combination and the best data model for calculating the first city indicator according to the learning error; returning to the operation of selecting the first city indicator with the highest priority from the city indicators that have not been selected in the original indicator set, until all the city indicators in the original indicator set are learned and trained.

[0040] The data relationship here is reflected by a variety of data models, including: linear regression model, ridge regression model, regression model, elastic network regression model, decision tree regression model, random forest regression model, LightGBM regression model. In this step, for each first-city indicator, the indicator combination that can most accurately calculate the current first-city indicator data is selected as the best indicator combination. The data model under this combination is the best data model, which realizes the quantitative representation of the data rules between city indicators.

[0041] Table 1 takes Hebei Province as an example, showing some city indicators and their corresponding optimal indicator combinations and optimal data models. Among them, the linear regression model can be expressed as a specific linear formula, while the specific structures and parameters of the decision tree regression model and the random forest regression model are relatively complex and cannot be expressed as a formula. Figures 2 to 6The representative decision trees in each decision tree regression model and each random forest regression model are visualized (the specific meaning of each parameter in the figure will be described in detail in the subsequent embodiments). In practical applications, when the method of this embodiment is implemented in a Python environment, the optimal data model of each city indicator can be saved separately through a PKL file. The PKL file is a Python serialized object file that can save and load the structure, parameters and other information of the model. After obtaining the optimal indicator combination data of a certain city indicator, the corresponding PKL file is loaded through Python to realize the quantitative calculation between the city indicator data. Below, taking the city indicator "total water consumption" in the fourth row of Table 1 as an example, the meaning of each column in Table 1 is explained in turn. "Total water consumption" belongs to the resource category, and the data unit is 100 million cubic meters. The optimal indicator combination (i.e., the pre-indicator for calculating the total water consumption) clarified by S130 includes sulfur dioxide emissions, and the corresponding optimal data model (i.e., the data relationship between the total water consumption and sulfur dioxide emissions) is a linear regression model, in which y represents the total water consumption and x represents the sulfur dioxide emissions. In addition, the query data in rows 14 and 15 in the table have a specific format of (year, data). For example, the first item of data in row 14 (2000, 6.888) means that the number of public transportation vehicles per 10,000 people in 2000 was 6.888 standard units.

[0042] Table 1

[0043]

[0044] The following further describes the various cycles of S120-S130 in conjunction with Table 1:

[0045] In a specific embodiment, the multiple city indicators in S110 include: highway diesel fuel consumption, highway freight turnover, number of traffic accidents, number of fires, total water consumption, sulfur dioxide emissions and highway diesel carbon dioxide emissions, among which the priority order of some city indicators is: highway diesel fuel consumption, highway freight turnover, number of traffic accidents and total water consumption.

[0046] For some of the above-mentioned city indicators, in S120, the highway diesel fuel consumption with the highest priority is selected as the first city indicator, and the second city indicator with highway diesel fuel consumption as the pre-indicator is determined, including highway diesel carbon dioxide emissions; highway diesel fuel consumption and the second city indicator including highway diesel carbon dioxide emissions are excluded from the original indicator set, and the available indicator set for calculating highway diesel fuel consumption is composed of other city indicators. It should be noted that the "exclusion" here only means exclusion from the consideration range, and it is not really deleted from the original indicator set. Then execute S130, and the best indicator combination for calculating highway diesel fuel consumption finally obtained includes highway freight turnover, and the best data model is a linear regression model: , where y represents highway diesel fuel consumption, x represents highway freight turnover, and k can be obtained according to the following formula: , where N is the number of years and i is the year index. represents the total diesel consumption in year i, It represents the freight kilometers in the i-th year. The constant k at this time can be simply referred to as the average diesel fuel consumption.

[0047] Then return to S120, select the highway freight turnover with the second highest priority as the first city indicator, and determine the second city indicator with highway freight turnover as the pre-indicator, including highway diesel fuel consumption; exclude highway freight turnover and the second city indicator including highway diesel fuel consumption from the original indicator set, and use other city indicators to form an available indicator set for calculating highway freight turnover. Then execute S130, and the best indicator combination for calculating highway freight turnover finally obtained includes the number of traffic accidents, and the best data model is a decision tree regression model. Exemplarily, the model structure is as follows Figure 2 As shown in the figure, in this model, x represents the number of traffic accidents, and the number of traffic accidents and highway freight turnover in the same year in each historical year constitute a sample. The condition on each node (for example, x≤5622) is the condition for the decision tree to split at the node. When the number of traffic accidents is less than or equal to 5622, the sample will move to the left child node; otherwise, it will move to the right child node. Sam in each node represents the number of samples that reach the node during the training phase (that is, the number of samples that satisfy x≤5622); y represents the average value of highway freight turnover in all samples that reach the node during the training phase, and is also the predicted value of highway freight turnover in the prediction phase; sq represents the mean square error, which represents the variance of highway freight turnover in all samples that reach the node during the training phase. The smaller the mean square error, the closer the highway freight turnover of the sample at the node is to its mean, and the better the prediction effect of the node in the prediction phase.

[0048] Then return to S120 to select the number of traffic accidents with the third priority as the first city indicator, and determine the second city indicators with the number of traffic accidents as the leading indicator, including highway freight turnover and highway diesel fuel consumption; exclude the number of traffic accidents and the second city indicators including highway freight turnover and highway diesel fuel consumption from the original indicator set, and the other city indicators constitute the available indicator set for calculating the number of traffic accidents. Then execute S130, and the best indicator combination for calculating the number of traffic accidents finally obtained includes the number of fires, and the best data model is the decision tree regression model. Exemplarily, the model structure is as follows Figure 3As shown in the figure, x represents the number of fires, and the number of fires and traffic accidents in the same year in each historical year constitutes a sample. The condition on each node (for example, x≤7334) is the condition for the decision tree to split at the node. When the number of fires is less than or equal to 7334, the sample will move to the left child node; otherwise, it will move to the right child node. The sam in each node represents the number of samples that reach the node during the training phase (that is, the number of samples that satisfy x≤7334); y represents the average number of traffic accidents in all samples that reach the node during the training phase, and is also the predicted value of the number of traffic accidents in the prediction phase; sq represents the mean square error, which represents the mean variance of the number of traffic accidents in all samples that reach the node during the training phase. The smaller the mean square error, the closer the number of traffic accidents in the samples at the node is to its mean, and the better the prediction effect of the node in the prediction phase. It should be noted that, for the sake of ease of expression, the independent variables of each data model in this embodiment are represented by x, z, x1, x2, etc., the dependent variables are represented by y, and the sample size (or proportion) and mean square error in each decision tree model are represented by sam and sq. However, the city indicators or sample definitions corresponding to the symbols of each variable are different in different models and should be treated differently in actual applications.

[0049] Finally, return to S120, select the total water consumption with the fourth priority as the first city indicator, and determine the second city indicator with the total water consumption as the pre-indicator. Exemplarily, at this time, the second city indicator is empty, then it is only necessary to exclude the total water consumption from the original indicator set, and the available indicator set for calculating the total water consumption is composed of other city indicators. Then execute S130, and the best indicator combination for calculating the total water consumption finally obtained includes sulfur dioxide emissions, and the best data model is a linear regression model. Exemplarily, the expression of the model is: , where y represents the total water consumption and x represents the sulfur dioxide emissions.

[0050] In another specific implementation, the multiple city indicators may also include traffic and transportation carbon emissions, highway diesel carbon dioxide emissions, water transport carbon dioxide emissions, railway carbon dioxide emissions, coke consumption, crude oil consumption, year-end permanent population, etc. After the data is processed in a cycle of S120-S130, the optimal indicator combination and optimal data model of each city indicator are finally obtained as follows:

[0051] The best indicator combination for traffic and transportation carbon emissions includes road diesel CO2 emissions, water transport CO2 emissions, and railway CO2 emissions, and the best data model is the linear regression model: , where y represents traffic and transportation carbon emissions, x1 represents road diesel carbon dioxide emissions, x2 represents water transport carbon dioxide emissions, and x3 represents railway carbon dioxide emissions.

[0052] The best indicator combination for coke consumption includes the year-end permanent population, and the best data model is the random forest regression model. The random forest regression model includes multiple decision tree regression models. The depth and node distribution of each decision tree regression model are shown in Figure 2. Figure 4 As shown, Figure 4 (a) shows the depth distribution of each decision tree regression model. The ordinate represents the number of decision tree regression models with a certain depth. Figure 4 (b) shows the distribution of the number of leaf nodes of each decision tree regression model, and the vertical axis represents the number of decision tree regression models with a certain number of leaf nodes. Figure 5 The structure of one of the decision tree models is given as an example, where x represents the year-end resident population, and the year-end resident population and coke consumption in each historical year constitute a sample. The meaning of each node is the same as Figure 2 and Figure 3 Similar, no further description is given here.

[0053] The best indicator combination for crude oil consumption includes the year-end permanent population, and the best data model is the random forest regression model. The random forest regression model includes multiple decision tree regression models. Figure 6 The structure of a representative decision tree model is given, where x represents the year-end resident population, the year-end resident population and crude oil consumption of the same year in each historical year constitute a sample, sam represents the proportion of the number of samples reaching this node in the training phase to the total number of samples, and the meanings of the remaining parameters and nodes are the same as Figure 2 and Figure 3 Similar, no further description is given here. It should be noted that Figure 2 , 3 , 5, 6 and the specific expressions of the above linear regression models are all specific implementations of the present invention and are not applicable to all target areas or all years. The data model in actual application may vary with the target area and the urban macro big data of historical years, and this embodiment does not make specific restrictions.

[0054] S140. According to the optimal indicator combination and the optimal data model corresponding to each city indicator, a knowledge graph with each city indicator as a node is constructed, and the city data of the target area in future years is calculated according to the knowledge graph.

[0055] This knowledge graph can cover all city indicators in the city macro big data, display the data relationship between all city indicators, and quantify the data relationship through the data model and model file shown in Table 1. Using this knowledge graph, we can not only intuitively obtain the elements related to each city indicator, but also reflect the impact path between each city indicator through the preceding and following relationships, and use the clear quantitative relationship to calculate the city data of the target area in any future year or in the future year by year, so as to systematically reflect the overall characteristics of the city big data.

[0056] Based on the above inventive concept, the implementation process of S120 - S140 is described in detail below through specific implementation methods.

[0057] In a specific embodiment, the first city indicator selected in each loop iteration in S120 and its corresponding optimal indicator combination are stored in the relationship tree, exemplarily, in the relationship tree dictionary, wherein the selected city indicator is used as the key, and the corresponding optimal indicator combination is used as the value, constituting a plurality of key-value pairs in the dictionary; the second city indicator generated in each loop iteration is stored in the second indicator set, exemplarily, in the list related_keys[], and each first city indicator corresponds to its own second indicator set. In the first loop, the relationship tree dictionary and the second indicator set are both empty. In subsequent loops, the data in the relationship tree dictionary will not be cleared, and the defined data model will be accumulated and stored. In the loop for the same first city indicator, the second indicator set will not be cleared, and the defined second city indicator will be accumulated and stored; after entering the loop for the next first city indicator, the second indicator set will be cleared, and the second city indicator corresponding to the next first city indicator will be stored again.

[0058] Under the above data structure, optionally, the step of determining each second city index with the first city index as a pre-indicator includes the following steps:

[0059] Step 1: Identify the best indicator combinations including the current first city indicator in the relationship tree, and use the first city indicators corresponding to the identified best indicator combinations in the relationship tree as the second city indicators. Specifically, the relationship tree dictionary can be traversed to compare each value in the relationship tree dictionary with the current first city indicator. If the current first city indicator exists in the best indicator combination corresponding to the current value, indicating that the first city indicator (key) corresponding to the current value in the relationship tree dictionary uses the current first city indicator as a pre-indicator, then the key is used as a second city indicator and stored in the second indicator set related_keys[].

[0060] Step 2: Take each second city index as the new current first city index, re-identify each new best index combination including the new current first city index in the relationship tree, and take the first city index corresponding to each new best index combination in the relationship tree as each new second city index; take each new second city index as the updated current first city index, return to the operation of identifying the best index combination in the relationship tree, and iterate continuously until the updated second city index cannot be obtained. This step determines the multi-level second city index with the current first city index as the pre-indicator in turn through multiple layers of loop iteration. Specifically, there are many pre-indicator methods, including direct pre-indicator, pre-pre ... Exemplarily, in the above-mentioned cycle with the number of traffic accidents as the first city indicator, the second city indicators determined include highway freight turnover and highway diesel consumption. The specific determination process may include: identifying the best indicator combination including the number of traffic accidents in the relationship tree, and using the identified best indicator combination in the relationship tree corresponding to the first city indicator highway freight turnover as a second city indicator, that is, the first-level second city indicator; then using the highway freight turnover as the new current first city indicator, re-identifying a new best indicator combination including the highway freight turnover in the relationship tree, and using the identified new best indicator combination in the relationship tree corresponding to the first city indicator highway freight turnover as another second city indicator, that is, the second-level second city indicator; and repeating this cycle until no new second city indicator can be generated.

[0061] Optionally, in order to realize the step-by-step processing of the second city indicator, new list data structures current_keys[] and new_keys[] are introduced, wherein current_keys[] is used to store the current level second city indicator being processed, and new_keys[] is used to store the next level second city indicator to be processed. Based on these lists, firstly, the first level second city indicator determined in step 1 is added to new_keys[] for initialization, and then the following loop is performed when new_keys[] is not empty: assign new_keys[] as the current level second city indicator being processed to current_keys[]; clear new_keys[] to store the next level second city indicator identified subsequently; extract each city indicator in current_keys[] as the new current first city indicator, traverse the relationship tree dictionary again, and compare each value in the relationship tree dictionary with the new current first city indicator. If the new current first city indicator exists in the best indicator combination corresponding to the current value, it indicates that the first city indicator (current key) corresponding to the current value in the relationship tree dictionary has the new current first city indicator as the preceding indicator. At this time, since the relationship tree dictionary has been traversed many times, the key may have been identified as the second city indicator in the previous traversal. In order to avoid repeated identification, the key can be compared with the current second indicator set related_keys[]. If the key already exists in related_keys[], the city indicator recorded by the key is abandoned as the second city indicator again; if the key does not exist in related_keys[], the city indicator recorded by the key is used as a new second city indicator and added to the second indicator set related_keys[] and the next level second city indicator list new_keys[]. This cycle is repeated until the next level second city indicator list new_keys[] to be processed is empty, and the identification of all levels of second city indicators is completed and stored in the second indicator set related_keys[] without duplication. The following core code in the python3.9 environment shows the above loop logic more clearly:

[0062] def find_second_features(target_col, over_namebase):

[0063] / / target_col represents the current first city index, over_namebase represents the relationship tree dictionary

[0064] related_keys = set() / / Define the second indicator set related_keys[]

[0065] new_keys = set() / / new_keys[] is used to store the next level second city indicators to be processed

[0066] / / The following for loop corresponds to step one

[0067] for key, value in over_namebase.items(): / / Traverse the relationship tree dictionary

[0068] if set(value) & set(target_col): / / If the current first city indicator exists in value (best indicator combination)

[0069] related_keys.add(key)

[0070] / / Store the key corresponding to value as the second city indicator in the second indicator set related_keys[]

[0071] new_keys.add(key) / / Save to the next level second city indicator list new_keys[] to be processed at the same time

[0072] / / The following while loop corresponds to step 2:

[0073] while new_keys:

[0074] / / The new_keys[] list is not empty (there are still next-level second-city indicators to be processed), and the following loop is executed:

[0075] current_keys = new_keys

[0076] / / Store new_keys[] as the current level second city index being processed in current_keys[]

[0077] new_keys = set() / / Clear new_keys[] to store the next level second city index identified later

[0078] for related_key, related_value in over_namebase.items(): / / Traverse the relationship tree dictionary again

[0079] if related_key not in related_keys and set(related_value) & set(current_keys):

[0080] / / If there is a new current first city indicator in the best indicator combination corresponding to value, and the corresponding key

[0081] No duplication with the already identified secondary city indicators

[0082] related_keys.add(related_key)

[0083] / / Store the key as the new second city indicator in the second indicator set related_keys[]

[0084] new_keys.add(related_key)

[0085] / / Store the key as the next level second city indicator in the list new_keys[]

[0086] After determining the second city indicators at all levels in the above manner, the current first city indicator and all second city indicators in related_keys[] are removed from the original city indicator set to obtain an available city indicator set that can be used to calculate the current first city indicator. In a specific implementation, determining the best indicator combination and the best data model for the first city indicator based on the available city indicator set in S130 may include the following steps:

[0087] Step A: In the available indicator set, a plurality of indicator combinations are randomly selected in units of a set number of indicators. For example, the set number of indicators may be 3, and K indicators are randomly selected from the available indicator set each time to obtain a plurality of indicator combinations without duplication and omission of indicators (ensuring that all combinations are comprehensive), which are used to further screen the leading indicators of the current first city indicator.

[0088] Step B: extract the data of each indicator combination of multiple years from the urban macro big data as input samples, and train the preset multiple data models respectively, so that the output of each model is constantly close to the data of the first urban indicator in the same year. This step extracts the data of the same year from the urban macro big data for each indicator combination to construct a sample, and divides the sample set into a training set and a validation set, traverses the multiple data models one by one, and inputs the data of the current indicator combination in multiple samples into the current data model, so that the output of the model is constantly close to the data of the first urban indicator in the same sample. That is, the trained model takes the current indicator combination data of a specific year in the target area as input, and takes the first urban indicator data of the same year in the target area as output.

[0089] Step C, after the training is completed, select the indicator combination and data model with the smallest error and meeting the threshold conditions as the optimal indicator combination and the optimal data model for calculating the first city indicator. Optionally, the model performance is verified by calculating the correlation index and the mean square error. The minimum correlation index and the maximum mean square error can be set as thresholds. If the threshold conditions are not satisfied, the model is discarded. It should be noted that the above steps B and C can be performed separately, that is, a unified comparison is performed after the training of all models is completed; they can also be performed alternately, that is, after each model is trained, the model is immediately checked to see if it meets the threshold conditions. If it does not meet the conditions, it will be discarded. If it meets the conditions, it will continue to be compared with the previous model that meets the threshold conditions, and the model with the smaller error will be discarded.

[0090] Furthermore, after the training is completed, if each data model cannot meet the threshold condition, the optimal indicator combination and the optimal data model cannot be determined. At this time, the value of the set indicator number K can be reduced, and the indicator combination can be reselected back to step A until a model that meets the threshold condition is selected. Specifically, reducing the number of pre-indicators is equivalent to reducing the complexity of the data relationship between urban indicators, which is more conducive to model convergence and can improve the success rate of data mining. In addition, there is another situation: when the set indicator number is reduced to 1, each data model still cannot meet the threshold condition. This shows that the data relationship between the first urban indicator and other urban indicators is indeed difficult to quantify. The optimal indicator combination of the first urban indicator can be set to empty, and the optimal data model can be set to LOOKUP data, referred to as query data, and data information is obtained by direct query or direct assignment. Exemplarily, "public transportation vehicles per 10,000 people" and "number of beds in medical and health institutions" in Table 1 belong to query data.

[0091] In a specific implementation, the city indicators selected according to priority in S130 and their corresponding optimal data models are stored in the data model dictionary, and the selected city indicators are used as keys (key), and the corresponding optimal indicator combinations are used as values ​​(value), forming multiple key-value pairs in the dictionary. Correspondingly, in S140, the values ​​in the relationship tree dictionary and the data model dictionary are externally connected through the same key to generate a knowledge graph centered on each city indicator. Optionally, the knowledge graph may include multiple presentation forms, which may be presented as a table structure (such as Table 1), with each city indicator being an index of each row in the table; it may also be presented as a graph structure (such as Figure 7 ): Take each city indicator as a node of the knowledge graph, and perform the following operations on each node: S1. Extract the pre-indicators of the current city indicator from the best indicator combination corresponding to the current node; S2. Add edges between each pre-indicator node and the current node, starting from each pre-indicator node and ending at the current node; S3. Mark the best data model corresponding to the current city indicator on each edge, and store the best data model in the model file library for the knowledge graph to call (the best data model and its model file are in Figure 7 In comparison, the knowledge graph with a table structure can clearly display the best city indicator combination and the best data model for each city indicator, which is conducive to directly extracting the calculation basis of each indicator; the knowledge graph with a graph structure can show the complexity of the correlation between each city indicator, which is conducive to showing the overall characteristics and complex relationships of the data. In practical applications, the appropriate presentation form can be selected according to specific needs, and this embodiment does not limit it. In addition, the knowledge graph can also add any required attributes for each city indicator, such as city indicator priority, weight, etc., to further improve the knowledge content.

[0092] Compared with the prior art, the method of this embodiment can quickly produce the knowledge graph of the input area, and has different training times as the thresholds of the mean square error and the correlation index are different. Taking the urban macro big data of Hebei Province as an example, with the training foundation, without using a GPU and only using an ordinary desktop CPU, and under the conditions of mean square error <50 and correlation index >0.9, the knowledge graph production time of Hebei Province can be shortened to less than 1 hour.

[0093] After the knowledge graph is constructed, it can be used to calculate the urban indicator data of the target area in any future year. Taking the knowledge graph of the above graph structure as an example, the data of some urban indicators in the target area in any future year can be obtained; then, according to the knowledge graph, the multi-level downstream urban indicators with the said part of urban indicators as the pre-indicators are determined, and the paths of the said part of urban indicators and each downstream urban indicator are recorded; the path is connected in sequence by multiple edges, and the length of the path is determined according to the number of edges on each path, and the best data model of each downstream urban indicator is called step by step, firstly, the downstream urban indicator with the shortest path is calculated by the said part of urban indicators, and then the downstream urban indicator is calculated by the said downstream urban indicator, and then the downstream urban indicator of the downstream is calculated by the downstream urban indicator of the downstream, and so on, until the downstream urban indicators at all levels are calculated, and the overall update of urban macro data is realized. Table 2 exemplarily shows the historical year data and future year data of some urban indicators in Hebei Province, which can prove the effectiveness of the method of this embodiment in the measurement of future urban data.

[0094] Table 2

[0095]

[0096] Based on the urban macro data of the target area, this embodiment systematically extracts the quantitative relationship between a large number of urban indicators such as highway diesel fuel consumption, highway freight turnover, number of traffic accidents, number of fires, total water consumption and sulfur dioxide emissions, and adds the quantitative relationship to the knowledge graph to form an overall or energy, traffic, transportation and other thematic relationship knowledge graph and urban system, which is used to measure the future urban data of the target area in terms of energy, traffic, transportation and other aspects. In particular, when processing a large number of urban indicator data, by excluding the second city indicator with the current first city indicator as the pre-indicator, the risk of cyclic call and infinite calculation between a large number of urban indicators is avoided, and the convergence and robustness of the knowledge graph system are guaranteed. At the same time, according to the different relationships between each second city indicator and the current first city indicator, multi-level second city indicators are divided, and data structures such as dictionaries and lists are cleverly used to realize the orderly identification of second city indicators at all levels with simple deep cycles, which fully guarantees the construction efficiency of the knowledge graph. Finally, in the urban data measurement based on the knowledge graph, the overall update of urban macro data is realized step by step by some urban indicators, giving full play to the systematic role of the knowledge graph on urban energy, traffic, transportation and other data.

[0097] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 8A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0098] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data processing method based on the big data city indicator knowledge graph in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned data processing method based on the big data city indicator knowledge graph.

[0099] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0100] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0101] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method based on the big data city indicator knowledge graph of any embodiment is implemented.

[0102] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0103] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0104] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0105] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on big data city indicator knowledge graph, characterized in that: include: S110, obtaining urban macro big data of multiple historical years in the target area, and sorting multiple urban indicators in the urban macro big data according to priority to jointly form an original indicator set; wherein the multiple urban indicators include at least one of the following: highway diesel fuel consumption, highway freight turnover, number of traffic accidents, number of fires, total water consumption and sulfur dioxide emissions; S120: Select a first city indicator with the highest priority from city indicators that have not been selected in the original indicator set, and determine each second city indicator with the first city indicator as a pre-indicator; and form an available indicator set for calculating the first city indicator from other indicators in the original indicator set except the first city indicator and each second city indicator; S130. In the available indicator set, a plurality of indicator combinations are randomly selected in units of a set number of indicators; data of indicator combinations of multiple years are extracted from the urban macro big data as input samples, and multiple preset data models are trained respectively, so that the output of each model continuously approaches the data of the first urban indicator corresponding to each same year; after the training is completed, the indicator combination and data model with the smallest error and satisfying the threshold condition are selected as the optimal indicator combination and the optimal data model for calculating the first urban indicator; wherein the multiple data models include: linear regression model, ridge regression model, regression model, elastic network regression model, decision tree regression model, random forest regression model, LightGBM regression model; return to the operation of selecting the first urban indicator with the highest priority from the urban indicators that have not been selected in the original indicator set, until all urban indicators in the original indicator set are trained; S140. According to the optimal indicator combination and the optimal data model corresponding to each city indicator, a knowledge graph with each city indicator as a node is constructed, and the city data of the target area in future years is calculated according to the knowledge graph.

2. The method according to claim 1, characterized in that The first city index selected in each loop iteration and its corresponding optimal index combination are stored in the relationship tree; Accordingly, the step of determining each second city index with the first city index as a pre-indicator includes: Identify each optimal indicator combination including the current first city indicator in the relationship tree, and use the first city indicators corresponding to each identified optimal indicator combination in the relationship tree as each second city indicator; Each second city indicator is used as a new current first city indicator, and each new best indicator combination including the new current first city indicator is re-identified in the relationship tree, and the first city indicator corresponding to each identified new best indicator combination in the relationship tree is used as each new second city indicator; each new second city indicator is used as an updated current first city indicator, and the operation of identifying the best indicator combination in the relationship tree is returned, and the process is iterated in a loop until no updated second city indicator can be obtained.

3. The method according to claim 2, characterized in that The second city index generated in each loop iteration is stored in the second index set; The step of combining the identified new optimal indicators with the corresponding first city indicators in the relationship tree as new second city indicators includes: Determine whether the first city indicator corresponding to any new optimal indicator combination identified in the relationship tree already exists in the second indicator set; If so, abandoning the corresponding first city indicator as the new second city indicator; If not present, the corresponding first city indicator is taken as a new second city indicator and added to the second indicator set.

4. The method according to claim 1, characterized in that: After the training is completed, the indicator combination and data model with the smallest error and greater than a set threshold are selected as the optimal indicator combination and optimal data model for calculating the first city indicator, including: After the training is completed, if all data models cannot meet the threshold condition, the set number of indicators is reduced, and the operation of randomly selecting multiple indicator combinations in units of the set number of indicators is returned until a data model that meets the threshold condition is selected.

5. The method according to claim 4, characterized in that The operation of randomly extracting multiple indicator combinations in units of a set number of indicators until a data model that meets the threshold condition is selected includes: If the number of set indicators has been reduced to 1, but each data model still cannot meet the threshold condition, the best indicator combination of the first city indicators is set to empty, and the best data model is set to query data.

6. The method according to claim 1, characterized in that The method of constructing a knowledge graph with each city indicator as a node according to the optimal indicator combination and the optimal data model corresponding to each city indicator includes: Take each city indicator as a node of the knowledge graph and perform the following operations on each node: S1. Extract the pre-indicators of the current city indicator from the best indicator combination corresponding to the current node; S2. Adding an edge between each preceding index node and the current node, with each preceding index node as a starting point and the current node as an end point; S3. Mark the best data model corresponding to the current city indicator on each edge, and store the best data model in the model library for the knowledge graph to call.

7. The method according to claim 1, characterized in that The calculating of city data of the target area in future years according to the knowledge graph includes: Obtain indicator data for some cities in the target area in future years; According to the knowledge graph, determine the multi-level downstream city indicators with the partial city indicators as the pre-indicators, and record the paths between the partial city indicators and the downstream city indicators; According to the number of edges on each path, the optimal data model of each downstream city indicator is called step by step, and the indicators of each downstream city are calculated step by step based on the data of the partial city indicators.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method based on the big data city indicator knowledge graph as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements the data processing method based on a big data city indicator knowledge graph as described in any one of claims 1-7.

Images