Method, apparatus, electronic device, and storage medium for processing urban data

Abstract

The present application provides a method, apparatus, electronic device and storage medium for processing urban data, which can be used in the field of intelligent transportation. The method includes: obtaining first state parameters of each preset city; performing data processing on each city by using a preset first index to obtain second state parameters of each city; adjusting the first index for each city based on the degree of difference between the first state parameter and the second state parameter of the same city to obtain a second index for data processing corresponding to each city; extracting a general index that is simultaneously applicable to each city based on each second index, replacing the second index with the general index, and processing the urban data of each city by using the general index. The embodiments of the present application can establish a general index system among different cities, promoting the construction of smart cities.

Description

Technical Field

[0001] This application relates to the field of smart cities, and particularly to a method, apparatus, electronic device, and storage medium for processing urban data. Background Art

[0002] In the field of smart cities, it is of great significance to improve the intelligence of urban data processing through the application of various information technologies. An essential basic processing process for managing a city through information technology is to analyze urban data, and analyzing urban data requires establishing a corresponding index system. In the prior art, due to the excessively high complexity of urban business logics and the huge differences in the actual situations of different cities, it is difficult to establish a generalizable index system among different cities, thus making it difficult to promote the construction of smart cities. Summary of the Invention

[0003] An object of this application is to propose a method, apparatus, electronic device, and storage medium for processing urban data, which can establish a generalizable index system among different cities and promote the construction of smart cities.

[0004] According to one aspect of the embodiments of this application, a method for processing urban data is disclosed. The method includes:

[0005] Obtain first state parameters of each preset city, where the first state parameters are used to describe the target values that the operation state levels of the corresponding cities are expected to reach;

[0006] Process the urban data of each city using a preset first index to obtain second state parameters of each city, where the second state parameters are used to describe the actual values that the operation state levels of the corresponding cities actually reach after processing the urban data of the corresponding cities using the first index;

[0007] Based on the degree of difference between the first state parameters and the second state parameters of the same city, adjust the first index for each city to obtain a second index for data processing of the corresponding city;

[0008] Based on each of the second indexes, extract general indexes that are simultaneously applicable to each city, replace the second indexes with the general indexes, and process the urban data of each city using the general indexes.

[0009] According to one aspect of the embodiments of this application, a device for processing urban data is disclosed. The device includes:

[0010] A first acquisition module, configured to acquire first state parameters of preset cities, where the first state parameters are used to describe target values that the expected operation state levels of the corresponding cities are to reach;

[0011] A second acquisition module, configured to process the city data of each city by using a preset first index to obtain second state parameters of each city, where the second state parameters are used to describe actual values that the actual operation state levels of the corresponding cities reach after processing the city data of the corresponding cities by using the first index;

[0012] An adjustment module, configured to adjust the first index for each city based on the degree of difference between the first state parameter and the second state parameter of the same city to obtain a second index for data processing of the corresponding city;

[0013] An extraction module, configured to extract a general index that is simultaneously applicable to each city based on each of the second indexes, replace the second indexes with the general index, and process the city data of each city by using the general index.

[0014] In an exemplary embodiment of the present application, the device is configured to:

[0015] Determine the adjustment range of the first index for each city based on the degree of interval difference between the first state parameter and the second state parameter of the same city;

[0016] Adjust the first index for each city according to the corresponding adjustment range.

[0017] In an exemplary embodiment of the present application, the device is configured to:

[0018] Determine the intersection interval between the first state parameter and the second state parameter of the same city;

[0019] Determine the union interval between the first state parameter and the second state parameter of the same city;

[0020] Determine the degree of interval difference between the first state parameter and the second state parameter of the same city based on the intersection interval and each of the union intervals.

[0021] In an exemplary embodiment of the present application, the device is configured to:

[0022] Acquire third state parameters of each city, where the third state parameters are used to describe actual values that the actual operation state levels of the corresponding cities reach after processing the city data of the corresponding cities by using the second index;

[0023] Determine the effectiveness scores of the second indicators based on the degree of difference between the first state parameter and the third state parameter of the same city, where the effectiveness scores are used to describe the level of effectiveness obtained after the corresponding second indicators are put into use;

[0024] Determine the weights corresponding to the second indicators based on the effectiveness scores of the second indicators;

[0025] Merge the second indicators based on the weights corresponding to the second indicators to obtain the general indicator.

[0026] In an exemplary embodiment of the present application, the device is configured to:

[0027] Obtain the city data of each city from the city business system of each city by presetting a push gateway at the access layer or by presetting a data extraction component at the access layer;

[0028] Store the city data of each city in the time series database of the storage layer, and analyze it according to the general indicator to obtain an analysis result;

[0029] Transmit the analysis result to the visualization component of the application display layer in real time, and visually display the analysis result through the visualization component.

[0030] In an exemplary embodiment of the present application, the device is configured to:

[0031] Divide the log data of the city data of each city into batch non-real-time log data before the target period and real-time log data within the target period according to the generation time;

[0032] Solidify and store the batch non-real-time log data in the time series library, and analyze it according to the general indicator;

[0033] Store the real-time log data in the time series library, and consume and analyze the real-time log data according to the general indicator by using the predefined Topology in the Storm system.

[0034] In an exemplary embodiment of the present application, the device is configured to:

[0035] Pull the real-time log data by using the Flume system and store it in the Kafka system in a time series data structure;

[0036] Use the predefined Topology in the Storm system to consume and analyze the real-time log data from the Kafka system according to the general indicator, and transmit the analysis result obtained by analyzing the real-time log data to the Redis database.

[0037] In an exemplary embodiment of the present application, the device is configured to:

[0038] Use the Zookeeper system to manage the consumption records of the real-time log data stored in the Kafka system by the Storm system.

[0039] According to an aspect of an embodiment of the present application, an electronic device is disclosed, including: a memory storing computer-readable instructions; a processor reading the computer-readable instructions stored in the memory to execute the method provided in the above embodiments.

[0040] According to an aspect of an embodiment of the present application, a computer program medium is disclosed, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method provided in the above embodiments.

[0041] According to an aspect of an embodiment of the present application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the above embodiments.

[0042] In the embodiments of the present application, since the second index is obtained by applying the first index to the corresponding city and has gone through the practice of the corresponding city. Therefore, the general index extracted based on each second index has more universal practical operability in the application of different cities. Through this method, a general index system can be established among different cities, promoting the construction of smart cities.

[0043] Other features and advantages of the present application will become apparent through the following detailed description, or will be partially learned through the practice of the present application.

[0044] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By referring to the accompanying drawings and describing its exemplary embodiments in detail, the above and other objects, features, and advantages of the present application will become more apparent.

[0046] Figure 1 Shows a schematic diagram of the application system architecture according to an embodiment of the present application.

[0047] Figure 2 Shows a flowchart of a method for processing urban data according to an embodiment of the present application.

[0048] Figure 3 The figure shows a schematic diagram of the relationship between a first indicator and a second indicator according to an embodiment of the present application.

[0049] Figure 4 The figure shows a schematic diagram of the network hierarchical structure of an urban data processing platform according to an embodiment of the present application.

[0050] Figure 5 The figure shows a schematic diagram of an application architecture adopted for processing urban data of each city according to an embodiment of the present application.

[0051] Figure 6 The figure shows a block diagram of a processing device for urban data according to an embodiment of the present application.

[0052] Figure 7 The figure shows a hardware diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted.

[0054] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more example embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the example embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, steps, etc. can be adopted. In other cases, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring the aspects of the present application.

[0055] Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] The present application provides a method for processing urban data, which relates to the application of big data in cloud computing in the field of smart cities.

[0057] Specifically, cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computing devices, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides resources is called the "cloud". The resources in the "cloud" seem to users to be infinitely expandable, and can be obtained at any time, used on demand, expanded at any time, and paid according to usage.

[0058] As a basic capability provider of cloud computing, a cloud computing resource pool (abbreviated as cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtual machines, including operating systems), storage devices, and network devices.

[0059] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. The SaaS layer can also be directly deployed on the IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is various business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0060] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time range. It is a massive, high-growth-rate, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery power, and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process a large amount of data tolerated within a certain time. Technologies applicable to big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0061] Figure 1 The schematic diagram of the application system architecture according to an embodiment of the present application is shown.

[0062] In this embodiment, the urban data processing platform 10, which serves as the data brain of the city, obtains urban data from various urban business systems 20 in the city where it is located. Then, the urban data processing platform 10 processes the urban data to provide support for the operation and management of the city based on the obtained processing results, so as to intelligently improve the operation status level of the city or control the operation status level of the city to remain at the expected level. Among them, the operation and management of the city can be automatically controlled by a module or system integrated with management strategies.

[0063] Each urban business system 20 is mainly used to collect urban data related to different businesses respectively as the data source provided to the urban data processing platform 10. For example, the urban traffic system collects urban data related to traffic, and the urban medical system collects urban data related to medical care.

[0064] The producers of urban data are mainly a large number of devices related to the operation status of the city, including but not limited to cameras, terminals, and sensors. The data generated by these devices is transmitted to the urban business system 20, and after being processed by the urban business system 20, the urban data for the urban data processing platform 10 is obtained.

[0065] Among them, each urban business system 20 is generally a server. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0066] Before describing the specific implementation process of the embodiments of the present application in detail, some concepts involved in the present application are briefly described.

[0067] Urban data refers to the data used to describe the operation status of the city.

[0068] The operation status of the city refers to the state in which the city is in the process of operation. Among them, the operation status of the city includes but is not limited to: the distribution status of natural persons in the city; the traffic status of the city; the distribution status of public facilities in the city.

[0069] General indicators refer to indicators with universality that can simultaneously adapt to data processing in different cities.

[0070] The first indicator refers to the indicator temporarily used for data processing of each city before obtaining the general indicator. The main function of the first indicator is to provide a baseline for obtaining the general indicator.

[0071] The second indicator refers to the indicator obtained after adjusting the first indicator for each city according to the feedback result after data processing when the first indicator is used for data processing of each city. Generally, the second indicators of different cities are different from each other.

[0072] The time series library refers to the time series database, which is mainly used to process and store data with time tags.

[0073] It should be noted that for the purpose of facilitating the understanding of the implementation process of this application, the subsequent description of the specific implementation process of the embodiments of this application is exemplarily based on the urban data processing platform as the execution subject of the embodiments of this application, but it should not limit the functions and scope of use of this application.

[0074] Figure 2 The flowchart of the method for processing urban data according to an embodiment of this application is shown. The method includes:

[0075] Step S310, obtain the first state parameter of each preset city, where the first state parameter is used to describe the target value that the operation state level of the corresponding city is expected to reach;

[0076] Step S320, perform data processing on each city using the preset first indicator to obtain the second state parameter of each city, where the second state parameter is used to describe the actual value that the operation state level of the corresponding city actually reaches after processing the urban data of the corresponding city using the first indicator;

[0077] Step S330, adjust the first indicator for each city based on the degree of difference between the first state parameter and the second state parameter of the same city to obtain the second indicator for data processing of the corresponding city;

[0078] Step S340, extract the general indicator that is simultaneously applicable to each city based on each second indicator, replace the second indicator with the general indicator, and perform data processing on the urban data of each city using the general indicator.

[0079] In the embodiment of this application, the method for processing urban data provided by the embodiment of this application is executed through the pre-established urban data processing platform.

[0080] The first state parameter of each city is preset in the urban data processing platform. The first state parameter is used to describe the target value that the operation state level of the corresponding city is expected to reach. Generally, the first state parameters between different cities are the same or approximate.

[0081] A first indicator is pre-configured in the urban data processing platform. Before obtaining the general indicator, the first indicator mainly serves as the indicator baseline for the urban data of each city, and the first indicator is used to process the urban data of each city. Furthermore, the data processing results obtained by processing the urban data can provide support for the operation and management of the city, thereby adjusting the operation state level of the city.

[0082] The urban data processing platform inputs the first indicator into each city, processes the data of each city, and obtains the second state parameter of each city. The second state parameter is used to describe the actual value actually reached by the operation state level of the corresponding city after processing the corresponding city's data using the first indicator.

[0083] Furthermore, for the same city, based on the degree of difference between its first state parameter and its second state parameter, the first indicator is adjusted for each city to obtain each second indicator. Compared with the first indicator, the second indicator is more in line with the target value expected to be reached by the operation state level of the corresponding city.

[0084] Furthermore, based on each second indicator, a general indicator that is applicable to each city is extracted, and the general indicator is used to replace each second indicator, and the general indicator is used to process the urban data of each city.

[0085] It can be seen that in the embodiment of the present application, since the second indicator is obtained by applying the first indicator to the corresponding city and has gone through the practice of the corresponding city. Therefore, the general indicator extracted based on each second indicator has more general practical operability in the application of different cities. Through this method, an indicator system with universality can be established between different cities, promoting the construction of smart cities.

[0086] In one embodiment, the cities managed by the urban data processing platform include: City A, City B, up to City N.

[0087] The urban data processing platform deploys the first indicator to City A, processes the urban data of City A using the first indicator, and then provides support for the module or system integrated with the management strategy with the obtained processing results, thereby adjusting the operation state level of City A. Since the first indicator is not specifically set for a single city, there is a certain difference between the actual value of the operation state level of City A and the target value of the operation state level of City A after processing the data of City A using the first indicator. In order to meet the target value of the operation state level of City A, an adjustment to the first indicator is triggered in City A to approach the target value of the operation state level of City A. The adjusted first indicator obtained by triggering the adjustment of the first indicator in City A is the second indicator of City A.

[0088] Similarly, when the adjustment of the first indicator is triggered in City B, the adjusted first indicator obtained is the second indicator of City B. This process continues until the adjustment of the first indicator is triggered in City N, and the adjusted first indicator obtained is the second indicator of City N.

[0089] Figure 3 The schematic diagram showing the relationship between the first indicator and the second indicator in an embodiment of the present application is shown.

[0090] Refer to Figure 3 As shown, after the preset first indicator is respectively deployed to City A, City B, and so on until City N, the second indicators of City A, City B, and so on until City N are obtained respectively.

[0091] Furthermore, after the effects obtained by putting the second indicators of each city into use are fed back to the first indicator, the first indicator, which is the baseline, is adjusted again. Then the adjusted first indicator is respectively deployed to each city again, and the second indicators of each city are obtained respectively.

[0092] After cycling the preset number of times, based on each second indicator, a general indicator that is simultaneously suitable for City A, City B, and so on until City N is extracted. In this embodiment, the general indicator can be regarded as the baseline after multiple iterative adjustments.

[0093] In one embodiment, the first state parameter is used to describe the interval in which the target value expected to be reached by the operating state level of the corresponding city is located, and the second state parameter is used to describe the interval in which the actual value actually reached by the operating state level of the corresponding city is located.

[0094] In this embodiment, the urban data processing platform determines the adjustment range of the first indicator for each city based on the degree of interval difference between the first state parameter and the second state parameter of the same city. Then, according to the corresponding adjustment range, the first indicator is adjusted for each city.

[0095] In one embodiment, the interval in which the target value of the operating state level of City A is located is [t1, t2]. After the first indicator is input into the data processing of City A, the interval in which the actual value of the operating state level of City A is located is [t3, t4]. That is, the first state parameter of City A is [t1, t2], and the second state parameter of City A is [t3, t4].

[0096] Determine the degree of interval difference between the two intervals [t1, t2] and [t3, t4], and then determine the adjustment range of the first indicator for City A based on this degree of interval difference. Then, according to the corresponding adjustment range, the first indicator is adjusted for City A. For example: If the degree of interval difference between the two intervals [t1, t2] and [t3, t4] is 25%, the adjustment range of the first indicator for City A can be determined to be 25%, and then the first indicator for City A is adjusted within the range of plus or minus 25%.

[0097] In one embodiment, the urban data processing platform determines the corresponding interval difference degree based on the intersection interval and the union interval between the state parameters.

[0098] In this embodiment, the urban data processing platform determines the intersection interval between the first state parameter and the second state parameter of the same city, and determines the union interval between the first state parameter and the second state parameter of the same city. Furthermore, based on the intersection interval and the union interval, the interval difference degree between the first state parameter and the second state parameter of the same city is determined.

[0099] In one embodiment, the first state parameter of City A is [90, 100], and the second state parameter of City A is [80, 95]. Then, in City A, the intersection interval between the first state parameter and the second state parameter is [90, 95], and the union interval between the first state parameter and the second state parameter is [80, 100]. The interval length of this intersection interval is 5, and the interval length of this union interval is 20. Then, the ratio of 25% obtained by dividing the interval length of the intersection interval by the interval length of the union interval can be used as the interval difference degree between the first state parameter and the second state parameter in City A.

[0100] In one embodiment, the urban data processing platform extracts general indicators based on the effects obtained after the second indicators are put into use.

[0101] In this embodiment, the urban data processing platform obtains the third state parameters of each city. Among them, the third state parameter is used to describe the actual value actually reached by the operation state level of the corresponding city after processing the urban data of the corresponding city using the second indicator.

[0102] Furthermore, based on the difference degree between the first state parameter and the third state parameter of the same city, the effectiveness scores of each second indicator are determined. Among them, the effectiveness score is used to describe the level of the effects obtained after the corresponding second indicator is put into use. Generally, the higher the effectiveness score, the better the effects obtained after the corresponding second indicator is put into use, which also means that the third state parameter is closer to the first state parameter after the corresponding second indicator is put into use.

[0103] Furthermore, based on the effectiveness scores of each second indicator, the weights corresponding to each second indicator are determined. Furthermore, based on the weights corresponding to each second indicator, the second indicators are combined to obtain general indicators.

[0104] In one embodiment, a benchmark score is set in advance. After obtaining the effectiveness scores of each second indicator, each effectiveness score is divided by the benchmark score to obtain the adjusted weights corresponding to each second indicator. Furthermore, the second indicators are weighted according to the obtained adjusted weights to obtain general indicators.

[0105] In one embodiment, after inputting the second indicator into the data processing of the corresponding city, the actual value of the operation status level of the corresponding city is obtained, and then the corresponding third state parameter is obtained. Furthermore, according to the degree of difference between the first state parameter and the third state parameter of the same city, the effectiveness score of the second indicator is determined.

[0106] Among them, the effectiveness score of the second indicator is inversely correlated with the degree of difference. That is, as the degree of difference between the first state parameter and the third state parameter of the same city becomes smaller, the effectiveness score of the second indicator is higher; as the degree of difference between the first state parameter and the third state parameter of the same city becomes larger, the effectiveness score of the second indicator is lower.

[0107] It can be understood that the process of determining the degree of difference between the first state parameter and the third state parameter of the same city is the same as the process of determining the degree of difference between the first state parameter and the second state parameter of the same city. Therefore, the process of determining the degree of difference between the first state parameter and the third state parameter of the same city will not be elaborated here.

[0108] In one embodiment, after quantifying the traffic congestion degree of the city, the value range of the traffic congestion degree is between 0 and 100. The larger the value of the traffic congestion degree, the more congested the overall traffic of the city is.

[0109] For City A, it is expected to control the traffic congestion degree of City A between 0 and 20, that is, the first state parameter of City A is [0, 20].

[0110] Input the first indicator into City A for traffic-related data processing, and provide the processed result to the system integrated with traffic management strategies. After one month, the actual traffic congestion degree of City A is between 10 and 30, that is, the second state parameter of City A is [10, 30].

[0111] The urban data processing platform adjusts the first indicator in City A according to the degree of difference between [0, 20] and [10, 30] to obtain the second indicator of City A.

[0112] Then input the second indicator of City A into City A for traffic-related data processing, and provide the processed result to the system integrated with traffic management strategies. After one month, the actual traffic congestion degree of City A is between 5 and 25, that is, the third state parameter of City A is [5, 25].

[0113] The urban data processing platform determines the effectiveness score of the second indicator of City A according to the degree of difference between [0, 20] and [5, 25]. Furthermore, based on the effectiveness score of the second indicator of City A, the weight of the second indicator of City A is determined.

[0114] Similarly, the urban data processing platform determines the weights of the second indicators for traffic management in each of the other cities, and then weights the second indicators for traffic management in all cities to obtain a general indicator for traffic management.

[0115] In one embodiment, each urban business system is set up in a city to collect corresponding urban data (for example: the personnel flow management system collects urban data used to describe the distribution state of natural persons in the city; the traffic management system collects urban data used to describe the traffic state in the city). Through each urban business system in the city, the urban data processing platform obtains the urban data of each city.

[0116] In one embodiment, the urban data processing platform obtains the urban data of each city from the urban business systems of each city by presetting a push gateway at the access layer.

[0117] Specifically, the urban data processing platform presets a push gateway at the access layer for the urban business systems of each city, and provides an interface and corresponding data structure standard for each business system through this push gateway. Thus, after the urban data collected by each urban business system is standardized according to the corresponding data structure standard, it is pushed into the urban data processing platform through this interface.

[0118] In one embodiment, the urban data processing platform obtains the urban data of each city from the urban business systems of each city by presetting a data extraction component at the access layer.

[0119] Specifically, the urban data processing platform presets a data extraction component at the access layer for the urban business systems of each city, and this data extraction component is used to pull data from the data source where a data connection is established. After the urban data processing platform establishes a data connection with the urban business systems of each city, it pulls the corresponding urban data from each urban business system by calling this data extraction component.

[0120] In one embodiment, after the urban data processing platform obtains the urban data of each city from the urban business system by presetting a push gateway at the access layer or by presetting a data extraction component at the access layer, it stores the urban data in the time series database of the storage layer, analyzes it according to the general indicator to obtain an analysis result; and transmits the analysis result to the visualization component of the application display layer in real time, and visualizes the analysis result through this visualization component.

[0121] Figure 4 Shows a schematic diagram of the network hierarchical structure of the urban data processing platform according to an embodiment of the present application.

[0122] Reference Figure 4As shown, in this embodiment, the push gateway in the access layer provides interfaces for the control center service and other services in the data source, enabling the control center service and other services to standardize the collected urban data according to the corresponding data structure standards and then PUSH it into the access layer through this interface.

[0123] The corresponding urban data is PULLed from databases such as the operation monitoring database to the access layer through the data extraction component.

[0124] The access layer passes the obtained urban data into the storage layer for storage through the time series database. Among them, the urban data is analyzed according to various general indicators in the index library (for example: call indicator, access indicator).

[0125] The storage layer passes the analysis result into the application display layer, and the analysis result is visually displayed through various visualization components in the application display layer. For example: the analysis result is charted through the chart toolbox in the support layer.

[0126] In one embodiment, the urban data processing platform divides the log data of the urban data of each city into batch non-real-time log data before the target period and real-time log data within the target period according to the generation time.

[0127] For the batch non-real-time log data, the urban data processing platform solidifies and stores it in the time series library and analyzes it according to the general indicators.

[0128] For the real-time log data, the urban data processing platform stores it in the time series library and uses the predefined Topology in the Storm system to consume and analyze it according to the general indicators. Among them, the Storm system is a distributed data stream processing system; the Topology is a graph structure task running in the Storm system.

[0129] The advantage of this embodiment is that by storing the data in segments according to the period, the storage of the data can support the accumulation effect of real-time data display at the front end.

[0130] In one embodiment, for the real-time log data, the urban data processing platform uses the Flume system to pull the real-time log data and stores it in the Kafka system in the time series data structure. Then, it uses the predefined Topology in the Storm system to consume and analyze the real-time log data from the Kafka system according to the general indicators, and transmits the analysis result obtained from analyzing the real-time log data into the Redis database. Among them, Flume is a distributed massive log collection, aggregation and transmission system; the Kafka system is an open-source message system; the Redis database is an open-source key-value pair-based database.

[0131] The advantage of this embodiment is that by using the predefined Topology in the Storm system to consume and analyze real-time log data, the orderliness of the processing of real-time log data is ensured.

[0132] In one embodiment, the urban data processing platform uses the Zookeeper system to manage the consumption records of the real-time log data stored in the Kafka system by the Storm system. Among them, the Zookeeper system is a distributed coordination service.

[0133] The advantage of this embodiment is that by using the Zookeeper system to manage the consumption records, the orderliness and real-time nature of the management of the consumption records are ensured. And the Storm system can be quickly restored through the Zookeeper system in the event of the Storm system crashing, improving the reliability of data processing.

[0134] Figure 5 The figure shows a schematic diagram of the application architecture used for processing the urban data of each city in an embodiment of the present application.

[0135] Reference Figure 5 As shown, in this embodiment, the urban data processing platform pulls real-time logs through Flume and stores them in a distributed Kafka system. The log information read from the Kafka system is sent to the Storm system, and the log information is analyzed in real time through a predefined Topology. The analysis results are sent to the Redis database for real-time display by the application display layer; and the Zookeeper system is used to manage the consumption records of the Kafka system in the Storm system; and the log information in the Storm system is sent to the general metric set for subsequent offline analysis. Among them, the general metric set refers to the set of various general metrics.

[0136] During the offline analysis process, the log information of batch non-real-time logs is pushed into the general metric set, and then, based on the general metrics stored in the general metric set, the log information of batch non-real-time logs and the log information of real-time logs are analyzed, and the analysis results are displayed on the application display layer.

[0137] In one embodiment, the urban data is stored in the time series database according to the data source.

[0138] Table 1 below shows the data structure obtained by storing the urban data in the time series database according to the data source.

[0139] Table 1. Data structure obtained by storing according to the data source

[0140]

[0141] As shown in Table 1 for reference, "timestamp" is used to describe the timestamp of the data source of urban data. "cluster" and "hostname" are used to describe the main dimensions of the data source of urban data. "cpu" and "iops" are the index items of urban data; among them, "cpu" is the central processing unit index item, and "iops" is the index item of the number of input / output operations per second of the disk.

[0142] In one embodiment, urban data is stored in a time series database according to index items.

[0143] The following Table 2 shows the data structure obtained by storing urban data in a time series database according to index items.

[0144] Table 2. Data structure obtained by storing according to index items

[0145]

[0146] As shown in Table 2 for reference, "metric" is used to describe the index items of urban data, and the index items include the central processing unit index item "cpu" and the index item of the number of input / output operations per second of the disk "iops". "timestamp" is used to describe the timestamp of the data source of urban data. "cluster" and "hostname" are used to describe the main dimensions of the data source of urban data. "metricvalue" is used to describe the measured value of the index item.

[0147] Figure 6 The processing device for urban data according to an embodiment of the present application is shown, and the device includes:

[0148] The first acquisition module 410 is configured to acquire the first state parameters of each preset city, where the first state parameters are used to describe the target value that the operating state level of the corresponding city is expected to reach;

[0149] The second acquisition module 420 is configured to process the urban data of each city using a preset first index to obtain the second state parameters of each city, where the second state parameters are used to describe the actual value that the operating state level of the corresponding city actually reaches after processing the urban data of the corresponding city using the first index;

[0150] The adjustment module 430 is configured to adjust the first index for each city based on the degree of difference between the first state parameter and the second state parameter of the same city to obtain a second index for data processing of the corresponding city;

[0151] The extraction module 440 is configured to extract general indicators that are applicable to each city based on each of the second indicators, replace the second indicators with the general indicators, and perform data processing on the city data of each city using the general indicators.

[0152] In an exemplary embodiment of the present application, the device is configured to:

[0153] Determine the adjustment range of the first indicator for each city based on the interval difference degree between the first state parameter and the second state parameter of the same city;

[0154] Adjust the first indicator for each city according to the corresponding adjustment range.

[0155] In an exemplary embodiment of the present application, the device is configured to:

[0156] Determine the intersection interval between the first state parameter and the second state parameter of the same city;

[0157] Determine the union interval between the first state parameter and the second state parameter of the same city;

[0158] Based on the intersection interval and each of the union intervals, determine the interval difference degree between the first state parameter and the second state parameter of the same city.

[0159] In an exemplary embodiment of the present application, the device is configured to:

[0160] Obtain the third state parameter of each city, where the third state parameter is used to describe the actual value actually reached by the operation state level of the corresponding city after processing the city data of the corresponding city using the second indicator;

[0161] Based on the difference degree between the first state parameter and the third state parameter of the same city, determine the effectiveness score of each of the second indicators, where the effectiveness score is used to describe the level of effectiveness obtained after the corresponding second indicator is put into use;

[0162] Based on the effectiveness scores of each of the second indicators, determine the weight corresponding to each of the second indicators;

[0163] Based on the weights corresponding to each of the second indicators, merge the second indicators to obtain the general indicator.

[0164] In an exemplary embodiment of the present application, the device is configured to:

[0165] Obtain the city data of each city from the city business system of each city by presetting a push gateway at the access layer or by presetting a data extraction component at the access layer;

[0166] Store the urban data of each city in the time series database of the storage layer, and analyze according to the general metrics to obtain the analysis results;

[0167] Transmit the analysis results to the visualization component of the application display layer in real time, and visually display the analysis results through the visualization component.

[0168] In an exemplary embodiment of the present application, the device is configured to:

[0169] Divide the log data of the urban data of each city into batch non-real-time log data before the target period and real-time log data within the target period according to the generation time;

[0170] Solidify and store the batch non-real-time log data in the time series database, and analyze according to the general metrics;

[0171] Store the real-time log data in the time series database, and use the predefined Topology in the Storm system to consume and analyze the real-time log data according to the general metrics.

[0172] In an exemplary embodiment of the present application, the device is configured to:

[0173] Use the Flume system to pull the real-time log data and store it in the Kafka system in a time series data structure;

[0174] Use the predefined Topology in the Storm system to consume and analyze the real-time log data from the Kafka system according to the general metrics, and transmit the analysis results obtained from analyzing the real-time log data to the Redis database.

[0175] In an exemplary embodiment of the present application, the device is configured to:

[0176] Use the Zookeeper system to manage the consumption records of the real-time log data stored in the Kafka system by the Storm system.

[0177] Next, refer to Figure 7 to describe the electronic device 50 according to an embodiment of the present application. Figure 7 The displayed electronic device 50 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0178] As Figure 7As shown, the electronic device 50 is presented in the form of a general computing device. The components of the electronic device 50 may include, but are not limited to: at least one of the above-mentioned processing units 510, at least one of the above-mentioned storage units 520, and a bus 530 that connects different system components (including the storage unit 520 and the processing unit 510).

[0179] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 510, so that the processing unit 510 executes the steps according to various exemplary embodiments of the present invention described in the description part of the above exemplary method of this specification. For example, the processing unit 510 can execute as Figure 2 shown in each step.

[0180] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202, and may further include a read-only storage unit (ROM) 5203.

[0181] The storage unit 520 may further include a program / utility 5204 having a set (at least one) of program modules 5205. Such program modules 5205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0182] The bus 530 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any bus structure in a variety of bus structures.

[0183] The electronic device 50 can also communicate with one or more external devices 600 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 50, and / or communicate with any device that enables the electronic device 50 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 550. The input / output (I / O) interface 550 is connected to the display unit 540. Also, the electronic device 50 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 560. As shown in the figure, the network adapter 560 communicates with other modules of the electronic device 50 through the bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 50, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0184] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by the way of software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on the network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0185] In an exemplary embodiment of the present application, there is also provided a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of the computer, the computer is enabled to execute the method described in the method embodiment part above.

[0186] According to an embodiment of the present application, there is also provided a program product for implementing the method in the above method embodiment. It can adopt a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0187] The program product may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, but not be limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the readable storage medium (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0189] The program code contained on the readable medium may be transmitted by any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0190] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as JAVA, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0191] It should be noted that although several modules or units of the devices for action execution are mentioned in the foregoing detailed description, such a division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into being embodied by multiple modules or units.

[0192] In addition, although the steps of the methods in this application are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all of the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.

[0193] From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0194] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of this application. This application is intended to cover any variations, uses, or adaptations of this application, which follow the general principles of this application and include known common knowledge or conventional technical means in the technical field not disclosed in this application. The specification and examples are only considered exemplary, and the true scope and spirit of this application are pointed out by the appended claims.

Claims

1. A method for processing urban data, characterized in that, The method includes: Obtaining first state parameters of each preset city, where the first state parameters are used to describe the interval in which the target value expected to be reached by the operation state level of the corresponding city is located; Processing the city data of each city by using a preset first index to obtain second state parameters of each city, where the second state parameters are used to describe the interval in which the actual value actually reached by the operation state level of the corresponding city is located after processing the city data of the corresponding city by using the first index; Based on the degree of interval difference between the first state parameter and the second state parameter of the same city, determining the adjustment range of the first index for each city, and adjusting the first index for each city according to the corresponding adjustment range to obtain a second index for data processing of the corresponding city; Based on each of the second indexes, extracting a general index that is simultaneously applicable to each city, replacing the second index with the general index, and processing the city data of each city by using the general index; Among them, extracting a general index that is simultaneously applicable to each city based on each of the second indexes includes: Obtaining third state parameters of each city, where the third state parameters are used to describe the actual value actually reached by the operation state level of the corresponding city after processing the city data of the corresponding city by using the second index; Based on the degree of difference between the first state parameter and the third state parameter of the same city, determining the effectiveness score of each of the second indexes, where the effectiveness score is used to describe the level of effectiveness obtained after the corresponding second index is put into use; Based on the effectiveness scores of each of the second indexes, determining the weights corresponding to each of the second indexes; Based on the weights corresponding to each of the second indexes, merging the second indexes to obtain the general index.

2. The method according to claim 1, wherein The method further includes: Determining the intersection interval between the first state parameter and the second state parameter of the same city; Determining the union interval between the first state parameter and the second state parameter of the same city; Based on the intersection interval and the union interval, determining the degree of interval difference between the first state parameter and the second state parameter of the same city.

3. The method according to claim 1, characterized in that, The method further includes: Obtaining the city data of each city from the city business system of each city by presetting a push gateway at the access layer or by presetting a data extraction component at the access layer; Storing the city data of each city in a time series database in the storage layer, and analyzing according to the general index to obtain an analysis result; Transmitting the analysis result to a visualization component in the application display layer in real time, and visually displaying the analysis result through the visualization component.

4. The method according to claim 3, characterized in that Storing the city data of each city in a time series database in the storage layer and analyzing according to the general index includes: Dividing the log data of the city data of each city according to the generation time into batch non-real-time log data before the target period and real-time log data within the target period; Solidifying and storing the batch non-real-time log data in the time series database, and analyzing according to the general index; Store the real-time log data in the time series database, and use the predefined Topology in the Storm system to consume and analyze the real-time log data according to the general metrics.

5. The method according to claim 4, wherein Store the real-time log data in the time series database, and use the predefined Topology in the Storm system to consume and analyze the real-time log data according to the general metrics, including: Use the Flume system to pull the real-time log data and store it in the Kafka system in a time series data structure; Use the predefined Topology in the Storm system to consume and analyze the real-time log data from the Kafka system according to the general metrics, and transmit the analysis results obtained from analyzing the real-time log data into the Redis database.

6. A data processing device for a city, characterized in that, The device includes: A first acquisition module configured to acquire the first state parameters of each preset city, where the first state parameters are used to describe the interval where the target value expected to be reached by the operating state level of the corresponding city is located; A second acquisition module configured to process the city data of each city using a preset first metric to obtain the second state parameters of each city, where the second state parameters are used to describe the interval where the actual value actually reached by the operating state level of the corresponding city is located after processing the city data of the corresponding city using the first metric; An adjustment module configured to determine the adjustment range of the first metric for each city based on the interval difference degree between the first state parameter and the second state parameter of the same city, and adjust the first metric for each city according to the corresponding adjustment range to obtain a second metric for data processing of the corresponding city; An extraction module configured to extract a general metric that is simultaneously applicable to each city based on each of the second metrics, replace the second metrics with the general metric, and process the city data of each city using the general metric; Wherein, the extraction module is further configured to: Acquire the third state parameters of each city, where the third state parameters are used to describe the actual value actually reached by the operating state level of the corresponding city after processing the city data of the corresponding city using the second metric; Determine the effectiveness score of each of the second metrics based on the difference degree between the first state parameter and the third state parameter of the same city, where the effectiveness score is used to describe the level of effectiveness obtained after the corresponding second metric is put into use; Determine the weight corresponding to each of the second metrics based on the effectiveness scores of each of the second metrics; Merge the second metrics based on the weights corresponding to each of the second metrics to obtain the general metric.

7. An electronic device, characterized in that, Including: A memory storing computer-readable instructions; A processor that reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1-5.

9. A computer program product, characterized in that, including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method according to any one of claims 1-5.

Images