A smart city data integration and sharing method, platform, device and storage medium

Abstract

The present application relates to a smart city data integration and sharing method, platform, device and storage medium, which belongs to the field of smart city information technology, wherein the method includes: acquiring business data generated by all city management systems in real time, predicting business needs based on the business data acquired in real time, generating constraints and corresponding minimum transformation sets according to the business needs; based on the minimum transformation set, extracting specific business data corresponding to data parameters, integrating and processing the business data, forming business needs and their corresponding integrated business data into demand content for storage; when receiving actual needs proposed by users, matching the actual needs with the stored demand content, and if the match is successful, outputting the demand content that is successfully matched; if the match is unsuccessful, determining the corresponding business data based on the actual needs and outputting it after integration. The present application has the effect of realizing data sharing while reducing the computing power burden consumed by data conversion and integration.

Description

Technical Field

[0001] The present application relates to the field of smart city information technology, and in particular to a smart city data integration and sharing method, platform, device and storage medium. Background Art

[0002] Smart cities improve urban operational efficiency and public service levels by integrating information and communication technologies. Currently, smart city construction has widely adopted technologies such as cloud computing, the Internet of Things, and big data for data collection, transmission, and processing. However, current smart city management systems mostly use decentralized data management methods, resulting in poor data interoperability between systems, making it difficult to effectively utilize data. To meet the needs of modern smart city management and achieve cross-system data sharing, existing technologies have proposed using data conversion methods to collect and standardize the data generated by all smart city management systems to unify the data format, thereby breaking down information silos and achieving data integration between systems, allowing managers to access cross-system data.

[0003] However, based on the above technical solutions, it can be seen that since the data generated by the smart city management system may be generated in real time, such as the traffic management system, meteorological management system, etc., it is easy to cause a large amount of data to be converted and integrated, thereby increasing the computing burden, so it needs to be improved. Summary of the Invention

[0004] In order to meet users' needs for data sharing and reduce the computing power burden consumed by data conversion and integration, this application provides a smart city data integration and sharing method, platform, device and storage medium.

[0005] In a first aspect, this application provides a method for integrating and sharing smart city data, including:

[0006] Acquire business data generated by all city management systems in real time, predict business needs based on the real-time business data, and generate constraints and corresponding minimum transformation sets based on the business needs; wherein the constraints include at least time conditions and space conditions, and the minimum transformation set includes data parameters from different city management systems that meet the corresponding constraints;

[0007] Based on the minimum conversion set, extract the specific business data corresponding to the data parameters, integrate the business data, and form the business requirements and the corresponding integrated business data into demand content for storage;

[0008] When receiving the actual needs proposed by the user, the actual needs are matched with the stored demand content first. If the match is successful, the matched demand content is output; if the match is unsuccessful, the corresponding business data is determined based on the actual needs and integrated and output.

[0009] By adopting the above technical solution, for the data collection, conversion and integration processing of all urban management systems, this application proposes: by predicting the scope boundaries of business needs (i.e., constraints), determining precise local data (i.e., the minimum conversion set that meets the constraints), and then, only converting and storing the local data. When the user makes an actual demand, the demand content that matches the actual demand is preferentially screened out from the stored demand content. On the one hand, early conversion is achieved to improve demand response efficiency. On the other hand, only local data is accurately converted, rather than converting and integrating global data (i.e., unified conversion and integration of data from all urban management systems), thereby reducing the computing power burden.

[0010] Optionally, the forming of the business demand and the corresponding integrated business data into demand content for storage includes:

[0011] The integrated and processed business data is used as data to be activated, and the association relationship between the data to be activated and the business needs is established;

[0012] Determining the demand frequency and change frequency of the data to be activated, and storing the data to be activated whose demand frequency is higher than a preset demand frequency threshold and whose change frequency is lower than the preset change frequency threshold as first data on the sharing platform;

[0013] For all data to be activated other than the first data, as the second data, based on the city management system to which the second data belongs, the second data is stored in a preset edge cache area corresponding to the city management system to which the second data belongs, and the edge cache area includes a dynamic cache area and a static cache area, and the second data with a change frequency higher than a preset change frequency threshold is stored in the dynamic cache area, and the second data with a change frequency not higher than the preset change frequency threshold is stored in the static cache area;

[0014] According to different business requirements associated with the second data stored in the edge cache, respectively configure processing engines for the second data associated with different business requirements in the edge cache, and make the processing engines correspond to the business requirements one by one;

[0015] If the match is successful, the content of the successfully matched requirement is output, including:

[0016] If the match is successful, the storage location of the data to be activated corresponding to the successfully matched demand content is determined. If it is stored in the shared platform, the data to be activated corresponding to the successfully matched demand content is output; if it is stored in the edge cache area, the data to be activated corresponding to the successfully matched demand content is retrieved and transmitted to the shared platform through the corresponding processing engine for output by the shared platform.

[0017] By adopting the above technical solution, the processed and integrated business data is partitioned and stored to reduce the memory usage of the shared platform. At the same time, a processing engine is configured for the data stored in the edge cache area to interact with the shared platform with the help of the processing engine, that is, the data in the cache area is retrieved and transmitted to the shared platform.

[0018] Optionally, taking the integrated and processed business data as data to be activated includes:

[0019] For the integrated business data, based on the corresponding constraints, the business data is initially sharded according to the preset spatial dimension, and then the business data after the initial sharding is secondary sharded according to the preset business cycle. The data sets of the business data contained in each layer after the secondary sharding are respectively used as data to be activated.

[0020] By adopting the above technical solution, after the business data is integrated and processed and before partitioned storage, two-dimensional sharding is performed based on the preset spatial dimension (i.e., geographic location) and the preset business cycle dimension (i.e., the demand for corresponding data at different times). The storage location is then determined for each shard data, thereby improving the refinement and accuracy of the data cache location.

[0021] Optionally, the step of forming demand content from the business demand and the corresponding integrated business data and storing the content further includes:

[0022] Using the second data whose change frequency is higher than a preset change frequency threshold as the third data, predicting the demand time of the business demand corresponding to the third data in a future time period, where the demand time is used to represent the time when the corresponding actual demand is proposed by the user when the corresponding business demand is successfully matched with the actual demand proposed by the user;

[0023] According to the demand time corresponding to the third data, the control processing engine retrieves the original data from the city management system to which the third data belongs at a specified time point before the demand time, converts the data into the third data, and then stores it in the corresponding dynamic cache area.

[0024] By adopting the above technical solution, for the third data whose change frequency is higher than the preset change frequency threshold, the present application proposes: predicting the demand time of the third data, that is, predicting the time when the user will demand the third data, and only retrieving the original data from the corresponding city management system at a specified time point before the demand time and converting the original data into the third data, and storing it in the dynamic cache area. On the one hand, this solution can reduce the real-time memory usage in the dynamic cache area, and on the other hand, it can also reduce the frequency of the processing engine's processing of the third data, reducing unnecessary computing power burden on the processing engine.

[0025] Optionally, configuring processing engines for the second data associated with different business requirements in the edge cache area includes:

[0026] Determine all target business requirements associated with the second data within the cache area, and calculate a combined density value of each target business requirement based on preset reference dimensions, where the reference dimensions include at least a city management system dimension to which the data belongs, a fine-grained dimension of the data, and a cache dimension to which the data belongs;

[0027] Based on the combined density value, the density level and corresponding processing engine type of each target business requirement are determined, and a processing engine corresponding to the determined processing engine type is configured for each target business requirement; wherein, the density level is used to characterize the processing complexity of the business requirement.

[0028] By adopting the above technical solution, when configuring the processing engine, the processing complexity of the corresponding business demand will be determined based on the combined density value of the business demand associated with the second data (i.e., the target business demand), and then the processing engine type will be matched specifically according to the processing complexity, so as to adapt to the efficient processing of business demands with processing complexity.

[0029] Optionally, the method further includes:

[0030] Analyze the changes in business data corresponding to each business requirement and predict the change trend in the future period. Based on the prediction results, determine and output the recommended monitoring frequency to the user. The recommended monitoring frequency is used to represent the frequency of the corresponding business requirements proposed by the user;

[0031] When an authorized push instruction from the user is received, the service data corresponding to the service requirement is output to the user according to the recommended monitoring frequency corresponding to the service requirement contained in the authorized push instruction.

[0032] By adopting the above technical solution, the solution reduces the amount of manual requests by users and improves the timeliness of discovering changes in corresponding business needs.

[0033] Optionally, the method further includes:

[0034] High-frequency business needs are updated and determined in real time, and an independent communication space is created for each high-frequency business need, including a raw data interface, a format conversion engine, and a user collaboration layer. A high-frequency business need refers to a business need that has successfully matched actual needs raised at different times or by different users more than a preset number of times. The independent communication space is set up in a one-to-one correspondence with the business need. The user collaboration layer is used to enable communication and data sharing among all users who have raised the corresponding business need. The raw data interface is used for users to retrieve raw data corresponding to the corresponding business need from the corresponding city management system. The format conversion engine is used for users to convert the raw data into a specified format.

[0035] When the actual demand proposed by the user is received, the following steps are included:

[0036] Receive demand instructions from users, display all independent communication spaces and corresponding business needs, and when users provide selected content with any independent communication space, configure permissions for users to enter and use the original data interface, format conversion engine, and user collaboration layer within the independent communication space; if users do not provide selected content, the user's feedback result will be used as the actual demand.

[0037] By adopting the above technical solution, an independent communication space is created separately for high-frequency business needs, which is used to provide services specifically for users who raise such business needs. The independent communication space is set up independently of the shared platform, allowing users to directly retrieve the original data corresponding to the business needs corresponding to the communication space, and also allowing users to convert the original data by themselves, eliminating the need for the shared platform to match the business needs raised by users every time, and instead allowing users to directly enter the corresponding independent communication space.

[0038] Secondly, this application provides a smart city data integration and sharing platform, including:

[0039] A business demand prediction module is used to obtain business data generated by all city management systems in real time, predict business needs based on the real-time business data, and generate constraints and corresponding minimum transformation sets based on the business needs; wherein the constraints include at least time conditions and space conditions, and the minimum transformation set includes data parameters from different city management systems that meet the corresponding constraints;

[0040] A prediction result storage module is used to extract specific business data corresponding to the data parameters based on the minimum transformation set, integrate the business data, and form demand content with the business demand and its corresponding integrated business data for storage;

[0041] The actual demand response module is used to match the actual demand with the stored demand content when receiving the actual demand proposed by the user. If the match is successful, the matched demand content is output; if the match is unsuccessful, the corresponding business data is determined based on the actual demand and integrated and output.

[0042] In a third aspect, the present application provides a smart city data integration and sharing device, comprising a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any of the methods described in the first aspect.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the methods described in the first aspect.

[0044] In summary, this application includes at least one of the following beneficial technical effects:

[0045] Regarding the data collection, conversion, and integration process for all city management systems, this application proposes: by prejudging the scope boundaries of business needs (i.e., constraints), determining precise local data (i.e., the minimum conversion set that satisfies the constraints), then converting and storing only the local data. When users raise actual needs, the stored needs are prioritized to select the ones that match the actual needs. This, on the one hand, achieves early conversion and improves demand response efficiency. On the other hand, it reduces the computing power burden by only converting the local data accurately, rather than converting and integrating the global data (i.e., uniformly converting and integrating the data of all city management systems).

[0046] Furthermore, the processed and integrated business data is partitioned and stored to reduce the memory usage of the shared platform. At the same time, a processing engine is configured for the data stored in the edge cache area to enable interaction with the shared platform through the processing engine, that is, to retrieve the data in the cache area and transmit it to the shared platform;

[0047] Furthermore, after the business data is integrated and processed, and before partitioned storage, two-dimensional sharding is performed based on the preset spatial dimension (i.e., geographic location) and the preset business cycle dimension (i.e., the demand for corresponding data at different times). The storage location is then determined for each shard data, thereby improving the refinement and accuracy of the data cache location. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 This is a flow chart of a smart city data integration and sharing method disclosed in an embodiment of the present application.

[0050] Figure 2 This is a structural block diagram of a smart city data integration and sharing platform disclosed in an embodiment of the present application.

[0051] Description of the accompanying drawings: 201, business demand prediction module; 202, prediction result storage module; 203, actual demand response module. DETAILED DESCRIPTION

[0052] The following is combined with Figure 1-2 This application is described in further detail.

[0053] The embodiment of the present application discloses a smart city data integration and sharing method (hereinafter referred to as the sharing method) which aims to aggregate and integrate the data generated by all city management systems, and when the user makes an actual demand (i.e., when the user makes a demand to access the data), the corresponding data is fed back to the user, so as to realize cross-system data sharing and break down information barriers. At the same time, the present application will only process and integrate the demand-related data according to the needs of the user, rather than uniformly converting and integrating the global data, so as to reduce the computing burden and resource consumption. The execution subject of the sharing method of the present application is a smart city data integration and sharing platform (hereinafter referred to as the sharing platform), which will be combined below. Figure 1 , specifically explains the execution process of the sharing platform for the sharing method.

[0054] S101, acquire the business data generated by all city management systems in real time, predict the business needs based on the business data acquired in real time, and generate constraints and corresponding minimum transformation sets according to the business needs; wherein the constraints include at least time conditions and space conditions, and the minimum transformation set includes data parameters that meet the corresponding constraints and originate from different city management systems.

[0055] In implementation, the urban management system can specifically include an intelligent traffic management system, an underground pipe network detection system, a medical emergency dispatch system, an educational resource management system, a meteorological monitoring system, etc., and realize the data access of all urban management systems to the shared platform through a unified adapter that supports four access modes: API, message queue, file transfer and database log. For example, Apache Kafka+Schema Registry access technology is used to access real-time streaming data; Apache Spark Delta Lake access technology is used to access batch data; Geomesa+GeoMesa Kafka access technology is used to access spatial data, and finally the shared platform can obtain the business data generated by all urban management systems, and the business data obtained at this time is the original data collected by the urban management system.

[0056] The sharing platform will analyze the aforementioned raw data acquired in real time. Specifically, the raw data from different urban management systems will be input into a preset combination pattern mining algorithm, so that the combination pattern mining algorithm outputs a combination of data parameters (i.e., a minimum transformation set) and corresponding constraints. The minimum transformation set here includes data parameters from more than one urban management system. For example, the minimum transformation set includes data parameter a and data parameter b, where data parameter a comes from urban management system A and data parameter b comes from urban management system B. For example, the minimum data set can be: hourly rainfall, drainage network load, and traffic congestion index; it can be seen that the three parameters included in this minimum data set come from the meteorological monitoring system, water system, and intelligent transportation management system respectively.

[0057] Constraints are conditions used to limit the numerical range of the data parameters in the minimum transformation set. For example, they include spatial and temporal conditions. Spatial conditions require that the specific values ​​of the data parameters in the minimum transformation set be within a fixed geographic area, while temporal conditions require that the specific values ​​of the data parameters in the minimum transformation set be data values ​​generated within a specified time period. In addition to specific data values, the raw data input into the combined pattern mining algorithm also includes geographic location (such as geographic coordinates) and data generation time (such as timestamp).

[0058] The combined pattern mining algorithm is specifically an improved FP-Growth algorithm. Specifically, it converts raw data into a spatiotemporal transaction set through a sliding window and spatial clustering (e.g., using the density-based DBSCAN algorithm with a preset distance threshold as a parameter). Data items within the same transaction must satisfy a spatial distance ≤ a preset distance threshold (e.g., 1 km) and a time interval ≤ a preset maximum time interval (e.g., 1 hour). Next, an FP-Tree is constructed, allowing candidate data lines to be inserted only if they meet spatiotemporal co-occurrence constraints with the current path node (spatiotemporal co-occurrence constraints are accelerated by R-tree indexing for spatial relationship calculation). Frequent item sets (i.e., minimum transformation sets) that meet a preset minimum support are mined from the FP-Tree, and their associated spatiotemporal ranges (i.e., constraints) are recorded.

[0059] Correspondingly, the minimum transformation set and constraint conditions are the specific manifestations of business requirements. An ID number can be generated for the business requirements, and an association relationship can be established between the business requirements and the minimum data set and constraint conditions.

[0060] S102 , based on the minimum conversion set, extracting specific business data corresponding to the data parameters, and integrating the business data, forming the business demand and its corresponding integrated business data into demand content for storage.

[0061] S103, when receiving the actual demand proposed by the user, the actual demand is matched with the stored demand content first. If the match is successful, the matched demand content is output; if the match is unsuccessful, the corresponding business data is determined based on the actual demand and integrated and output.

[0062] During implementation, the sharing platform is used to access specific values ​​(i.e., business data) corresponding to specific data parameters contained in the minimum transformation set that meet the constraints from the urban management system, and use a preset integration algorithm to convert the format of the business data within the same minimum transformation set to achieve unification. The conversion process specifically includes field mapping of structured data, unit unification, and coordinate conversion of spatial data. Since data format conversion and standardization are existing technologies, they will not be elaborated here.

[0063] The sharing platform is used to consolidate and store the integrated business data and its corresponding business requirements into demand content. It is also used to receive actual requirements submitted by users in real time. It should be explained that users can access the sharing platform in web form and enter the data parameter combination and constraints they want to consult in the preset input boxes on the sharing platform's access interface, thereby forming an actual requirement and triggering the submission of the actual requirement. The sharing platform is used to receive the actual requirement and then compare the data parameter combination contained in the actual requirement with the minimum transformation set of all stored demand content. If the comparison is consistent (i.e., the data parameters contained in the data parameter combination are completely consistent with the data parameters contained in the minimum transformation set, and the post-constraint conditions corresponding to the actual requirement are included in the constraints corresponding to the demand content), it is considered a successful match, and the business data in the successfully matched demand content that meets the constraints corresponding to the actual requirement is output. If the match is unsuccessful, based on the data parameter combination contained in the actual demand and its corresponding constraints, the original data corresponding to each data parameter in the data parameter combination that meets the constraints is retrieved from the corresponding city system, and then converted and unified to form the business data for output.

[0064] Optionally, “storing the business demand and its corresponding integrated business data into demand content” in S102 includes:

[0065] For the integrated business data, based on the corresponding constraints, the business data is initially sharded according to the preset spatial dimension, and then the business data after the initial sharding is secondary sharded according to the preset business cycle. The data sets of the business data contained in each layer after the secondary sharding are respectively used as data to be activated; and the association relationship between the data to be activated and the business needs is established.

[0066] The demand frequency and change frequency of the data to be activated are determined, and the data to be activated whose demand frequency is higher than a preset demand frequency threshold and whose change frequency is lower than the preset change frequency threshold is stored as first data in the sharing platform.

[0067] For all data to be activated that are not the first data, as the second data, the second data is stored in a preset edge cache area corresponding to the city management system to which the second data belongs, and the edge cache area includes a dynamic cache area and a static cache area. The second data whose change frequency is higher than the preset change frequency threshold is stored in the dynamic cache area, and the second data whose change frequency is not higher than the preset change frequency threshold is stored in the static cache area.

[0068] According to the different business needs associated with the second data stored in the cache area, all target business needs associated with the second data in the cache area are determined, and based on preset reference dimensions, the combined density value of each target business need is calculated, wherein the reference dimensions include at least the city management system dimension to which the data belongs, the fine-grained dimension of the data, and the cache dimension to which the data belongs.

[0069] Based on the combined density value, the density level and corresponding processing engine type of each target business requirement are determined, and a processing engine corresponding to the determined processing engine type is configured for each target business requirement; and the processing engine and the business requirement are made to correspond one to one; wherein the density level is used to characterize the processing complexity of the business requirement.

[0070] The second data whose change frequency is higher than the preset change frequency threshold is used as the third data, and the demand time of the business demand corresponding to the third data in the future time period is predicted. The demand time is used to represent the time when the corresponding actual demand is proposed by the user when the corresponding business demand is successfully matched with the actual demand proposed by the user.

[0071] According to the demand time corresponding to the third data, the control processing engine retrieves the original data from the city management system to which the third data belongs at a specified time point before the demand time, converts the data into the third data, and then stores it in the corresponding dynamic cache area.

[0072] And the step of “if the match is successful, outputting the successfully matched requirement content” in S103 further includes the following sub-steps:

[0073] If the match is successful, the storage location of the data to be activated corresponding to the successfully matched demand content is determined. If it is stored on the shared platform, the data to be activated corresponding to the successfully matched demand content is output; if it is stored in the cache area, the data to be activated corresponding to the successfully matched demand content is retrieved and transmitted to the shared platform through the corresponding processing engine for output by the shared platform.

[0074] During implementation, the sharing platform pre-stores the demand heat index of each data parameter generated at different geographical locations in the city and at different time periods corresponding to each geographical location. The demand heat index is used to characterize the frequency with which the corresponding data parameter is viewed by users at the corresponding geographical location and during the corresponding time period. For the integrated and processed business data, the sharing platform determines its demand heat index range based on the spatial range and time range contained in the corresponding constraint conditions, and then divides the determined demand heat index range into several demand heat index sub-ranges. The spatial range and time range contained in the constraint conditions are divided according to the demand heat index sub-ranges, that is, the spatial sub-ranges and time sub-ranges that meet the demand heat index sub-ranges are divided, and the spatial sub-ranges and time sub-ranges are formed into constraint sub-conditions. Finally, the business data is segmented to form data to be activated, so that the data to be activated corresponds to the constraint sub-conditions one-to-one. The data to be activated is the data to be activated that meets the corresponding constraint sub-conditions. Finally, based on the business data to which the data to be activated belongs, an association relationship is established between the data to be activated and the business demand.

[0075] Next, the sharing platform is used to determine the demand frequency and change frequency of the data to be activated. The demand frequency here can be considered as the corresponding demand index sub-range, and if the difference between the original data corresponding to adjacent timestamps is greater than the preset difference, it is considered that the original data has changed. The change frequency can be expressed by the time difference between the timestamps of the original data before and after the change when the difference is greater than the preset difference (that is, a change has occurred).

[0076] The data to be activated whose demand frequency is higher than the preset demand frequency threshold and whose change frequency is lower than the preset change frequency threshold is stored in the sharing platform as the first data (ie, the data to be activated with high demand and high stability).

[0077] All other data to be activated other than the first data are collectively referred to as the second data. The second data will be further split according to the city management system to which the data parameter corresponding to the second data belongs. That is, according to the different city management systems to which the corresponding data parameters belong, the second data will be stored separately in the city management system to which the corresponding data parameters belong and in the preset edge cache area. It should be noted here that the preset edge cache area of ​​each city management system can be specifically split into a dynamic cache area and a static cache area, and the second data with a change frequency higher than the preset change frequency threshold will be stored in the dynamic cache area, and the second data with a change frequency not higher than the preset change frequency threshold will be stored in the static cache area.

[0078] For the business data stored in the edge cache, we can further determine all the business requirements associated with the business data in all edge caches (hereinafter referred to as target business requirements) based on the different business requirements associated with the business data, and configure a processing engine for each target business requirement. This achieves a one-to-one correspondence between the processing engine and the target business requirement. The specific configuration process is as follows:

[0079] According to the preset reference dimensions (i.e., the city management system dimension, the data fine-grained dimension, and the data cache dimension), determine the reference dimension values ​​for each target business requirement. The dimension value corresponding to the city management system dimension is the number of city management systems to which the data parameters contained in the target business requirement belong. For example, if the target business requirement contains data parameters for three different city management systems, the corresponding city management system dimension is 3.

[0080] The data fine-grained dimension is specifically used to characterize the complexity of the data parameters. The dimensional data corresponding to the data fine-grained dimension can be obtained by weighted summation based on the business data processing time and data volume corresponding to all data parameters included in the target business needs. The data cache dimension is used to characterize the complexity of the storage method of all data parameters included in the target business needs. The dimensional value corresponding to the data cache dimension can be obtained by the score accumulation method. For example, the score corresponding to the shared platform storage is 1, the score for the static cache area storage is 2, and the score for the dynamic cache area storage is 3. The storage method scores of the business data corresponding to each data parameter included in the target business needs are accumulated to obtain the dimensional value corresponding to the data cache dimension. Based on the preset weight of each parameter dimension and the dimensional value of each parameter dimension, the combined density value of each target business need is calculated by weighted summation.

[0081] Based on several pre-stored combination density ranges, the density level corresponding to each combination density range, and the processing engine type corresponding to each density level, the combination density range of each target business demand is determined, and then the processing engine type and the corresponding processing engine corresponding to each target business demand are determined. According to the corresponding type of processing engine, the business data of the associated target business demand is retrieved and format converted.

[0082] Accordingly, the processing engine types disclosed in the embodiments of this application specifically include the following three types:

[0083] 1. Lightweight engine (CPU-optimized), specifically x86 / ARM multi-core CPU, supporting AVX512 instruction set; reads data directly from edge cache or shared platform.

[0084] Map business data to the process address space through memory mapping (mmap), set NUMA node affinity to ensure that the CPU core accesses local memory, and use the AVX-512 instruction set to process data fields in parallel.

[0085] 2. Acceleration engine (GPU+FPGA heterogeneous type). In this case, business data in the dynamic cache is transferred to the FPGA via PCIe Gen4DMA. The FPGA is used to perform real-time cleaning and format conversion on the business data in the dynamic cache. The pre-processed data is transferred to the GPU memory via GPUDirect RDMA. The CUDA kernel function is started to perform parallel computing. The FPGA and GPU exchange metadata via NVLink, establishing a shared memory window to achieve sub-microsecond synchronization. This forms a heterogeneous acceleration engine that combines FPGA streaming pre-processing with GPU batch computing.

[0086] 3. Distributed collaboration engine

[0087] Business data in the dynamic cache is sharded by time using Kafka, and data in the static cache is sharded by geographic space using GeoHash. Business data in the dynamic cache is processed using the streaming computing engine (Flink), and analyzed using the batch processing engine (Spark SQL). The results are then integrated using incremental materialized views.

[0088] In addition, the sharing platform calls the second data in the edge cache area, whose change frequency is higher than the preset change frequency threshold, the third data. The sharing platform is also used to predict the demand time of the business demand corresponding to the third data in the future time period. The specific prediction method can be obtained by referring to the demand heat index of each data parameter generated in different geographical locations of the city and different time periods corresponding to each geographical location, which is pre-stored in the relevant sharing platform mentioned above. The demand heat index is used to characterize the frequency of user access to the corresponding data parameter at the corresponding geographical location and in the corresponding time period; the time when the demand heat index corresponding to the third data in the future time period is higher than the specified index is used as the demand time.

[0089] Only when the current time reaches the specified time before the demand time for the third data will the processing engine retrieve the corresponding raw data from the city management system to which the data parameter corresponding to the third data belongs, convert it into the third data, and then store the third data in the corresponding dynamic buffer. Furthermore, the following conditions must be met: the change frequency of the third data > (demand time - time).

[0090] Correspondingly, the sharing platform will retrieve the data to be activated according to the storage location of the data to be activated. If the data to be activated is stored in the sharing platform, the corresponding data to be activated will be directly retrieved and output from the sharing platform. If it is stored in the edge cache area, the processing engine will retrieve the corresponding data to be activated and transmit it to the sharing platform so that the sharing platform can output it.

[0091] Optionally, the sharing method further includes the following steps:

[0092] Analyze the changes in business data corresponding to each business requirement and predict the change trend in the future period. Based on the prediction results, determine and output the recommended monitoring frequency to the user. The recommended monitoring frequency is used to represent the frequency of users raising corresponding business requirements;

[0093] When an authorized push instruction from the user is received, the business data corresponding to the business requirement is output to the user according to the recommended monitoring frequency corresponding to the business requirement contained in the authorized push instruction.

[0094] During implementation, based on the frequency of changes in business data corresponding to each business demand and the uniformly changing prediction rule, the data values ​​of the corresponding business data in the future time period are predicted, and then based on the prediction result, the recommended monitoring frequency is output to the user. The recommended monitoring frequency can be the difference in timestamps before and after the change when the business data change amplitude is greater than the preset change amplitude (that is, the time difference of the timestamps of the original data before and after the change is greater than the preset difference and greater than the difference threshold). The user can feedback the authorization push instruction, that is, entrust the sharing platform to automatically output the latest business data of the corresponding business demand to the user regularly according to the corresponding recommended detection frequency.

[0095] Optionally, the sharing method further includes the following steps:

[0096] High-frequency business needs are updated and determined in real time. An independent communication space is created for each high-frequency business need, including a raw data interface, a format conversion engine, and a user collaboration layer. A high-frequency business need refers to a business need that successfully matches actual needs raised at different times or by different users more than a preset number of times. Independent communication spaces are set up in a one-to-one correspondence with business needs. The user collaboration layer is used for all users who raise corresponding business needs to communicate and share data. The raw data interface is used for users to retrieve raw data corresponding to the corresponding business need from the corresponding city management system. The format conversion engine is used for users to convert raw data into a specified format.

[0097] The “when receiving the actual demand proposed by the user” in S103 includes the following steps:

[0098] Receive demand instructions from users, display all independent communication spaces and corresponding business needs, and when users provide selected content with any independent communication space, configure permissions for users to enter and use the original data interface, format conversion engine, and user collaboration layer within the independent communication space; if users do not provide selected content, the user's feedback result will be used as the actual demand.

[0099] In practice, high-frequency business demands can be considered as business demands with a demand frequency higher than a preset demand frequency threshold. The sharing platform is used to create an independent communication space for high-frequency business demands, so that users whose actual demands are successfully matched with the high-frequency business demands can enter the independent communication space, and communicate and interact online between users through the user collaboration layer. That is, users who put forward the same high-frequency business demand can share and interact with each other in the independent communication space. The independent communication space also provides a raw data interface function for establishing a communication connection with the city management system to which the business data corresponding to the corresponding high-frequency business demand belongs, so that users can directly retrieve the raw data of the business data contained in the corresponding high-frequency business demand in the independent communication space, and the format conversion engine is used to convert the retrieved raw data into business data in a unified format.

[0100] The present application also discloses a smart city data integration and sharing platform. Figure 2 ,include:

[0101] The business demand prediction module 201 is configured to acquire business data generated by all city management systems in real time, predict business demand based on the acquired business data, and generate constraints and corresponding minimum transformation sets based on the business demand; wherein the constraints include at least time conditions and / or space conditions, and the minimum transformation set includes data parameters from different city management systems that satisfy the corresponding constraints;

[0102] The prediction result storage module 202 is used to extract the specific business data corresponding to the data parameters based on the minimum transformation set, integrate the business data, and form the business demand and its corresponding integrated business data into demand content for storage;

[0103] The actual demand response module 203 is used to match the actual demand with the stored demand content when receiving the actual demand proposed by the user. If the match is successful, the matched demand content is output; if the match is unsuccessful, the corresponding business data is determined based on the actual demand and integrated and output.

[0104] Optionally, the prediction result storage module 202 is also used to use the integrated and processed business data as data to be activated, establish an association relationship between the data to be activated and the business needs; determine the demand frequency and change frequency of the data to be activated, and store the data to be activated whose demand frequency is higher than a preset demand frequency threshold and whose change frequency is lower than a preset change frequency threshold as first data on the shared platform; for all data to be activated that are not first data, use them as second data, and store the second data in a preset edge cache area corresponding to the city management system to which the second data belongs, according to the city management system to which the second data belongs, and the edge cache area includes a dynamic cache area and a static cache area, and the second data with a change frequency higher than the preset change frequency threshold is stored in the dynamic cache area, and the second data with a change frequency not higher than the preset change frequency threshold is stored in the static cache area; it is also used to configure processing engines for the second data associated with different business needs in the edge cache area according to the different business needs associated with the second data stored in the edge cache area, and make the processing engines correspond one-to-one with the business needs.

[0105] The actual demand response module 203 is also used to determine the storage location of the data to be activated corresponding to the successfully matched demand content if the match is successful. If it is stored in the shared platform, the data to be activated corresponding to the successfully matched demand content is output; if it is stored in the edge cache area, the data to be activated corresponding to the successfully matched demand content is retrieved and transmitted to the shared platform through the corresponding processing engine for output by the shared platform.

[0106] Optionally, the prediction result storage module 202 is also used to perform initial sharding processing on the integrated business data according to preset spatial dimensions based on corresponding constraints, and then perform secondary sharding processing on the business data after the initial sharding processing according to a preset business cycle, and use the data sets of the business data contained in each layer after the secondary sharding processing as data to be activated.

[0107] Optionally, the prediction result storage module 202 is also used to take the second data with a change frequency higher than a preset change frequency threshold as the third data, and predict the demand time of the business demand corresponding to the third data in a future time period, and the demand time is used to represent the time when the corresponding actual demand is proposed by the user when the corresponding business demand successfully matches the actual demand proposed by the user; according to the demand time corresponding to the third data, the control processing engine retrieves the original data from the city management system to which the third data belongs at a specified time point before the demand time, converts it into the third data, and then stores it in the corresponding dynamic cache area.

[0108] Optionally, the prediction result storage module 202 is also used to determine all target business demands associated with the second data in the cache area, and calculate the combined density value of each target business demand based on preset reference dimensions, wherein the reference dimensions include at least the city management system dimension to which the data belongs, the fine-grained dimension of the data, and the cache dimension to which the data belongs; according to the combined density value, the density level and the corresponding processing engine type of each target business demand are determined, and a processing engine corresponding to the determined processing engine type is configured for each target business demand; wherein the density level is used to characterize the processing complexity of the business demand.

[0109] Optionally, it also includes a regular automatic push module, which is used to analyze the changes in business data corresponding to each business demand and predict the changing trends in future time periods. Based on the prediction results, the recommended monitoring frequency is determined and output to the user. The recommended monitoring frequency is used to characterize the frequency with which users propose corresponding business demands; when the user's authorized push instruction is received, the business data corresponding to the business demand is output to the user according to the recommended monitoring frequency corresponding to the business demand contained in the authorized push instruction.

[0110] Optionally, it further includes a high-frequency business independent communication module for updating and determining high-frequency business needs in real time, and creating an independent communication space including a raw data interface, a format conversion engine, and a user collaboration layer for each high-frequency business need; wherein the high-frequency business need refers to a business need that has successfully matched actual needs raised at different times or by different users more than a preset number of times, the independent communication space is set in a one-to-one correspondence with the business need, the user collaboration layer is used for all users who raise corresponding business needs to achieve communication and data sharing, the raw data interface is used for users to retrieve raw data corresponding to the corresponding business need from the corresponding city management system, and the format conversion engine is used for users to convert the raw data into a specified format;

[0111] The actual demand response module 203 is also used to receive demand instructions proposed by users, display all independent communication spaces and corresponding business needs, and when the user feedback contains selected content with any independent communication space, configure the user with permissions to enter and use the original data interface, format conversion engine and user collaboration layer in the independent communication space; if the user does not feedback the selected content, the user's feedback result will be used as the actual demand.

[0112] An embodiment of the present application also discloses a smart city data integration and sharing device, which includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and execute the smart city data integration and sharing method as described above.

[0113] An embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program that can be loaded by a processor and execute the smart city data integration and sharing method as described above. The computer-readable storage medium includes, for example: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0114] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0115] The above embodiments are intended only to illustrate the technical solutions of this application and are not intended to limit the scope of protection of this application. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on these embodiments, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are also within the scope of protection to be protected by this application.

Claims

1. A smart city data integration and sharing method, characterized in that: include: Acquire business data generated by all city management systems in real time, predict business needs based on the real-time business data, and generate constraints and corresponding minimum transformation sets based on the business needs; wherein the constraints include at least time conditions and space conditions, and the minimum transformation set includes data parameters from different city management systems that meet the corresponding constraints; Based on the minimum conversion set, extract the specific business data corresponding to the data parameters, integrate the business data, and form the business requirements and the corresponding integrated business data into demand content for storage; When receiving the actual needs proposed by the user, the actual needs are matched with the stored needs content first. If the match is successful, the matched needs content is output; if the match is unsuccessful, the corresponding business data is determined based on the actual needs and integrated and output; Generating the constraint conditions and the corresponding minimum transformation set according to the business requirements includes: Inputting business data from different city management systems into a preset combination pattern mining algorithm so that the combination pattern mining algorithm outputs a minimum transformation set and corresponding constraint conditions, wherein the minimum transformation set includes data parameters from more than one city management system, and the constraint conditions are conditions for limiting the numerical range of the data parameters in the minimum transformation set. The constraint conditions specifically include spatial conditions and temporal conditions. The spatial conditions require that the specific values ​​of the data parameters in the minimum transformation set are within a fixed geographical area, and the corresponding temporal conditions require that the specific values ​​of the data parameters in the minimum transformation set are data values ​​generated within a specified time period. The combined pattern mining algorithm is specifically an improved FP-Growth algorithm, which is used to convert business data into a spatiotemporal transaction set through sliding windows and spatial clustering, wherein data items within the same transaction must satisfy a spatial distance ≤ a preset distance threshold, and a time interval ≤ a preset maximum time interval; by constructing an FP-Tree, and during the construction, only inserting data items that satisfy a preset spatiotemporal co-occurrence constraint condition with the current path node, and finally mining a frequent item set that satisfies a preset minimum support from the FP-Tree as a minimum transformation set, with the associated spatiotemporal range as a constraint condition.

2. The smart city data integration and sharing method according to claim 1 is characterized in that: The step of forming the business demand and the corresponding integrated business data into demand content for storage includes: The integrated and processed business data is used as data to be activated, and the association relationship between the data to be activated and the business needs is established; Determining the demand frequency and change frequency of the data to be activated, and storing the data to be activated whose demand frequency is higher than a preset demand frequency threshold and whose change frequency is lower than the preset change frequency threshold as first data on the sharing platform; For all data to be activated other than the first data, as the second data, based on the city management system to which the second data belongs, the second data is stored in a preset edge cache area corresponding to the city management system to which the second data belongs, and the edge cache area includes a dynamic cache area and a static cache area, and the second data with a change frequency higher than a preset change frequency threshold is stored in the dynamic cache area, and the second data with a change frequency not higher than the preset change frequency threshold is stored in the static cache area; According to different business requirements associated with the second data stored in the edge cache, respectively configure processing engines for the second data associated with different business requirements in the edge cache, and make the processing engines correspond to the business requirements one by one; If the match is successful, the content of the successfully matched requirement is output, including: If the match is successful, the storage location of the data to be activated corresponding to the successfully matched demand content is determined. If it is stored in the shared platform, the data to be activated corresponding to the successfully matched demand content is output; if it is stored in the edge cache area, the data to be activated corresponding to the successfully matched demand content is retrieved and transmitted to the shared platform through the corresponding processing engine for output by the shared platform.

3. The smart city data integration and sharing method according to claim 2 is characterized in that: The process of using the integrated and processed business data as data to be activated includes: For the integrated business data, based on the corresponding constraints, the business data is initially sharded according to the preset spatial dimension, and then the business data after the initial sharding is secondary sharded according to the preset business cycle. The data sets of the business data contained in each layer after the secondary sharding are respectively used as data to be activated.

4. The smart city data integration and sharing method according to claim 2, characterized in that: The forming of the business requirements and the corresponding integrated business data into demand content for storage also includes: Using the second data whose change frequency is higher than a preset change frequency threshold as the third data, predicting the demand time of the business demand corresponding to the third data in a future time period, where the demand time is used to represent the time when the corresponding actual demand is proposed by the user when the corresponding business demand is successfully matched with the actual demand proposed by the user; According to the demand time corresponding to the third data, the control processing engine retrieves the original data from the city management system to which the third data belongs at a specified time point before the demand time, converts the data into the third data, and then stores it in the corresponding dynamic cache area.

5. The smart city data integration and sharing method according to claim 2, characterized in that: The configuring of processing engines for the second data associated with different business requirements in the edge cache area includes: Determine all target business requirements associated with the second data within the cache area, and calculate a combined density value of each target business requirement based on preset reference dimensions, where the reference dimensions include at least a city management system dimension to which the data belongs, a fine-grained dimension of the data, and a cache dimension to which the data belongs; Based on the combined density value, the density level and corresponding processing engine type of each target business requirement are determined, and a processing engine corresponding to the determined processing engine type is configured for each target business requirement; wherein, the density level is used to characterize the processing complexity of the business requirement.

6. The smart city data integration and sharing method according to claim 2 is characterized in that: The method further comprises: Analyze the changes in business data corresponding to each business requirement and predict the change trend in the future period. Based on the prediction results, determine and output the recommended monitoring frequency to the user. The recommended monitoring frequency is used to represent the frequency of the corresponding business requirements proposed by the user; When an authorized push instruction from the user is received, the service data corresponding to the service requirement is output to the user according to the recommended monitoring frequency corresponding to the service requirement contained in the authorized push instruction.

7. The smart city data integration and sharing method according to claim 1, characterized in that: The method further comprises: High-frequency business needs are updated and determined in real time, and an independent communication space is created for each high-frequency business need, including a raw data interface, a format conversion engine, and a user collaboration layer. A high-frequency business need refers to a business need that has successfully matched actual needs raised at different times or by different users more than a preset number of times. The independent communication space is set up in a one-to-one correspondence with the business need. The user collaboration layer is used to enable communication and data sharing among all users who have raised the corresponding business need. The raw data interface is used for users to retrieve raw data corresponding to the corresponding business need from the corresponding city management system. The format conversion engine is used for users to convert the raw data into a specified format. When the actual demand proposed by the user is received, the following steps are included: Receive demand instructions from users, display all independent communication spaces and corresponding business needs, and when users provide selected content with any independent communication space, configure permissions for users to enter and use the original data interface, format conversion engine, and user collaboration layer within the independent communication space; if users do not provide selected content, the user's feedback result will be used as the actual demand.

8. A smart city data integration and sharing platform, characterized by: include, A business demand prediction module (201) is used to obtain business data generated by all city management systems in real time, and predict business needs based on the business data obtained in real time, and generate constraint conditions and corresponding minimum transformation sets according to the business needs; wherein the constraint conditions include at least time conditions and space conditions, and the minimum transformation set includes data parameters that meet the corresponding constraint conditions and are derived from different city management systems; A prediction result storage module (202) is used to extract specific business data corresponding to the data parameters based on the minimum conversion set, integrate the business data, and form demand content from the business demand and its corresponding integrated business data for storage; The actual demand response module (203) is used to, when receiving the actual demand proposed by the user, give priority to matching the actual demand with the stored demand content, and if the match is successful, output the successfully matched demand content; if the match is unsuccessful, determine the corresponding business data based on the actual demand and output it after integration; The business demand prediction module (201) is also used to input business data from different city management systems into a preset combination pattern mining algorithm, so that the combination pattern mining algorithm outputs a minimum transformation set and corresponding constraints, wherein the minimum transformation set includes data parameters from more than one city management system, and the constraints are conditions for limiting the numerical range of the data parameters in the minimum transformation set. The constraints specifically include spatial conditions and temporal conditions. The spatial conditions require that the specific values ​​of the data parameters in the minimum transformation set are within a fixed geographical area, and the corresponding temporal conditions require that the specific values ​​of the data parameters in the minimum transformation set are data values ​​generated within a specified time period. The combination pattern mining algorithm is specifically an improved FP-Growth algorithm, which is used to convert business data into a spatiotemporal transaction set through sliding windows and spatial clustering, wherein data items in the same transaction must meet the spatial distance ≤ a preset distance threshold, and the time interval ≤ a preset maximum time interval; by constructing an FP-Tree, and when constructing, only inserting data items that meet the preset spatiotemporal co-occurrence constraints with the current path node, and finally mining a frequent item set that meets the preset minimum support from the FP-Tree as the minimum transformation set, and taking the associated spatiotemporal range as the constraint.

9. A smart city data integration and sharing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images