Smart city data integration and sharing method, platform, equipment and storage medium

Abstract

The invention relates to a smart city data integration and sharing method, platform and device and a storage medium, and belongs to the technical field of smart city information, and the method comprises the steps: obtaining business data generated by all city management systems in real time, carrying out the prediction based on the business data obtained in real time, and obtaining a business demand; generating constraint conditions and corresponding minimum conversion sets according to business requirements; on the basis of the minimum conversion set, extracting specific business data corresponding to the data parameters, performing integration processing on the business data, and forming the business requirements and the corresponding integrated business data into requirement contents for storage; when an actual demand proposed by a user is received, matching the actual demand with the stored demand content, and if the matching is successful, outputting the successfully matched demand content; and if the matching is unsuccessful, determining the corresponding business data based on the actual demand, integrating the business data and outputting the integrated business data. The method has the advantages that data sharing is achieved, and meanwhile the computing power burden consumed by data conversion and integration is reduced.

Description

Technical Field

[0001] The present application relates to the field of smart city information technology, and in particular to a smart city data integration and sharing method, platform, device and storage medium. Background Art

[0002] Smart cities improve urban operation efficiency and public service levels by integrating information technology and communication technology. At present, the construction of smart cities has widely adopted cloud computing, Internet of Things, big data and other technologies for data collection, transmission and processing. However, the current smart city management systems mostly adopt decentralized data management methods, and there is poor data interoperability between systems, which makes it difficult to effectively use data. In this regard, in order to meet the needs of modern smart city management and realize cross-system data sharing, the existing technology proposes to use data conversion methods to collect and standardize the data generated by all smart city management systems to unify the data format, thereby breaking the information island and realizing data integration between systems, so that managers can obtain cross-system data.

[0003] However, based on the above technical solutions, it can be seen that since the data generated by the smart city management system may be generated in real time, such as the traffic management system, meteorological management system, etc., it is easy to cause a large amount of data to be converted and integrated, thereby increasing the computing burden, so it needs to be improved. Summary of the invention

[0004] In order to reduce the computing power burden consumed by data conversion and integration while meeting users' needs for data sharing, the present application provides a smart city data integration and sharing method, platform, device and storage medium.

[0005] In a first aspect, the present application provides a smart city data integration and sharing method, comprising: Acquire the business data generated by all city management systems in real time, predict business needs based on the business data acquired in real time, and generate constraint conditions and corresponding minimum transformation sets according to the business needs; wherein the constraint conditions at least include time conditions and space conditions, and the minimum transformation set includes data parameters that meet the corresponding constraint conditions and originate from different city management systems; Based on the minimum conversion set, the specific business data corresponding to the data parameters are extracted, and the business data are integrated and processed, and the business requirements and the corresponding integrated business data are formed into demand content for storage; When receiving the actual needs proposed by the user, the actual needs are matched with the stored demand content first. If the match is successful, the matched demand content is output; if the match is unsuccessful, the corresponding business data is determined based on the actual needs and integrated and output.

[0006] By adopting the above technical solution, for the process of collecting, transforming, integrating and processing data of all urban management systems, the present application proposes: by predicting the scope boundary of business requirements (i.e., constraint conditions), determining precise local data (i.e., the smallest transformation set that meets the constraint conditions), and then only storing the transformed local data. When the user puts forward actual requirements, the requirement content that matches the actual requirements is preferentially screened out from the stored requirement content. On the one hand, it realizes early transformation and improves the demand response efficiency. On the other hand, only precise transformation of local data is realized, rather than transformation and integration of global data (i.e., unified transformation and integration of data of all urban management systems), so as to reduce the computing power burden.

[0007] Optionally, storing the business requirements and the corresponding integrated and processed business data as requirement content includes: Taking the integrated and processed business data as data to be activated, and establishing an association relationship between the data to be activated and the business requirements; Judging the required frequency and change frequency of the data to be activated, and storing the data to be activated with a required frequency higher than the preset required frequency threshold and a change frequency lower than the preset change frequency threshold as first data in the shared platform; Regarding all the data to be activated that are not the first data as second data, and storing the second data in the corresponding preset edge buffer area of the urban management system to which the second data belongs. The edge buffer area includes a dynamic buffer area and a static buffer area. Storing the second data with a change frequency higher than the preset change frequency threshold in the dynamic buffer area, and storing the second data with a change frequency not higher than the preset change frequency threshold in the static buffer area; According to the different business requirements associated with the second data stored in the edge buffer area, configuring processing engines for the second data associated with different business requirements in the edge buffer area respectively, and making the processing engines correspond to the business requirements one by one; The step of, if the matching is successful, outputting the matching successful requirement content includes: If the matching is successful, determining the storage location of the data to be activated corresponding to the matching successful requirement content. If it is stored in the shared platform, outputting the data to be activated corresponding to the matching successful requirement content; if it is stored in the edge buffer area, calling and transmitting the data to be activated corresponding to the matching successful requirement content to the shared platform through the corresponding processing engine for the shared platform to output.

[0008] By adopting the above technical solution, the integrated business data is partitioned and stored to reduce the memory occupancy of the sharing platform. Meanwhile, a processing engine is configured for the data stored in the edge buffer to realize the interaction with the sharing platform by means of the processing engine, that is, the data in the buffer is retrieved and transmitted to the sharing platform.

[0009] Optionally, taking the integrated business data as the data to be activated includes: For the integrated business data, based on the corresponding constraint conditions, the business data is first sliced according to a preset spatial dimension, and then the sliced business data is sliced again according to a preset business cycle. The data sets of the business data included in each layer after the secondary slicing are respectively used as the data to be activated.

[0010] By adopting the above technical solution, after the business data is integrated and before being partitioned and stored, a two-dimensional slicing process is performed based on a preset spatial dimension (i.e., geographical location) and a preset business cycle dimension (i.e., the demand for corresponding data at different times), and then the storage location is determined for each sliced data, improving the refinement and accuracy of determining the data caching location.

[0011] Optionally, storing the business requirements and their corresponding integrated business data as demand content further includes: Taking the second data with a change frequency higher than a preset change frequency threshold as the third data, predicting the demand time of the business requirements corresponding to the third data in a future time period, where the demand time is used to represent the time when the corresponding actual demand is proposed by the user when the corresponding business requirement matches the actual demand proposed by the user; According to the demand time corresponding to the third data, controlling the processing engine to retrieve the original data from the urban management system to which the third data belongs at a specified time point before the demand time, convert the original data to form the third data, and store it in the corresponding dynamic buffer.

[0012] By adopting the above technical solution, for the third data with a change frequency higher than a preset change frequency threshold, this application proposes: predicting the demand time of the third data, that is, predicting the time when the user has a demand for the third data, and only retrieving the original data from the corresponding urban management system at a specified time point before the demand time, converting the original data to form the third data, and storing it in the dynamic buffer. This solution can, on the one hand, reduce the real-time memory occupancy in the dynamic buffer, and on the other hand, reduce the processing frequency of the processing engine for the third data, reducing the unnecessary computing power burden on the processing engine.

[0013] Optionally, processing engines are respectively configured for second data associated with different service requirements in the edge buffer area, including: Determine all target service requirements associated with the second data in the buffer area, and calculate the combined density value of each target service requirement based on a preset reference dimension, where the reference dimension at least includes the urban management system dimension to which the data belongs, the data granularity dimension, and the buffer dimension to which the data belongs; According to the combined density value, determine the density level and the corresponding processing engine type of each target service requirement, and configure a processing engine corresponding to the determined processing engine type for each target service requirement; where the density level is used to characterize the processing complexity of the service requirement.

[0014] By adopting the above technical solution, when configuring the processing engine, the processing complexity of the corresponding service requirement will be determined according to the combined density value of the service requirement (i.e., the target service requirement) associated with the second data, and then the processing engine type will be targeted according to the processing complexity to adapt to the efficient processing of the service requirement with processing complexity.

[0015] Optionally, the method further includes: Analyze the change situation of the service data corresponding to each service requirement, predict the change trend in the future period, and based on the prediction result, determine and output a recommended monitoring frequency to the user, where the recommended monitoring frequency is used to characterize the frequency at which the user proposes the corresponding service requirement; When receiving the authorized push instruction of the user, output the service data corresponding to the service requirement according to the recommended monitoring frequency corresponding to the service requirement included in the authorized push instruction to the user.

[0016] By adopting the above technical solution, this solution reduces the manual request volume of the user and improves the timeliness of discovering the change situation of the corresponding service requirement at the same time.

[0017] Optionally, the method further includes: Determine and update high-frequency service requirements in real time, and create an independent communication space including an original data interface, a format conversion engine, and a user collaboration layer for each high-frequency service requirement; where the high-frequency service requirement refers to a service requirement whose number of successful matches with the actual requirements proposed at different times or by different users exceeds a preset number, the independent communication space is set in one-to-one correspondence with the service requirement, the user collaboration layer is used for all users who propose the corresponding service requirement to achieve communication and data sharing, the original data interface is used for the user to retrieve the original data corresponding to the corresponding service requirement from the corresponding urban management system, and the format conversion engine is used for the user to convert the original data into a specified format; When receiving the actual requirement proposed by the user, it includes: Receive the requirement instructions proposed by the user, display all independent communication spaces and the corresponding business requirements. When the user feedbacks the selection content with any independent communication space, configure the permissions for the user to access and use the original data interface, format conversion engine, and user collaboration layer within the independent communication space; if the user does not feedback the selection content, then use the feedback result of the user as the actual requirement.

[0018] By adopting the above technical solution, an independent communication space is separately created for high-frequency business requirements to serve the users who propose such business requirements. The independent communication space is independently set relative to the shared platform, and users can directly retrieve the original data corresponding to the business requirements corresponding to this communication space, and can also convert the original data by themselves, thus eliminating the need for the shared platform to match the business requirements proposed by the user each time, and instead allowing the user to directly enter the corresponding independent communication space.

[0019] In a second aspect, the present application provides a smart city data integration and sharing platform, including, A business requirement prediction module, which is used to obtain in real time all the business data generated by the urban management systems, predict the business requirements based on the real-time obtained business data, and generate constraint conditions and corresponding minimum conversion sets according to the business requirements; wherein, the constraint conditions at least include time conditions and space conditions, and the minimum conversion set contains data parameters that meet the corresponding constraint conditions and originate from different urban management systems; A prediction result storage module, which is used to extract the specific business data corresponding to the data parameters based on the minimum conversion set, perform integration processing on the business data, and store the business requirements and the corresponding integrated business data as demand content; An actual requirement response module, which is used to, when receiving the actual requirements proposed by the user, first match the actual requirements with the stored demand content. If the match is successful, output the demand content with a successful match; if the match is not successful, determine the corresponding business data based on the actual requirements and output it after integration.

[0020] In a third aspect, the present application provides a smart city data integration and sharing device, including a memory and a processor, and a computer program capable of being loaded and executed by the processor as described in any one of the methods in the first aspect is stored on the memory.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, storing a computer program capable of being loaded and executed by the processor as described in any one of the methods in the first aspect.

[0022] In summary, the present application includes at least one of the following beneficial technical effects: For the process of collecting, transforming, integrating, and processing the data of all urban management systems, this application proposes: by predicting the scope boundary of business requirements (i.e., constraints), determining precise local data (i.e., the smallest transformation set that meets the constraints), and then only storing the transformed local data. When the user puts forward actual requirements, first screen out the requirement content that matches the actual requirements from the stored requirement content. On the one hand, it realizes early transformation and improves the demand response efficiency. On the other hand, only precise transformation of local data is achieved, rather than transforming and integrating global data (i.e., unifying the transformation and integration of the data of all urban management systems), so as to reduce the computing power burden; Furthermore, the processed and integrated business data is stored in partitions to reduce the memory occupancy of the shared platform. At the same time, a processing engine is configured for the data stored in the edge buffer area to be used to interact with the shared platform through the processing engine, that is, to retrieve and transfer the data in the buffer area to the shared platform; Even further, after integrating and processing the business data and before partitioning and storing it, perform two-dimensional sharding based on a preset spatial dimension (i.e., geographical location) and a preset business cycle dimension (i.e., the demand for corresponding data at different times), and then determine the storage location for each shard of data respectively, improving the refinement and precision of determining the data cache location. Brief Description of the Drawings

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 It is a flowchart showing a method for integrating and sharing smart city data disclosed in an embodiment of this application.

[0025] Figure 2 It is a block diagram showing the structure of a smart city data integration and sharing platform disclosed in an embodiment of this application.

[0026] Explanation of Reference Numerals: 201, Business Requirement Prediction Module; 202, Prediction Result Storage Module; 203, Actual Requirement Response Module. Detailed Embodiment

[0027] The following is a further detailed description of this application in combination with the attached Figure 1-2 to further illustrate this application in detail.

[0028] The embodiments of this application disclose a method for integrating and sharing smart city data (hereinafter referred to as the sharing method), which aims to aggregate and integrate the data generated by all urban management systems, and when a user puts forward an actual demand (that is, when putting forward a demand for accessing data), feedback the corresponding data to the user to achieve cross-system data sharing and break information barriers. At the same time, this application will only process and integrate the data related to the demand according to the user's demand, rather than uniformly converting and integrating the global data, so as to reduce the computing burden and resource consumption. The execution entity of the sharing method of this application is a smart city data integration and sharing platform (hereinafter referred to as the sharing platform), and the execution process of the sharing platform for the sharing method will be specifically described below in combination with Figure 1 , and the execution process of the sharing platform for the sharing method will be specifically elaborated.

[0029] S101, obtain the business data generated by all urban management systems in real time, predict the business requirements based on the real-time obtained business data, and generate constraint conditions and corresponding minimum conversion sets according to the business requirements; among them, the constraint conditions at least include time conditions and space conditions, and the minimum conversion set contains data parameters that meet the corresponding constraint conditions and originate from different urban management systems.

[0030] In implementation, the urban management system can specifically include an intelligent transportation management system, an underground pipeline detection system, a medical emergency dispatch system, an education resource management system, a meteorological monitoring system, etc. The data of all urban management systems are accessed to the sharing platform through a unified adapter that supports four access modes: API, message queue, file transfer, and database log. For example, Apache Kafka + Schema Registry access technology is used to access real-time stream data; Apache Spark Delta Lake access technology is used to access batch data; Geomesa + GeoMesa Kafka access technology is used to access spatial data, and finally the sharing platform obtains the business data generated by all urban management systems, and the business data obtained at this time is the original data collected by the urban management system.

[0031] The sharing platform will analyze the above-mentioned originally obtained data in real time. Specifically, the originally obtained data from different urban management systems will be input into a preset combined pattern mining algorithm, so that the combined pattern mining algorithm outputs a data parameter combination (i.e., the minimum transformation set) and the corresponding constraint conditions. The minimum transformation set here contains data parameters from more than one urban management system. For example, the minimum transformation set contains data parameter a and data parameter b, where data parameter a comes from urban management system A and data parameter b comes from urban management system B. Exemplarily, the minimum data set can be: hourly rainfall, drainage network load, traffic congestion index. It can be seen that the 3 parameters included in this minimum data set come from the meteorological monitoring system, the water service system, and the intelligent transportation management system respectively.

[0032] The constraint condition is a condition used to limit the numerical range of the data parameters in the minimum transformation set, such as including spatial conditions and time conditions. The spatial condition requires that the specific numerical values of the data parameters in the minimum transformation set are within a fixed geographical area range. Correspondingly, the time condition requires that the specific numerical values of the data parameters in the minimum transformation set are data values generated within a specified time period. Moreover, the originally obtained data input into the combined pattern mining algorithm contains not only specific data values, but also geographical locations (such as geographical coordinates) and data generation times (such as timestamps).

[0033] The combined pattern mining algorithm is specifically an improved FP-Growth algorithm. Specifically, the originally obtained data is converted into a spatio-temporal transaction set through a sliding window and spatial clustering (such as using the density-based DBSCAN algorithm with preset distance threshold as parameters). Among them, the data items within the same thing need to satisfy the spatial distance ≤ preset distance threshold (such as 1 km) and the time interval ≤ preset maximum time interval (such as 1 hour). Then, an FP-Tree is constructed. When constructing it, insertion is only allowed when the candidate data line and the current path node meet the spatio-temporal co-occurrence constraint condition (the spatio-temporal co-occurrence constraint condition accelerates the calculation of spatial relationships through an R-tree index). The frequent item set (i.e., the minimum transformation set) that meets the preset minimum support degree is mined from the FP-Tree, and its associated spatio-temporal range (i.e., the constraint condition) is recorded.

[0034] Correspondingly, the minimum transformation set and the constraint conditions are the specific manifestation forms of the business requirements. An ID number can be generated for the business requirements, and the association relationship between the business requirements, the minimum data set, and the constraint conditions can be established.

[0035] S102. Based on the minimum transformation set, extract the specific business data corresponding to the data parameters, and perform integration processing on the business data. Store the business requirements and the corresponding integrated business data as demand content.

[0036] S103. When receiving the actual requirements proposed by the user, first match the actual requirements with the stored requirement content. If the match is successful, output the requirement content with a successful match; if the match is unsuccessful, determine the corresponding business data based on the actual requirements, integrate it, and then output.

[0037] In implementation, the sharing platform is used to access from the urban management system the specific numerical values (i.e., business data) corresponding to the specific data parameters included in the minimum conversion set that meet the constraint conditions, and use a preset integration algorithm to convert the formats of the business data within the same minimum conversion set to achieve unity. The conversion process specifically includes field mapping of structured data, unit unification, and coordinate conversion of spatial data; since the conversion and standardization of data formats are existing technologies, they will not be elaborated here.

[0038] The sharing platform is used to merge the integrated business data and its corresponding business requirements to form requirement content and store it. It is also used to receive the actual requirements proposed by the user in real time. Here, it should be explained that the user can access the sharing platform in the form of a web page and enter the combination of data parameters and constraint conditions required for viewing in a preset input box on the access interface of the sharing platform, so as to form actual requirements to trigger the proposal of actual requirements. The sharing platform is used to receive the actual requirements, and then compare the combination of data parameters included in the actual requirements with each minimum conversion set in all the stored requirement content one by one. If the comparison is consistent (that is, the data parameters included in the combination of data parameters are exactly the same as the data parameters included in the minimum conversion set, and the constrained conditions corresponding to the actual requirements are included in the constrained conditions corresponding to the requirement content), it means that the match is successful, and output the business data that meets the constrained conditions in the requirement content with a successful match. If the match is unsuccessful, according to the combination of data parameters included in the actual requirements and its corresponding constrained conditions, retrieve the original data corresponding to each data parameter that meets the constrained conditions in the combination of data parameters from the corresponding urban system, and then output it after conversion and unification to form business data.

[0039] Optionally, "forming requirement content by storing the business requirements and the corresponding integrated business data" in S102 includes: For the integrated business data, based on the corresponding constrained conditions, perform primary slicing processing on the business data according to a preset spatial dimension, and then perform secondary slicing processing on the business data after primary slicing processing according to a preset business cycle. Use the data set of the business data included in each slice after secondary slicing processing as the data to be activated respectively; establish an association relationship between the data to be activated and the business requirements.

[0040] Determine the required frequency and change frequency of the data to be activated, and store the data to be activated with a required frequency higher than the preset required frequency threshold and a change frequency lower than the preset change frequency threshold as the first data in the shared platform.

[0041] All the data to be activated that is not the first data is used as the second data. According to the urban management system to which the second data belongs, store the second data in the corresponding preset edge buffer area of the urban management system to which it belongs. And the edge buffer area includes a dynamic buffer area and a static buffer area. Store the second data with a change frequency higher than the preset change frequency threshold in the dynamic buffer area, and store the second data with a change frequency not higher than the preset change frequency threshold in the static buffer area.

[0042] According to the different business requirements associated with the second data stored in the buffer area, determine all the target business requirements associated with the second data in the buffer area. Based on the preset reference dimensions, calculate the combined density value of each target business requirement. Among them, the reference dimensions at least include the dimension of the urban management system to which the data belongs, the data granularity dimension, and the buffer dimension to which the data belongs.

[0043] According to the combined density value, determine the density level and the corresponding processing engine type of each target business requirement, and configure a processing engine corresponding to the determined processing engine type for each target business requirement; and make the processing engine correspond to the business requirement one by one; among them, the density level is used to characterize the processing complexity of the business requirement.

[0044] Use the second data with a change frequency higher than the preset change frequency threshold as the third data, and predict the required time of the business requirement corresponding to the third data in the future time period. The required time is used to characterize the time when the corresponding actual requirement is proposed by the user when the corresponding business requirement matches the actual requirement proposed by the user.

[0045] According to the required time corresponding to the third data, control the processing engine to retrieve the original data from the urban management system to which the third data belongs at a specified time point before the required time, convert it into the third data, and then store it in the corresponding dynamic buffer area.

[0046] And the "if the match is successful, output the required content of the successful match" in S103 further includes the following sub-steps: If the match is successful, determine the storage location of the data to be activated corresponding to the required content of the successful match. If it is stored in the shared platform, output the data to be activated corresponding to the required content of the successful match; if it is stored in the buffer area, retrieve and transmit the data to be activated corresponding to the required content of the successful match to the shared platform through the corresponding processing engine for the shared platform to output.

[0047] In implementation, the sharing platform pre-stores the demand heat index of each data parameter generated at different geographical locations in the city and at different time periods corresponding to each geographical location. The demand heat index is used to characterize the frequency of the corresponding data parameter being accessed by users at the corresponding geographical location and during the corresponding time period. For the integrated business data, the sharing platform determines the range of its demand heat index according to the spatial range and time range included in the corresponding constraint conditions, and then divides the determined range of the demand heat index into several sub-ranges of the demand heat index. According to the sub-ranges of the demand heat index, the spatial range and time range included in the constraint conditions are divided respectively, that is, spatial sub-ranges and time sub-ranges that meet the sub-ranges of the demand heat index are formed. The spatial sub-ranges and time sub-ranges are formed into constraint sub-conditions. Finally, the business data is fragmented to form data to be activated, so that the data to be activated corresponds one by one to the constraint sub-conditions. The data to be activated is the data to be activated that meets the corresponding constraint sub-conditions. Finally, according to the business data to which the data to be activated belongs, an association relationship between the data to be activated and the business requirements is established.

[0048] Next, the sharing platform is used to judge the demand frequency and change frequency of the data to be activated. Here, the demand frequency can be considered as the corresponding sub-range of the demand index. And if the difference between the original data corresponding to adjacent timestamps is greater than the preset difference, it is considered that the original data has changed. The change frequency can be represented by the time difference between the timestamps of the original data before and after the change when the difference is greater than the preset difference (that is, a change occurs).

[0049] The data to be activated with a demand frequency higher than the preset demand frequency threshold and a change frequency lower than the preset change frequency threshold is stored in the sharing platform as the first data (that is, the data to be activated with high demand and high stability).

[0050] All other data to be activated that is not the first data is collectively referred to as the second data. The second data will be further split according to the urban management system to which the data parameter corresponding to the second data belongs, that is, according to the difference in the urban management system to which the corresponding data parameter belongs, the second data is stored in the preset edge buffer area of its corresponding urban management system. And it should be noted here that the preset edge buffer area of each urban management system can be specifically split into a dynamic buffer area and a static buffer area. The second data with a change frequency higher than the preset change frequency threshold is stored in the dynamic buffer area, and the second data with a change frequency not higher than the preset change frequency threshold is stored in the static buffer area.

[0051] For the service data stored in the edge cache, all the service requirements associated with the service data in all edge caches (hereinafter referred to as target service requirements) can be further determined according to the differences in the service requirements associated with the service data, and a processing engine is configured for each target service requirement, that is, the one-to-one correspondence between the processing engine and the target service requirement is realized. The specific configuration process is as follows: According to the preset reference dimensions (i.e., the urban management system dimension, the data granularity dimension, and the data cache dimension), determine the reference dimension values of each target service requirement. The dimension value corresponding to the urban management system dimension is the number of urban management systems to which the data parameters included in the target service requirement belong. For example, if the target service requirement includes data parameters of 3 different urban management systems, the corresponding urban management system dimension is 3.

[0052] The data granularity dimension is specifically used to characterize the complexity of the data parameters. The dimension data corresponding to the data granularity dimension can be obtained by weighted summation according to the service data processing time and the data volume corresponding to all data parameters included in the target service requirement. The data cache dimension is used to characterize the complexity of the storage method of all data parameters included in the target service requirement. The dimension value corresponding to the data cache dimension can be obtained by score accumulation. For example, the score corresponding to the storage method of the shared platform storage is 1, the score of the static cache area is 2, and the score of the dynamic cache area is 3. The storage method scores of the service data corresponding to each data parameter included in the target service requirement are accumulated to obtain the dimension value corresponding to the data cache dimension. Then, based on the preset weight of each parameter dimension and the dimension value of each parameter dimension, the combined density value of each target service requirement is calculated by weighted summation.

[0053] According to the pre-stored several combined density ranges, the density level corresponding to each combined density range, and the processing engine type corresponding to each density level, determine the combined density range in which each target service requirement is located, and then determine the processing engine type and the corresponding processing engine corresponding to each target service requirement, and implement the processing operations of retrieving and format conversion of the service data of the associated target service requirement according to the processing engine of the corresponding type.

[0054] Correspondingly, the processing engine types disclosed in the embodiments of the present application specifically include the following three types: 1. Lightweight engine (CPU optimized type), which can specifically be an x86 / ARM multi-core CPU and supports the AVX512 instruction set; directly reads data from the edge cache or the shared platform.

[0055] Map business data to the process address space through memory mapping (mmap), set the NUMA node affinity to ensure that CPU cores access local memory, and use the AVX-512 instruction set to process data fields in parallel.

[0056] 2. Acceleration engine (GPU+FPGA heterogeneous type). Among them, the business data in the dynamic buffer is transmitted to the FPGA through PCIe Gen4 DMA. The FPGA is used to perform real-time cleaning and format conversion on the business data in the dynamic buffer. The preprocessed data is transmitted to the GPU video memory through GPUDirect RDMA. The CUDA kernel function is started to perform parallel computing. The FPGA and the GPU exchange metadata through NVLink and establish a shared memory window to achieve sub-microsecond synchronization; a heterogeneous acceleration engine including FPGA streaming preprocessing and GPU batch computing collaboration is formed. 3. Distributed collaboration engine The business data in the dynamic buffer is sharded by Kafka according to the time dimension, and the data in the static buffer is sharded by GeoHash according to the geographical space dimension. The business data in the dynamic buffer is processed by the streaming computing engine (Flink), and the business data in the static buffer is analyzed by the batch processing engine (Spark SQL), and then the results are fused through the incremental materialized view.

[0057] In addition, the sharing platform refers to the second data in the edge buffer with a change frequency higher than the preset change frequency threshold as the third data. The sharing platform is also used to predict the demand time of the business demand corresponding to the third data in the future time period. The specific prediction method can be to refer to the demand heat index of each data parameter generated at different geographical locations in the city and at different time periods corresponding to each geographical location pre-stored in the sharing platform described above. The demand heat index is used to characterize the frequency of the corresponding data parameter being viewed by users at the corresponding geographical location and at the corresponding time period; the time when the demand heat index corresponding to the third data in the future time period is higher than the specified index is used as the demand time.

[0058] Only when the current time reaches the specified time before the demand time of the third data, the processing engine will retrieve the corresponding original data from the urban management system to which the data parameter corresponding to the third data belongs, convert it to form the third data, and then store the third data in the corresponding dynamic buffer. And it satisfies: the change frequency of the third data > (demand time - current time).

[0059] Correspondingly, the sharing platform will retrieve the data to be activated according to the storage location of the data to be activated. If the data to be activated is stored in the sharing platform, the corresponding data to be activated will be directly retrieved and output from the sharing platform. If it is stored in the edge buffer, the processing engine will retrieve the corresponding data to be activated and transmit it to the sharing platform so that the sharing platform can output it.

[0060] Optionally, the sharing method further includes the following steps: Analyze the change situation of the business data corresponding to each business requirement, and predict the change trend in the future period. Based on the prediction result, determine and output the recommended monitoring frequency to the user. The recommended monitoring frequency is used to represent the frequency at which the user proposes the corresponding business requirement; When receiving the user's authorization push instruction, output the business data corresponding to the business requirement according to the recommended monitoring frequency corresponding to the business requirement included in the authorization push instruction.

[0061] In implementation, according to the change frequency of the business data corresponding to each business requirement, according to the prediction rule of uniform change, predict the data value of the corresponding business data in the future period. Then, based on the prediction result, output the recommended monitoring frequency to the user. The recommended monitoring frequency can be the time difference between the timestamps before and after the change when the change range of the business data is greater than the preset change range (that is, the time difference between the timestamps of the original data before and after the change is greater than the preset difference and greater than the difference threshold). The user can feedback the authorization push instruction, that is, entrust the sharing platform to automatically output the latest business data corresponding to the corresponding business requirement to the user regularly according to the corresponding recommended detection frequency.

[0062] Optionally, the sharing method further includes the following steps: Determine the high-frequency business requirements in real time and update them. Create an independent communication space for each high-frequency business requirement, which includes an original data interface, a format conversion engine, and a user collaboration layer. Among them, the high-frequency business requirement refers to the business requirement whose number of successful matches with the actual requirements proposed at different times or by different users exceeds the preset number. The independent communication space is set one-to-one corresponding to the business requirement. The user collaboration layer is used for all users who propose the corresponding business requirement to achieve communication and data sharing. The original data interface is used for users to retrieve the original data corresponding to the corresponding business requirement from the corresponding urban management system. The format conversion engine is used for users to convert the original data into a specified format; The "when receiving the actual requirement proposed by the user" in S103 includes the following steps: Receive the requirement instruction proposed by the user, display all independent communication spaces and the corresponding business requirements. When the user feedbacks the selection content with any independent communication space, configure the user with the permission to enter and use the original data interface, format conversion engine, and user collaboration layer in the independent communication space; if the user does not feedback the selection content, use the user's feedback result as the actual requirement.

[0063] In implementation, high-frequency business requirements can be considered as business requirements with a requirement frequency higher than a preset required frequency threshold. The sharing platform is used to create an independent communication space for high-frequency business requirements, so that users whose proposed actual requirements match the high-frequency business requirements can enter the independent communication space to conduct online communication and interaction among users through the user collaboration layer. That is, users who propose the same high-frequency business requirement can share and interact with data among themselves within the independent communication space. Moreover, the independent communication space also provides an original data interface function for establishing a communication connection with the urban management system to which the business data corresponding to the corresponding high-frequency business requirement belongs, so that users can directly retrieve the original data of the business data included in the corresponding high-frequency business requirement within the independent communication space. The format conversion engine is used to convert the retrieved original data into business data in a unified format.

[0064] The embodiment of the present application also discloses a smart city data integration and sharing platform. Refer to Figure 2 , including: A business demand prediction module 201, configured to obtain in real time the business data generated by all urban management systems, predict business requirements based on the real-time obtained business data, and generate constraint conditions and corresponding minimum conversion sets according to the business requirements; wherein, the constraint conditions at least include time conditions and / or space conditions, and the minimum conversion set contains data parameters that meet the corresponding constraint conditions and originate from different urban management systems; A prediction result storage module 202, configured to extract the specific business data corresponding to the data parameters based on the minimum conversion set, perform integration processing on the business data, and store the business requirements and the business data after the corresponding integration processing as demand content; An actual demand response module 203, configured to, when receiving an actual demand proposed by a user, first match the actual demand with the stored demand content. If the match is successful, output the demand content with a successful match; if the match is unsuccessful, determine the corresponding business data based on the actual demand and output it after integration.

[0065] Optionally, the prediction result storage module 202 is further configured to use the integrated business data as data to be activated, establish an association relationship between the data to be activated and the business requirements; determine the demand frequency and change frequency of the data to be activated, and store the data to be activated with a demand frequency higher than the preset demand frequency threshold and a change frequency lower than the preset change frequency threshold as first data in the shared platform; for all the data to be activated that are not the first data as second data, according to the urban management system to which the second data belongs, store the second data in the corresponding preset edge buffer area of the urban management system to which it belongs, and the edge buffer area includes a dynamic buffer area and a static buffer area, store the second data with a change frequency higher than the preset change frequency threshold in the dynamic buffer area, and store the second data with a change frequency not higher than the preset change frequency threshold in the static buffer area; and is further configured to, according to the different business requirements associated with the second data stored in the edge buffer area, configure processing engines for the second data associated with different business requirements in the edge buffer area respectively, and make the processing engines correspond to the business requirements one by one.

[0066] The actual demand response module 203 is further configured to, if the matching is successful, determine the storage location of the data to be activated corresponding to the matching successful demand content. If it is stored in the shared platform, output the data to be activated corresponding to the matching successful demand content; if it is stored in the edge buffer area, retrieve and transfer the data to be activated corresponding to the matching successful demand content to the shared platform through the corresponding processing engine for the shared platform to output.

[0067] Optionally, the prediction result storage module 202 is further configured to, for the integrated business data, on the basis of the corresponding constraint conditions, perform primary sharding processing on the business data according to the preset spatial dimension, and then perform secondary sharding processing on the business data after the primary sharding processing according to the preset business cycle, and use the data set of the business data included in each layer after the secondary sharding processing as the data to be activated respectively.

[0068] Optionally, the prediction result storage module 202 is further configured to use the second data with a change frequency higher than the preset change frequency threshold as third data, and predict the demand time of the business requirements corresponding to the third data in the future time period, where the demand time is used to represent the time when the corresponding actual demand is proposed by the user when the corresponding business requirement matches the actual demand proposed by the user; according to the demand time corresponding to the third data, control the processing engine to retrieve the original data from the urban management system to which the third data belongs at a specified time point before the demand time, convert it into the third data, and store it in the corresponding dynamic buffer area.

[0069] Optionally, the prediction result storage module 202 is further configured to determine all target business requirements associated with the second data in the buffer, calculate the combined density value of each target business requirement based on a preset reference dimension, where the reference dimension at least includes the dimension of the urban management system to which the data belongs, the data granularity dimension, and the buffer dimension to which the data belongs; determine the density level and the corresponding processing engine type of each target business requirement according to the combined density value, and configure a processing engine corresponding to the determined processing engine type for each target business requirement; where the density level is used to characterize the processing complexity of the business requirement.

[0070] Optionally, it further includes a regular automatic push module, which is configured to analyze the change situation of the business data corresponding to each business requirement, predict the change trend in the future period, determine and output a recommended monitoring frequency based on the prediction result, where the recommended monitoring frequency is used to characterize the frequency at which the user proposes the corresponding business requirement; when receiving the user's authorized push instruction, output the business data of the corresponding business requirement to the user according to the recommended monitoring frequency corresponding to the business requirement included in the authorized push instruction.

[0071] Optionally, it further includes a high-frequency business independent communication module, which is configured to update and determine high-frequency business requirements in real time, and create an independent communication space including an original data interface, a format conversion engine, and a user collaboration layer for each high-frequency business requirement; where the high-frequency business requirement refers to a business requirement whose number of successful matches with the actual requirements proposed at different times or by different users exceeds a preset number, the independent communication space is set in one-to-one correspondence with the business requirement, the user collaboration layer is used for all users who propose the corresponding business requirement to achieve communication and data sharing, the original data interface is used for the user to retrieve the original data corresponding to the corresponding business requirement from the corresponding urban management system, and the format conversion engine is used for the user to convert the original data into a specified format; The actual demand response module 203 is further configured to receive the demand instruction proposed by the user, display all independent communication spaces and the corresponding business requirements, and when the user feedbacks the selection content with any independent communication space, configure the user with the permission to enter and use the original data interface, the format conversion engine, and the user collaboration layer in the independent communication space; if the user does not feedback the selection content, use the user's feedback result as the actual demand.

[0072] An embodiment of the present application also discloses a smart city data integration and sharing device, which includes a memory and a processor, and a computer program capable of being loaded and executed by the processor, such as the smart city data integration and sharing method described above, is stored on the memory.

[0073] An embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program that can be loaded and executed by a processor to perform the smart city data integration and sharing method as described above. The computer-readable storage medium includes, for example, various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0074] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0075] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting the protection scope of the application. Obviously, the described embodiments are only partial embodiments of the present application, rather than all embodiments. Based on these embodiments, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope to be protected by the present application.

Claims

1. A method for integrating and sharing smart city data, characterized in that Including: Real-time obtaining all business data generated by the urban management system, predicting business requirements based on the real-time obtained business data, and generating constraint conditions and corresponding minimum transformation sets according to the business requirements; wherein, the constraint conditions at least include time conditions and space conditions, and the minimum transformation set contains data parameters that meet the corresponding constraint conditions and originate from different urban management systems; Based on the minimum transformation set, extracting the specific business data corresponding to the data parameters, integrating and processing the business data, and storing the business requirements and the corresponding integrated business data as demand content; When receiving the actual requirements proposed by the user, first matching the actual requirements with the stored demand content. If the match is successful, outputting the demand content with a successful match; if the match is unsuccessful, determining the corresponding business data based on the actual requirements, integrating it, and then outputting it.

2. The smart city data integration and sharing method according to claim 1, wherein The storing the business requirements and the corresponding integrated business data as demand content includes: Taking the integrated business data as the data to be activated, and establishing an association relationship between the data to be activated and the business requirements; Judging the demand frequency and change frequency of the data to be activated, and storing the data to be activated with a demand frequency higher than the preset demand frequency threshold and a change frequency lower than the preset change frequency threshold as the first data in the shared platform; Regarding all the data to be activated that are not the first data as the second data, and storing the second data in the corresponding preset edge buffer of the urban management system to which the second data belongs. The edge buffer includes a dynamic buffer and a static buffer. Storing the second data with a change frequency higher than the preset change frequency threshold in the dynamic buffer, and storing the second data with a change frequency not higher than the preset change frequency threshold in the static buffer; According to the different business requirements associated with the second data stored in the edge buffer, configuring processing engines for the second data associated with different business requirements in the edge buffer respectively, and making the processing engines correspond to the business requirements one by one; The if the match is successful, outputting the demand content with a successful match includes: If the match is successful, determining the storage location of the data to be activated corresponding to the demand content with a successful match. If it is stored in the shared platform, outputting the data to be activated corresponding to the demand content with a successful match; if it is stored in the edge buffer, calling and transmitting the data to be activated corresponding to the demand content with a successful match to the shared platform through the corresponding processing engine for the shared platform to output.

3. The smart city data integration and sharing method according to claim 2, wherein The taking the integrated business data as the data to be activated includes: For the integrated business data, on the basis of the corresponding constraint conditions, performing primary sharding processing on the business data according to the preset spatial dimension, and then performing secondary sharding processing on the primarily sharded business data according to the preset business cycle. Taking the data set of the business data contained in each layer after the secondary sharding processing as the data to be activated respectively.

4. The method for integrating and sharing smart city data according to claim 2, wherein The storing the business requirements and the corresponding integrated business data as demand content further includes: The second data with a change frequency higher than the preset change frequency threshold is used as the third data, and the demand time of the service demand corresponding to the third data in the future time period is predicted. The demand time is used to characterize the time when the corresponding actual demand is proposed by the user when the corresponding service demand matches the actual demand proposed by the user; According to the demand time corresponding to the third data, the processing engine is controlled to retrieve the original data from the urban management system to which the third data belongs at a specified moment before the demand time, convert it into the third data, and store it in the corresponding dynamic buffer.

5. The method for integrating and sharing smart city data according to claim 2, wherein, For the second data in the edge buffer area that is associated with different service demands, a processing engine is configured respectively, including: Determine all target service demands in the buffer area that are associated with the second data, and calculate the combined density value of each target service demand based on the preset reference dimensions. Among them, the reference dimensions at least include the dimension of the urban management system to which the data belongs, the data granularity dimension, and the buffer dimension to which the data belongs; According to the combined density value, determine the density level and the corresponding processing engine type of each target service demand, and configure a processing engine corresponding to the determined processing engine type for each target service demand; among them, the density level is used to characterize the processing complexity of the service demand.

6. The method for integrating and sharing smart city data according to claim 2, wherein The method further includes: Analyze the change situation of the service data corresponding to each service demand, and predict the change trend in the future time period. Based on the prediction result, determine and output the recommended monitoring frequency to the user. The recommended monitoring frequency is used to characterize the frequency at which the user proposes the corresponding service demand; When receiving the authorized push instruction from the user, output the service data of the corresponding service demand to the user according to the recommended monitoring frequency corresponding to the service demand included in the authorized push instruction.

7. The method for integrating and sharing smart city data according to claim 1, wherein The method further includes: Determine and update the high-frequency service demands in real time, and create an independent communication space including the original data interface, the format conversion engine, and the user collaboration layer for each high-frequency service demand; among them, the high-frequency service demand refers to the service demand whose number of times of matching the actual demands proposed at different times or by different users exceeds the preset number of times. The independent communication space is set in one-to-one correspondence with the service demand. The user collaboration layer is used for all users who propose the corresponding service demand to realize communication and data sharing. The original data interface is used for users to retrieve the original data corresponding to the corresponding service demand from the corresponding urban management system, and the format conversion engine is used for users to convert the original data into a specified format; When receiving the actual demand proposed by the user, it includes: Receive the demand instruction proposed by the user, display all the independent communication spaces and the corresponding service demands. When the user feedbacks the selection content with any independent communication space, configure the user with the permission to enter and use the original data interface, the format conversion engine, and the user collaboration layer in the independent communication space; if the user does not feedback the selection content, then use the feedback result of the user as the actual demand.

8. A smart city data integration and sharing platform, characterized in that, Including, The business demand prediction module (201) is used to obtain in real time the business data generated by all urban management systems, predict the business demand based on the real-time obtained business data, and generate constraint conditions and corresponding minimum transformation sets according to the business demand; wherein, the constraint conditions at least include time conditions and space conditions, and the minimum transformation sets contain data parameters that meet the corresponding constraint conditions and originate from different urban management systems; The prediction result storage module (202) is used to extract the specific business data corresponding to the data parameters based on the minimum transformation sets, perform integration processing on the business data, and store the business demand and the integrated business data corresponding thereto as demand content; The actual demand response module (203) is used to, when receiving the actual demand proposed by the user, first match the actual demand with the stored demand content. If the match is successful, the demand content with a successful match is output; if the match is unsuccessful, the corresponding business data is determined based on the actual demand and output after integration.

9. A smart city data integration and sharing device, characterized in that It includes a memory and a processor, and a computer program capable of being loaded and executed by the processor, such as the method according to any one of claims 1 to 7, is stored on the memory.

10. A computer-readable storage medium, characterized in that, A computer program capable of being loaded and executed by the processor, such as the method according to any one of claims 1 to 7, is stored.

Images