CIM platform based on cloud native distributed architecture and platform management method

Abstract

The invention discloses a CIM platform based on a cloud native distributed architecture and a platform management method, and belongs to the technical field of data processing. The platform comprises a data access module, a data analysis module and a data service module, the data analysis module comprises a data management module and a data management module, and the data management module is used for data slicing, format conversion and time-space fusion. And the data management module is used for data source management, rule management, metadata management, service management, data resource management, directory management and operation and maintenance management. According to the application, the multi-source data stream is transmitted through the data access module, then the data change condition of the multi-source data stream is analyzed through the data analysis module, then the change trend of the multi-source data stream is predicted, and the prediction result is fed back to the user interface, so that full-life-cycle processing of the multi-source data can be flexibly realized, and the user experience is improved. And the multi-source data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a CIM platform and platform management method based on a cloud-native distributed architecture. Background Art

[0002] At present, with the advancement of smart city construction, City Information Modeling (CIM), as a digital twin technology, plays an increasingly important role in urban planning, management, operation and maintenance. Among related technologies, CIM integrates urban data from multiple sources to form a digital transformation of urban physical entities, supporting cross-departmental collaboration and intelligent decision-making. However, the existing CIM platform cannot flexibly process these multi-source heterogeneous data during data collection and storage, resulting in difficulty in data integration and low data processing efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a CIM platform and platform management method based on a cloud-native distributed architecture, aiming to solve the technical problem that the existing CIM platform cannot flexibly process these multi-source heterogeneous data, resulting in difficulty in data integration and low data processing efficiency.

[0005] To achieve the above-mentioned objectives, the present application provides a CIM platform based on a cloud-native distributed architecture, the platform comprising: a data access module, integrating a multi-source data interface, for accessing multi-source data; a data analysis module, for receiving and analyzing the multi-source data, the data analysis module comprising a data governance module and a data management module, the data governance module is used for data slicing, format conversion and spatiotemporal fusion, and performs real-time monitoring of the multi-source data, predicts the change trend of the multi-source data according to the dynamic changes of the multi-source data, the data management module is used for data source management, rule management, metadata management, service management, data resource management, directory management and operation and maintenance management; a data service module is used to present the change trend on the Web front end in the form of a city twin model.

[0006] In one embodiment, the platform management method is applied to a CIM platform based on a cloud native distributed architecture and includes the following steps: Receive multi-source data streams input from a multi-source data interface through a data access module; Monitoring dynamic changes of the multi-source data streams through a data analysis module; Predicting a change trend of the multi-source data stream according to the dynamic change; The change trend is combined with the three-dimensional application scenario and displayed on the user page through the data service module.

[0007] In one embodiment, before the step of receiving the multi-source data streams inputted from the multi-source data interface through the data access module, the following steps are included: Initialize the initial cluster using the container orchestration tool; According to business requirements, configure relevant parameters of the microservice module, which is a collection of service modules of the management system; Based on the relevant parameters, the microservice module is deployed in the initialized initial cluster.

[0008] In one embodiment, after the step of deploying the microservice module in the initialized initial cluster based on the relevant parameters, the method further includes: Managing the microservice module based on service grid technology and determining the communication strategy of the microservice module; According to the communication strategy, traffic control is performed on the communication between the microservice modules.

[0009] In one embodiment, the step of monitoring the dynamic changes of the multi-source data streams by a data analysis module includes: Preprocessing the multi-source data streams; The preprocessed multi-source data stream is analyzed by the data analysis module to determine the dynamic change.

[0010] In one embodiment, the step of predicting the change trend of the multi-source data stream according to the dynamic change includes: Analyze the historical data through the data analysis module to extract data features of the historical data; Process the data features based on a preset machine learning model and output predicted data; The change trend is determined based on the predicted data and the historical data.

[0011] In one embodiment, after the step of determining the change trend based on the predicted data and the historical data, the method further includes: Analyze the real-time data through the data analysis module to detect abnormal data in the real-time data; Based on the abnormal data and preset abnormal event response rules, alarm information is generated and displayed on the user interface.

[0012] In one embodiment, the step of displaying the change trend in combination with the three-dimensional application scene on the user page through the data service module includes: Based on the change trend and the city digital twin model, construct the three-dimensional application scenario; The three-dimensional application scene is displayed on the user page.

[0013] One or more technical solutions proposed in the present application have at least the following technical effects: the present application manages the data access module, the data analysis module and the data service module in a distributed manner, receives the multi-source data stream transmitted by the multi-source data interface through the data access module, converts the multi-source data stream into a unified format, and then analyzes the data changes of the multi-source data stream through the data analysis module to determine the dynamic changes of the multi-source data stream, inputs a specific machine learning model according to the dynamic changes of the data stream, predicts the change trend of the multi-source data stream, and then feeds back the prediction results to the user interface. Therefore, it is possible to flexibly process multi-source data, including intelligent prediction, decision-making speeches and trend analysis, etc., effectively improving the processing efficiency of multi-source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 This is the core architecture diagram of the first embodiment of the platform management method of this application; Figure 2 This is a flow chart of the second embodiment of the application platform management method; Figure 3 This is a flow chart of the third embodiment of the application platform management method; Figure 4 This is a flow chart of the fourth embodiment of the platform management method of this application; Figure 5 This is a flowchart of the fifth embodiment of the platform management method of this application; Figure 6 This is a flowchart of the sixth embodiment of the application platform management method; Figure 7 This is a flow chart of the seventh embodiment of the application platform management method; Figure 8 This is a flow chart of the eighth embodiment of the application platform management method; Fig. 9 This is the technical support diagram for the management method of this application platform; Fig.10It is a structural diagram of the platform management device of the hardware operating environment involved in the embodiment of the present application.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0020] It should be noted that Fig. 9 The spatial analysis emphasizes the ability to deeply analyze and evaluate urban space. The data access center receives data from various places, including influencing terrain, oblique photography, BIM models, point cloud models, video data, IoT data, geocoding and business data. After obtaining the core data and basic data, it is transferred to the data processing platform layer for management. The platform layer includes the data governance center, data management center, service management center and security monitoring center. The processed data is rendered at the service layer and then output to the application layer. The output can be in the form of two-dimensional review and consultation, key area planning and construction management, smart gas application, smart environmental protection application and smart affordable housing application.

[0021] In the CIM platform, urban data is diverse, including static and dynamic data. Static data includes the city's geographic information and the location of buildings, etc. Dynamic data includes traffic flow and energy consumption, structured and unstructured data. Structured data includes numbers, dates and texts, etc. Unstructured data includes text, images, audio and video, etc. These data usually come from different Internet of Things (IoT) devices, Geographic Information Systems (GIS), Building Information Modeling (BIM), etc. Therefore, it is necessary to integrate multi-source data access interfaces, including file data access, service interface data access, database data access, video data access and IoT access. In this system, multi-source data streams are received to realize intelligent prediction, decision support and trend analysis of urban information.

[0022] However, when processing multi-source data streams, the CIM data management system needs to take into account both the huge amount of historical data and the real-time nature of real-time data, and data storage management requires a lot of calculations. Therefore, when managing multi-source data, the existing CIM platform cannot flexibly handle these multi-source data, resulting in difficulty in data integration and low data processing efficiency.

[0023] In order to solve the above problems, the present application proposes a platform management method, which manages data access modules, data analysis modules and data service modules in a distributed manner, receives multi-source data streams transmitted from a multi-source data interface through the data access module, converts the multi-source data streams into a unified format, and then analyzes the data changes of the multi-source data streams through the data analysis module, determines the dynamic changes of the multi-source data streams, inputs a specific machine learning model according to the dynamic changes of the data streams, predicts the change trends of the multi-source data streams, and then feeds back the prediction results to the user interface. In the above scheme, the multi-source data stream includes historical data and real-time data. The historical data is calculated offline through a big data processing framework, and the real-time data is analyzed in real time through a stream data processing framework. Therefore, it is possible to flexibly process multi-source data, including intelligent prediction, decision-making speeches and trend analysis.

[0024] Based on this, the first embodiment proposes a CIM platform based on a cloud native distributed architecture. Figure 1 , Figure 1 This is a core architecture diagram of a CIM platform based on a cloud native distributed architecture. The platform includes a data access module, a data analysis module and a data service module. The data access module receives and analyzes multi-source data. The data analysis module includes a data governance module and a data management module. The data governance module is used to receive and process the received multi-source data, including data slicing, format conversion and spatiotemporal fusion. The data management module is used for data source management, rule management, metadata management, service management, data resource management, catalog management and operation and maintenance management.

[0025] Exemplarily, the CIM platform based on cloud-native distributed architecture is divided into a data access module, a data analysis module and a data service module, wherein the data analysis module includes a data governance module and a data management module. The data access module is used for data access of the data source, which can be accessed in various forms. The data source can be basic data, core data or manufacturer data. The data source is not limited here. Specifically, the data access methods include file data access, service interface access, database data access, video data access, spatial data engine access and IOT Internet of Things access; the data governance module governs the accessed data through data slicing, format conversion or spatiotemporal fusion. Among them, data slicing includes image slicing, terrain slicing and oblique photography, etc. Format conversion includes BIM conversion, coordinate conversion and point cloud conversion, etc. Spatiotemporal fusion includes element mapping, IoT fusion, video fusion, etc., and then provides data services with the governance results. Data services can be creating applications, configuring applications, associating services, service publishing and API console. Among them, the data access module, data governance module and data service module all need the data management module to manage the data. The data management module is used for data source management, planning management, metadata management, service management, data resource management, catalog management and operation and maintenance management. Therefore, the CIM platform based on cloud native distributed architecture clearly describes the architecture of CIM applications and the actual application of its components in urban planning, construction and management.

[0026] Based on the first embodiment, the second embodiment of the present application provides a platform management method, referring to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the application platform management method.

[0027] In this embodiment, the platform management method includes steps S100 to S400: Step S100: receiving a multi-source data stream inputted from a multi-source data interface through a data access module.

[0028] It should be noted that the execution subject of this embodiment can be a platform management device, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the platform management device as an example to illustrate this embodiment and the following embodiments.

[0029] The platform management system integrates multi-source data access interfaces, including but not limited to file data access, service interface data access, database data access, and IOT Internet of Things access, etc. These interfaces collect dynamic and static urban data in real time. After the data access module receives data from multi-source data streams, it will first perform a preliminary initialization operation to convert the format of the multi-source data streams and perform data cleaning.

[0030] As an optional solution for integrating multi-source data access interfaces in a platform management system, multiple data interfaces are integrated to provide a dedicated interface for each data type source. When the system is split according to business function modules to build a distributed system, each module is serviced and deployed on different servers. They interact with each other through interfaces. Each interface is designed for a specific data source or data type to ensure that it can accept and process data from different IoT devices, GIS systems, BIM models, etc.

[0031] Specifically, multiple data interfaces are based on multiple communication protocols, which can be application programming interfaces based on the REST (Representational State Transfer) architecture, WebSocket or other communication protocols. By defining clear interface specifications and data formats, seamless connection with different data sources can be achieved. After the data is connected, it is preprocessed by the data collection module to meet the format requirements of data reception.

[0032] For example, in the smart city CIM system, a special data interface is defined according to the customer's data source format and communication protocol, and interface specifications are designed specifically for the data source of each data interface. The interface specifications include request parameters, response format, error handling, etc. During the interface implementation process, a format conversion service is set in the data access module to convert the original data into a unified data format within the platform.

[0033] As another optional solution for integrating multi-source data access interfaces into a platform management system, a unified integrated interface is developed to receive multiple data.

[0034] Specifically, the interface adopts a common data format and communication protocol, such as JSON, XML or a custom data exchange format. Complex parsing and conversion logic is set up inside the interface. When data structures and format differences are received from different data sources, these data structures and format differences are unified through the built-in parsing and conversion logic to achieve unified data access and distribution.

[0035] Step S200: monitoring the dynamic changes of the multi-source data streams through a data analysis module.

[0036] After receiving multi-source data streams and converting the data formats of multi-source data streams into a unified format, develop data analysis modules according to business needs to implement real-time analysis, batch analysis, anomaly detection, trend prediction and other functions. Select appropriate algorithms and models to process and analyze data, which can be time series analysis, machine learning algorithms, etc., to control the dynamic changes of multi-source data streams in real time.

[0037] Specifically, a big data processing engine (Apache Spark) is used to perform batch analysis of historical data, combined with machine learning algorithms to make predictions and optimization recommendations for urban traffic, population flow, and energy consumption; a real-time data analysis engine (Apache Flink) is used to process multi-source data streams uploaded by IoT devices, supporting real-time response and early warning to emergencies, such as fires and traffic accidents.

[0038] As an optional solution for batch analysis of historical data using a big data processing engine (Apache Spark), historical data such as urban traffic, population mobility, and energy consumption are collected and stored in a distributed file system (Hadoop Distributed File System, HDFS). Spark's DataFrame API is then used to clean, transform, and aggregate the data. Spark's MLlib library is then used to extract and transform features. The extracted features are then trained using a machine learning algorithm to output prediction results. This allows optimization suggestions to be made based on historical data, such as traffic planning adjustments and energy efficiency improvements.

[0039] As another optional implementation method of using the real-time data analysis engine (Apache Flink) to process multi-source data streams uploaded by IoT devices, Flink's Source function is used to access multi-source data streams from real-time data sources such as IoT devices and sensors, and Flink's DataStream API is used to perform real-time processing on multi-source data streams, such as window operations and aggregate calculations. Flink's event-driven characteristics are then used to detect and respond to emergencies in real time. Flink's CEP (complex event processing) library is then used to define event patterns and detect these patterns to achieve early warning of complex events.

[0040] Step S300: predicting the change trend of the multi-source data stream according to the dynamic change.

[0041] After training the data, extract useful features based on business needs and data characteristics, and then divide the data into training set and validation set. Use the training set to train the model, and use the validation set to validate the model.

[0042] As an optional solution to predict changing trends, computer vision technology combined with large model technology can be used to detect and analyze the construction site in real time. By analyzing the on-site images taken by the camera, real-time detection and analysis can be carried out, and combined with the analysis of the on-site images taken by the camera, large model technology plus intelligent analysis can be used to determine.

[0043] Step S400: Display the change trend in combination with the three-dimensional application scenario on a user page through a data service module.

[0044] 3D application scenes, such as buildings, terrain, and equipment, are constructed through 3D modeling tools, and real-time rendering is performed using a rendering engine. In the data service module, appropriate visualization technology is selected based on the characteristics and display requirements of the data, which may be heat maps, height maps, and particle systems, etc., to map the data to the 3D application scene. At the same time, a user operation interface is provided, including navigation, menus, toolbars, and information panels, etc., to achieve real-time data communication and synchronization between the data service module and the 3D application scene.

[0045] As an alternative to building a 3D application model through heat maps, it should be noted that heat maps are a visualization technique that uses color changes to represent data distribution and density. Therefore, in a 3D scene, heat maps can be applied to terrain, building surfaces, or other 3D objects. By calculating the data value of each area and using color gradients to represent the size of these values, an intuitive data distribution map is created.

[0046] For example, at two locations 100 meters apart, A is a bustling shopping mall, and B is an office building that has been built but not yet opened. When constructing a three-dimensional application scenario for this area, the flow of people at these two locations is collected separately, and a gradient scheme is selected, from blue (low value) to red (high value) to represent the flow of people. The collected flow of people is mapped to the corresponding area of ​​the three-dimensional application scenario. Thus, a blue three-dimensional model of the office building and a red three-dimensional model of the shopping mall can be obtained. The terrain of the middle 100 meters determines the depth of blue or red according to the distribution map of the flow of people. Optionally, from the three-dimensional model of the office building to the three-dimensional model of the shopping mall, the color changes from dark blue to light blue, then changes to light red, and then changes from light red to dark red. Thus, an intuitive data distribution map is created.

[0047] Optionally, after creating a three-dimensional application scene, in order to enhance the user experience, interactive functions are added, such as displaying specific data values ​​of the three-dimensional application model when the mouse hovers, and jumping to a related data page when clicked.

[0048] As an optional implementation scheme for displaying the changing trend in a three-dimensional application model, after collecting multi-source data, the potential changes in the historical data and the real-time changes in the real-time data are determined by analyzing the historical data and the real-time data, and the dynamic changes of the data analyzed separately are used to determine the changing trend of the multi-source data stream. The potential risks and real-time warnings in the changing trend are classified in the form of labels, and the data is determined as a scene based on the common labels of multiple data, and a three-dimensional application scenario of the multi-source data stream is constructed, and the three-dimensional application scenario is displayed on the user page.

[0049] Exemplarily, through the historical data and real-time data collected by traffic cameras, the three-time standard deviation method is used to detect the legacy flow value in the traffic flow data. The label corresponding to the abnormal data is determined by the type and nature of the abnormal data. For example, when analyzing the historical data, the historical vehicle flow, vehicle congestion duration, and number of accidents are counted to determine the labels of "congestion" and "accident-prone areas". The real-time abnormal data is collected by the traffic camera, and labels are generated according to the abnormal data with the highest frequency. When the real-time traffic flow is abnormal, the abnormal label is determined by the preset threshold, such as traffic congestion, traffic accidents, and road construction. For example, the current vehicle waiting time exceeds ten minutes or the vehicle speed is less than ten kilometers per hour for half an hour, which is considered as traffic congestion. Traffic accidents detect that there are motor vehicles that have not moved and occupy the channel. Among them, road construction can identify abnormal situations based on the unified lane change of motor vehicles in a certain area. Therefore, in this system, according to the collected abnormal situations, it can be determined as current traffic congestion, and multiple abnormal labels are counted. According to the label information combined with the specific surrounding location information, it can be a gas station, a traffic light, or a public toilet, etc., so as to determine the specific three-dimensional application scene, and finally display the three-dimensional application scene on the user interface through visualization technology.

[0050] Optionally, after determining the three-dimensional application scenario, the statistical abnormal labels are marked in the corresponding three-dimensional application scenario. For example, accident-prone areas are marked in the three-dimensional application scenario, and warning signs are set in the actual scenario according to the marked positions. If there is road congestion caused by traffic lights in the surrounding area, the time setting of the traffic lights is predicted and adjusted according to the vehicle and pedestrian flow, which can more intuitively reflect the real information of the three-dimensional city.

[0051] This embodiment provides a platform management method, which collects data streams from multiple interfaces in real time through a data access module, and then monitors and analyzes these data to identify dynamic changes; then, a machine learning model is used to predict future trends based on these changes. Finally, the data service module intuitively displays these trends on the user interface through three-dimensional visualization technology, so that users can clearly understand and respond to changes in multi-source data streams.

[0052] Based on any of the above embodiments, the third embodiment of the present application provides a platform management method, referring to Figure 3 , Figure 3 This is a flow chart of the third embodiment of the platform management method of this application. Before step S100, steps S50 to S70 are included: Step S50: Initialize the initial cluster using a container orchestration tool.

[0053] Step S60, configuring relevant parameters of a microservice module according to business requirements, wherein the microservice module is a set of service modules of the management system.

[0054] Step S70: deploying the microservice module in the initialized initial cluster based on the relevant parameters.

[0055] It should be noted that container orchestration tools are software used to manage and automate the deployment, expansion, and operation of containerized applications. Business requirements are based on the front-end actions performed by users and a series of requirements defined for the output results. Microservice modules can be adjusted by changing orchestration parameters.

[0056] Use Kubernetes to install and initialize the cluster. Set relevant parameters in the configuration file of the microservice module, including but not limited to container images, resource limits, environment variables, storage volumes, and network settings. After setting the relevant parameters, write a Kubernetes deployment list to define the deployment details of the microservice, and then use the deployment command in Kubernetes to deploy the microservice module according to the deployment list.

[0057] This embodiment provides a platform management method that initializes the initial cluster through a container orchestration tool, and configures relevant parameters of the microservice module according to business needs, providing an efficient, scalable and easy-to-manage operating environment for microservices, ensuring that microservices can be quickly deployed, flexibly expanded and stably operated in a containerized architecture, thereby meeting the dynamic changes of the business and the needs of continuous integration / continuous deployment.

[0058] Based on any of the above embodiments, the fourth embodiment of the present application provides a platform management method, referring to Figure 4 , Figure 4 This is a flow chart of the fourth embodiment of the platform management method of the present application. After step S70, steps S80 to S90 are also included: Step S80: managing the microservice module based on the service grid technology and determining the communication strategy of the microservice module.

[0059] It should be noted that service mesh technology refers to the infrastructure layer used to handle communication between services. By deploying a Sidecar proxy for each application service in the microservice architecture, complex communication between services is completed to form a network topology structure, namely the service mesh. The communication strategy is configured through the control plane of the service mesh, which can be routing rules, load balancing strategy, and fault recovery strategy. The communication strategy is not limited here and is determined based on the specific requirements of the microservice module.

[0060] After deploying the microservice modules, select a service mesh technology to manage traffic control, secure communication, and service discovery between microservices, and then deploy the service mesh control plane in the initialized cluster; then, based on business needs, define the communication strategy between microservice modules, including routing rules, retry strategies, and supermarket settings.

[0061] Step S90: performing flow control on the communication between the microservice modules according to the communication strategy.

[0062] First, you need to define and configure the corresponding routing rules and traffic management policies in the service grid. According to the configuration of the policies, the control plane of the service grid will send these policies to all relevant proxies, and then these proxies will control the traffic between microservices according to the configured rules.

[0063] As an optional implementation method for a service grid to issue policies to related proxies, in the service grid, routing rules are configured to direct a portion of user traffic (for example, 10%) to the new recommended service version, while the remaining traffic (90%) flows to the old version. The control plane of the service grid will then push these rules to the relevant Sidecar proxies, which will then route requests according to these rules. If the new version performs well, the traffic ratio can be gradually increased, otherwise it can be quickly rolled back to the old version, thereby ensuring that the stability of the system and the user experience are not affected.

[0064] This embodiment provides a platform management method, which provides a fine-grained traffic control mechanism so that developers can accurately manage communication paths, traffic distribution, fault recovery, and security policies between microservices, thereby optimizing the calling efficiency between services, improving the stability and reliability of the system, and ensuring effective service discovery and load balancing in complex microservice architectures.

[0065] Based on any of the above embodiments, the fifth embodiment of the present application provides a platform management method, referring to Figure 5 , Figure 5 This is a flowchart of the fifth embodiment of the platform management method of this application, step S200 includes steps S210~S220: Step S210: pre-processing the multi-source data stream.

[0066] It should be noted that preprocessing includes but is not limited to data cleaning, data conversion and data aggregation.

[0067] Data cleaning includes using Spark's RDD or DataFrame API to delete missing values ​​or outliers; removing duplicate records to ensure data uniqueness; correcting erroneous data, for example, unifying incorrect date formats or correcting erroneous numerical records; data transformation includes converting timestamps to date and time formats to facilitate analysis of trends within a specific time period, and converting categorical data (such as vehicle type, cargo type) to numerical data so that machine learning algorithms can process it; data aggregation includes aggregating traffic flow data by time period (such as hours, days, weeks), calculating the average flow for each time period, and aggregating energy consumption data by region or equipment type to identify energy usage patterns.

[0068] Optionally, after step S210, a suitable distributed storage system may be selected according to the characteristics of the data and business requirements, such as HDFS, HBase, etc., and then shard management may be performed according to the attribute dimensions of the data.

[0069] As an alternative to sharding management based on data attribute dimensions, HDFS is selected for data sharding. When a file is written to HDFS, the HDFS client will split the file into multiple data blocks. It is feasible to split a 512MB file into four 128MB data blocks.

[0070] After sharding, the preprocessed data stream is written into the distributed storage system through the distributed module. Corresponding to the range-based sharding method, it is necessary to record the data interval range of each node, and the interval information in the metadata also needs to be updated synchronously when the data block is split.

[0071] Specifically, the metadata in HDFS is managed by the NameNode. The metadata includes information such as file directory structure, file attributes, and data block location. When the file is split and stored in the data node, the NameNode will update the metadata to record the storage location and replica information of each data block.

[0072] Furthermore, through the distributed message queue, data streams from different sources are received, and the distributed message queue distributes the data streams to different partitions. Each partition is regarded as a shard. After the data is preprocessed, it is written to the corresponding partition according to the range of the shard key. Therefore, each partition is responsible for storing data within a specific range and can determine the storage location of the data.

[0073] For example, the data attributes include item ID and user ID. With user ID as the sharding key, the sharding can store data with user IDs from 1 to 1000 in the first shard, data with 1001 to 2000 in the second shard, and so on until the data sharding is completed. When writing to a partition, the stored data is limited according to the range of the user ID in the partition.

[0074] Step S220: analyzing the preprocessed multi-source data stream by the data analysis module to determine the dynamic change.

[0075] By receiving, processing and analyzing pre-processed multi-source data streams in real time, you can gain insights into the immediate status and change trends of multi-source data.

[0076] For example, in a real-time data stream processing scenario, Kafka can be used as a data source. KafkaConsumer can be used to connect to Kafka and subscribe to a specific topic, and then messages can be read and processed continuously. By analyzing this data, patterns and anomalies in the data can be identified, and future trends can be predicted. It should be noted that these messages can be sensor data, transaction records, or any form of real-time information.

[0077] This embodiment provides a platform management method, which converts raw and possibly disorganized data into a structured, clean and easy-to-analyze format by preprocessing multi-source data streams and then analyzing the dynamic changes of the preprocessed data streams through a data analysis module, so as to accurately capture and understand the evolution trends, patterns and anomalies of the data over time, thereby providing real-time, data-driven insights for decision-making.

[0078] Based on any of the above embodiments, the sixth embodiment of the present application provides a platform management method, referring to Figure 6 , Figure 6 This is a flowchart of the sixth embodiment of the platform management method of this application, step S300 includes steps S310~S330: Step S310: Analyze the historical data through the data analysis module to extract data features of the historical data.

[0079] First, historical data of urban traffic is obtained through traffic monitoring cameras, traffic sensors, GPS data, and public transportation card swiping records. After obtaining the historical data, the historical data is cleaned to eliminate duplicate data, process missing values, and correct erroneous data. Then, the data format is converted, and the key data features in the historical data are extracted using machine learning algorithms.

[0080] As an optional solution for extracting key data features of historical data based on machine learning algorithms, by converting data into a format suitable for processing by machine learning algorithms, selecting a suitable clustering algorithm, which can be K-mean, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), setting algorithm parameters according to data characteristics and requirements, and obtaining the correlation between data by training the collected data.

[0081] For example, a CIM platform collected urban traffic flow data for the past year, including traffic volume, speed, congestion, etc. at each intersection, as well as external factors such as weather and holidays that may affect traffic flow. After removing invalid data, the daily traffic peak hours, the impact of holidays on traffic flow and other characteristics are extracted, and different traffic flow patterns are identified through clustering algorithms, and then the association relationship between different traffic flows is discovered using association rule mining algorithms.

[0082] Step S320, processing the data features based on a preset machine learning model and outputting predicted data.

[0083] It should be noted that the preset machine learning model is determined according to the nature of the data. It can be a time series model with strong time series characteristics, including the ARIMA (Autoregressive Integrated Moving Average Model) model and the LSTM (Long Short-Term Memory) model; it can be a regression model, including a linear model, a random forest model, a gradient regression model, etc.; and a hybrid model, including LightGBM.

[0084] As an optional solution for processing data features, after collecting statistical features, time series features, and possible machine learning features of traffic flow, the feature matrix and the corresponding traffic flow labels are divided into training sets, validation sets, and test sets. The library of the random forest regression model is loaded, and the number of feature dimensions is used as model parameters. The model is trained using the features and labels of the training set. During the training process, the model learns the relationship between the features and the target variable. The performance of the model is then evaluated and the variance is calculated using the validation set, and the model parameters are adjusted according to business needs. When the performance is optimized to the target qualified value, the predicted data is output.

[0085] Step S330: determining the change trend based on the predicted data and the historical data.

[0086] As an optional implementation method for determining the change trend, the prediction data and historical data are combined to input into the prediction model, and the prediction model is optimized. The change of the prediction data can be determined by modifying the parameters of the prediction model.

[0087] As another optional implementation method for determining the changing trend, the historical data and the predicted data are visualized using charts, and the trend of the data is analyzed, including rise, fall, and fluctuation, so as to simply analyze the changing trend of the data direction.

[0088] This embodiment provides a platform management method, which mines patterns and features in historical data through a data analysis module, and then uses these features to analyze through a preset machine learning model to generate future prediction data. By comparing these prediction data with actual historical data, the change trend of the data can be identified, thereby providing a basis for decision-making.

[0089] Based on any of the above embodiments, the seventh embodiment of the present application provides a platform management method, referring to Figure 7 , Figure 7 This is a flow chart of the seventh embodiment of the platform management method of this application. After step S330, steps S340 to S350 are also included: Step S340: Analyze the real-time data through the data analysis module to detect abnormal data in the real-time data.

[0090] Input real-time data into the configured data analysis module, use statistical analysis methods (such as z-score, box plot, etc.) to detect outliers in the data, and output anomaly detection results, including abnormal data points, anomaly types, anomaly degrees and other information.

[0091] Step S350: Based on the abnormal data and preset abnormal event response rules, generate alarm information and display it on the user interface.

[0092] It should be noted that the preset abnormal event response rules refer to the formulation of abnormal event response rules based on business needs and data characteristics. The rules should clearly define which types of abnormal data need to trigger alarms, as well as the alarm level, content, triggering conditions, etc.

[0093] Abnormal data is identified through the rules established in the alarm system, and the results are displayed in a visual form (such as scatter plots, line charts, etc.) to facilitate a more intuitive understanding of the abnormal data. According to the abnormal detection results, appropriate processing measures are taken and relevant personnel are notified through emails, text messages, message push, etc. The alarm information should include detailed information on the abnormal data, possible causes, and recommended processing measures.

[0094] Exemplarily, the pre-processed real-time traffic flow data is input into the data analysis module, and the data analysis module uses the mean and standard deviation algorithm to analyze the data in real time. For example, at a certain time point, the system detects that the traffic flow on a certain road suddenly decreases by 40%, and this decreasing trend has lasted for 15 minutes. The system identifies the data point as abnormal data and classifies it as a possible traffic congestion event. According to the preset abnormal event response rules, when the traffic flow suddenly decreases by more than 30% and lasts for more than 10 minutes, a first-level alarm is triggered. The system constructs an alarm message "Traffic congestion event occurred on a certain road, the traffic flow decreased by 40%, and the duration has exceeded 15 minutes. At the same time, it is recommended to take traffic diversion measures." The alarm information is displayed to the user in real time through the console of the traffic monitoring system, and the user can take corresponding processing measures according to the alarm information.

[0095] This embodiment provides a platform management method, which monitors and analyzes data streams in real time through a data analysis module to identify abnormal data points therein. Once an anomaly is detected, the system automatically generates alarm information according to preset abnormal event response rules and displays this information on the user interface so that users can understand it in time and take corresponding actions.

[0096] Based on any of the above embodiments, Embodiment 8 of the present application provides a platform management method, referring to Figure 8 , Figure 8 This is a flow chart of Embodiment 8 of the platform management method of this application, step S400 includes steps S410 to S420: Step S410: construct the three-dimensional application scenario based on the change trend and the city digital twin model.

[0097] Step S420: display the three-dimensional application scene on the user page.

[0098] As an optional solution for building a 3D application scenario, the traffic change trend obtained by analysis is combined with the city digital twin model to achieve dynamic mapping of traffic data, and reflect abnormal situations such as traffic congestion and accidents in real time in the model, as well as corresponding emergency response measures. Furthermore, a 3D rendering engine (such as Unity, Unreal Engine, etc.) is used to render the combined model to generate a realistic 3D scene.

[0099] Exemplarily, a three-dimensional view of the city is constructed on the user page, including buildings, roads, traffic facilities, etc. In the view, real-time traffic data (such as vehicle speed and congestion level) on different roads and traffic nodes are displayed in the form of dynamic icons or color changes; the changing trend of traffic data in the past period of time is displayed in the form of animation or timeline; the traffic conditions at different time points can be viewed by dragging the timeline. When abnormal events such as traffic congestion and accidents occur, the system will prompt the user in a striking way (such as a red flashing icon) in the view and provide corresponding handling suggestions; or click buttons or icons on the page to switch different view modes (such as looking down, looking straight), adjust the viewing angle, zoom in or out the view, etc. Optionally, choose to view detailed information or historical data of a specific traffic node.

[0100] This embodiment provides a platform management method, which enables users to monitor and analyze the city's operating status in real time, predict future development trends, and provide decision support by providing an intuitive and interactive virtual environment. It helps to achieve in-depth analysis and effective management of urban data through three-dimensional visualization in multiple fields such as urban planning, traffic management, and environmental monitoring.

[0101] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the platform management method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0102] The present application provides a platform management device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the platform management method in the above-mentioned embodiment one.

[0103] Reference below Fig.10 , which shows a schematic diagram of the structure of a platform management device suitable for implementing an embodiment of the present application. The platform management device in the embodiment of the present application may include but is not limited to mobile terminals such as laptops, PADs (Portable Application Description: tablet computers), etc., and fixed terminals such as desktop computers. Fig.10 The platform management device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Fig.10As shown, the platform management device may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the platform management device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the platform management device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a platform management device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0105] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0106] The platform management device provided by the present application adopts the platform management method in the above embodiment, which can solve the technical problem that the existing CIM platform cannot flexibly process these multi-source heterogeneous data, resulting in difficulty in data integration and low data processing efficiency. Compared with the prior art, the beneficial effects of the platform management device provided by the present application are the same as the beneficial effects of the platform management method provided by the above embodiment, and the other technical features in the platform management device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0109] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the platform management method in the above-mentioned embodiment.

[0110] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0111] The computer-readable storage medium may be included in the platform management device; or may exist independently without being installed in the platform management device.

[0112] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the platform management device, the platform management device can write computer program codes for performing the operations of the present application in one or more programming languages ​​or a combination thereof. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0115] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned platform management method, and can solve the technical problem that the existing CIM platform cannot flexibly process these multi-source heterogeneous data, resulting in difficulty in data integration and low data processing efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the platform management method provided in the above-mentioned embodiment, and will not be repeated here.

[0116] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A CIM platform based on cloud native distributed architecture, characterized in that: The platform includes: Data access module, integrating multi-source data interfaces, used to access multi-source data; A data analysis module is used to receive and analyze the multi-source data. The data analysis module includes a data governance module and a data management module. The data governance module is used for data slicing, format conversion, and spatiotemporal fusion, and performs real-time monitoring of the multi-source data, and predicts the change trend of the multi-source data according to the dynamic changes of the multi-source data. The data management module is used for data source management, rule management, metadata management, service management, data resource management, catalog management, and operation and maintenance management; The data service module is used to present the changing trend in the form of a city twin model on the Web front end.

2. A platform management method, characterized in that: Applied to a CIM platform based on a cloud-native distributed architecture, the platform management method comprises the following steps: Receive multi-source data streams input from a multi-source data interface through a data access module; Monitoring dynamic changes of the multi-source data streams through a data analysis module; Predicting the change trend of the multi-source data stream according to dynamic changes; The change trend is combined with the three-dimensional application scenario and displayed on the user page through the data service module.

3. The platform management method according to claim 2, characterized in that: Before the step of receiving the multi-source data streams inputted from the multi-source data interface through the data access module, the method includes: Initialize the initial cluster using the container orchestration tool; According to business requirements, configure relevant parameters of the microservice module, which is a collection of service modules of the management system; Based on the relevant parameters, the microservice module is deployed in the initialized initial cluster.

4. The platform management method according to claim 3, characterized in that: After the step of deploying the microservice module in the initialized initial cluster based on the relevant parameters, the method further includes: Managing the microservice module based on service grid technology and determining the communication strategy of the microservice module; According to the communication strategy, traffic control is performed on the communication between the microservice modules.

5. The platform management method according to claim 2, characterized in that: The step of monitoring the dynamic changes of the multi-source data streams by means of a data analysis module comprises: Preprocessing the multi-source data streams; The preprocessed multi-source data stream is analyzed by the data analysis module to determine the dynamic change.

6. The platform management method according to claim 2, characterized in that: The step of predicting the change trend of the multi-source data stream according to the dynamic change comprises: Analyze the historical data through the data analysis module to extract data features of the historical data; Process the data features based on a preset machine learning model and output predicted data; The change trend is determined based on the predicted data and the historical data.

7. The platform management method according to claim 6, characterized in that: After the step of determining the change trend based on the forecast data and the historical data, the method further includes: Analyze the real-time data through the data analysis module to detect abnormal data in the real-time data; Based on the abnormal data and preset abnormal event response rules, alarm information is generated and displayed on the user interface.

8. The platform management method according to claim 2, characterized in that: The step of displaying the change trend in combination with the three-dimensional application scene on the user page through the data service module includes: Based on the change trend and the city digital twin model, construct the three-dimensional application scenario; The three-dimensional application scene is displayed on the user page.

Images