Smart city data distributed processing and transmission method and system

Abstract

The invention belongs to the technical field of smart cities, and particularly relates to a smart city data distributed processing and transmission method and system. A geographic space distribution algorithm is utilized to configure multilevel data routing nodes among urban subsystems, and each node is provided with a special data processing unit and a short-time cache region. The system dynamically adjusts a data access point and a routing path based on a predictive analysis technology, encapsulates a data stream segment, endows the data stream with an independent processing mark, and maps the data stream segment to different nodes. And the data segments are transmitted to the routing node in parallel for primary processing, re-aggregated according to the mark after processing, and a result is transmitted to the target subsystem. And the target subsystem restores the data content, updates the processing state, and confirms the processing result to the sender. According to the method, delay is reduced, and system expansibility and safety are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart cities, and specifically relates to a method and system for distributed processing and transmission of smart city data. Background Art

[0002] With the acceleration of the urbanization process, the development of smart cities has been rapid, and urban management and services have become more dependent on big data technology. The data involved in smart cities is huge and diverse, including many subsystems such as traffic flow, public safety, environmental monitoring, and public services. Traditional centralized data processing methods face many challenges, such as data transmission delay, poor system scalability, data security and privacy protection, etc. Therefore, distributed data processing and transmission methods are particularly important, but there are still deficiencies in the existing technologies, and further innovation is urgently needed to meet the efficient operation requirements of smart cities;

[0003] Current data processing technologies in smart cities usually adopt static data routing and processing node configuration, lacking effective response to data mobility and real-time changes. These systems often fail to fully consider the data flow differences and real-time changes between urban subsystems, resulting in low data processing efficiency and inability to dynamically optimize data streams and processing resources. In addition, most existing methods fail to predict changes in data traffic and adjust data processing strategies according to the prediction results, which limits the adaptability and scalability of the systems. Summary of the Invention

[0004] (1) Technical Problems to be Solved

[0005] The present invention mainly aims at the above problems and proposes a method and system for distributed processing and transmission of smart city data, aiming to improve the efficiency and real-time performance of smart city data processing and transmission, reduce data transmission delay, and enhance the scalability and security of the system by dynamically adjusting data routing and processing node configuration.

[0006] (2) Technical Solutions

[0007] To achieve the above object, in the first aspect of the present invention, a method for distributed processing and transmission of smart city data is provided, including the following steps:

[0008] Using a geospatial distribution algorithm, configure multi-level data routing nodes between urban subsystems, and each node is equipped with a dedicated data processing unit and a short-term buffer;

[0009] Based on predictive analysis technology, evaluate the type and scale of the data traffic to be processed, and dynamically adjust the data access points and routing paths of each subsystem;

[0010] Segment and encapsulate the received data stream, assign an independent processing mark to each segment of data, and map it to different data routing nodes;

[0011] After the data encapsulation is completed, the encapsulated data segments are transmitted in parallel to the specified routing nodes for preliminary processing;

[0012] After each routing node completes the preliminary data processing, it re-aggregates the processing results according to the data tags and transmits the aggregated data to the target subsystem;

[0013] After receiving the aggregated data, the target subsystem restores the data content and updates the node processing status information;

[0014] After the data is restored, the target subsystem sends a processing result confirmation signal to the data sender.

[0015] Furthermore, the specific steps for configuring multi-level data routing nodes among the subsystems of the city include:

[0016] Determine the physical locations and data requirements of each subsystem in the city's geographic information system;

[0017] Based on the geospatial distribution algorithm, divide different regions, and set up a main routing node and several auxiliary routing nodes in each region;

[0018] Select the optimal location for each main routing node and deploy a dedicated data processing unit and a short-term buffer at this location;

[0019] According to the data traffic characteristics and communication frequencies of each subsystem, set up multiple data access points and connect them to the corresponding main routing nodes;

[0020] Optimize the connection relationship between each main routing node and the auxiliary routing nodes it covers through network topology analysis;

[0021] Establish an efficient data interaction channel between the main routing nodes and the auxiliary routing nodes to achieve a multi-level data routing structure;

[0022] Uniformly number all routing nodes and record the numbering information in the central control system.

[0023] Furthermore, the steps of the method for dynamically adjusting the data access points and routing paths of the smart city based on predictive analysis technology include:

[0024] Based on historical data and real-time monitoring data, establish a data traffic prediction model for each subsystem of the city;

[0025] Using the data traffic prediction model, regularly generate future data traffic prediction values for each subsystem;

[0026] Based on the future data traffic prediction values of each subsystem, calculate the optimal data access point location for each subsystem;

[0027] Adjust the settings of the actual data access point according to the calculated optimal data access point location;

[0028] Compare the deviation between historical data and predicted data, and update the parameters of the data traffic prediction model;

[0029] Based on the geospatial distribution algorithm, combined with the data traffic prediction results, re-evaluate and select the connection relationship between the main routing node and the auxiliary routing node.

[0030] Furthermore, a method for dynamically adjusting the data access point and routing path of a smart city based on predictive analysis technology, and the specific algorithm calculation formula is as follows:

[0031] Step A: Use the exponential smoothing method to predict the future data traffic of each subsystem. The formula is:

[0032] F t+1 = αD t +(1 - α)F t

[0033] Where F t+1 is the predicted value at the next time step, D t is the observed value at the current time step, F t is the predicted value at the current time step, and α is the smoothing coefficient, with a value range of 0 < α < 1;

[0034] Step B: Calculate the optimal data access point location of each subsystem, and use the weighted shortest path algorithm to optimize the data transmission path. The formula is:

[0035]

[0036] Where L opt is the optimal access point location, w i is the traffic weight from node p i to the predetermined access point p opt , and d(p i , p opt ) is the distance from node pi to p opt ;

[0037] Step C: Compare the deviation between historical data and predicted data, and use the mean square error to update the prediction model parameters. The formula is:

[0038]

[0039] Where D t is the actual observed value, F t is the corresponding predicted value, and N is the number of observation points;

[0040] Step D: Combine the prediction results and data traffic, and use the dynamic programming algorithm to evaluate and optimize the connection relationship between the primary routing node and the secondary routing node.

[0041] Further, the method further includes:

[0042] Deploy a consistency verification module on each node for recording and monitoring the received data processing requests and responses;

[0043] Generate a unique transaction identifier for each data segment transmitted and processed, and append the identifier to the corresponding data segment;

[0044] Before the data segment is transmitted to the target node, the source node stores a data copy in the short-term buffer to retain the initial state of the data change;

[0045] After receiving the data segment, each target node processes the data segment according to the transaction identifier and records the processing situation in the transaction log in real time;

[0046] Establish a global transaction coordinator, and collect the processing results of all nodes under the current transaction identifier by polling the transaction logs of each node;

[0047] The global transaction coordinator compares the processing status of each node based on the collected processing results to check if there are inconsistent situations;

[0048] When detecting inconsistencies, the global transaction coordinator notifies the relevant nodes to roll back to the initial state and resynchronize the data segments;

[0049] For completed consistent transactions, the global transaction coordinator sends an acknowledgment signal to each node, instructing each node to update its data status to the final determined value;

[0050] After receiving the acknowledgment signal, each node writes the corresponding transaction identifier and the final data status to the permanent storage area for later retrieval and verification.

[0051] Further, the method further includes monitoring the status of each node, and the steps of monitoring the status of each node include:

[0052] Deploy a status monitoring agent on each node for collecting and reporting the current status information of the node;

[0053] Configure a unique identifier for each status monitoring agent and associate the identifier with the corresponding node;

[0054] Periodically, the status monitoring agent sends a heartbeat signal containing the unique identifier and the node status information to the central monitoring server;

[0055] The central monitoring server receives and parses the heartbeat signals of each node, and records the status information of the corresponding node;

[0056] The central monitoring server analyzes the collected status information according to predefined rules to identify potential abnormal nodes;

[0057] Send a detailed status query request to the potential abnormal node to obtain more specific status information for further confirmation;

[0058] After the abnormal node responds to the status query request, it feeds back the detailed status information to the central monitoring server;

[0059] The central monitoring server conducts in-depth analysis based on the fed-back detailed status information to determine whether there is a node failure;

[0060] After confirming the node failure, the central monitoring server notifies the relevant operation and maintenance personnel and generates a fault handling task;

[0061] The operation and maintenance personnel perform diagnosis and repair operations on the faulty node according to the fault handling task provided by the central monitoring server;

[0062] After the faulty node returns to the normal state, the operation and maintenance personnel update the node status information on the central monitoring server and close the fault handling task.

[0063] Further, the data aggregation step includes a data synchronization algorithm.

[0064] Further, sending a processing result confirmation signal includes applying a digital signature and a timestamp.

[0065] Further, it also includes using distributed ledger technology to record all data processing and transmission activities.

[0066] To achieve the above object, a second aspect of the present invention provides a smart city data distributed processing and transmission system, including:

[0067] Multi-level data routing nodes, each node is configured with a dedicated data processing unit and a short-term buffer;

[0068] Geographic information system, used to determine the physical locations and data requirements of each subsystem;

[0069] Data traffic prediction module, which establishes a data traffic prediction model based on historical data and real-time monitoring data;

[0070] Data encapsulation module, used to segment and encapsulate the received data stream and assign a processing mark;

[0071] Data transmission module, used to parallelly transmit the encapsulated data segments to the specified routing node for preliminary processing;

[0072] A data aggregation module, configured to re-aggregate the processing results according to data tags;

[0073] A data restoration module, configured to restore the data content after the target subsystem receives the aggregated data;

[0074] A status monitoring agent, deployed on each node, configured to collect and report the current status information of the node;

[0075] A global transaction coordinator, configured to coordinate the consistency verification and fault handling among nodes.

[0076] (3) Advantageous effects

[0077] Compared with the prior art, a method and system for distributed processing and transmission of smart city data provided by the present invention can predict future data requirements according to real-time and historical data traffic, and optimize data access points and routing paths accordingly. By adjusting the data flow direction and processing resource allocation in real time, the data processing speed can be significantly improved, the system latency can be reduced, and data security and privacy protection can be enhanced.

[0078] Specifically, the exponential smoothing method is used to predict the data traffic, and the data access points and routing paths are dynamically adjusted according to the prediction results to achieve a fast response to data mobility; the weighted shortest path algorithm is used to dynamically select the best data access points to ensure that the data is transmitted through the most effective paths, reducing network congestion and improving transmission efficiency; the mean square error is used to evaluate the accuracy of the prediction model, and the prediction model parameters are dynamically updated according to the actual data traffic to improve the prediction accuracy; the dynamic programming algorithm is used to optimize the connection structure between the main routing node and the auxiliary routing nodes according to the real-time changes of the data traffic, enhancing the stability and elasticity of the network. Description of the drawings

[0079] Figure 1 It is a flowchart of a method for distributed processing and transmission of smart city data disclosed in this application.

[0080] Figure 2 It is a configuration diagram of multi-level data routing nodes disclosed in this application.

[0081] Figure 3 It is a flowchart of data traffic prediction disclosed in this application.

[0082] Figure 4 It is a schematic diagram of data encapsulation and transmission disclosed in this application.

[0083] Figure 5 It is a flowchart of data processing and re-aggregation disclosed in this application.

[0084] Figure 6Schematic diagram of data restoration and confirmation signal transmission disclosed in this application.

[0085] Figure 7 Node status monitoring diagram disclosed in this application.

[0086] Figure 8 Global transaction coordination flowchart disclosed in this application.

[0087] Figure 9 Schematic diagram of data synchronization algorithm disclosed in this application.

[0088] Figure 10 Schematic diagram of digital signature and timestamp application disclosed in this application.

[0089] Figure 11 Schematic diagram of distributed ledger technology application disclosed in this application.

[0090] Figure 12 Framework diagram of a smart city data distributed processing and transmission system disclosed in this application. Detailed implementation manners

[0091] The present invention will be described in detail below with reference to the accompanying drawings. The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0092] As Figure 1 、 Figure 2 shown, the present invention provides a smart city data distributed processing and transmission method, and the method includes the following steps:

[0093] Step S1, using a geospatial distribution algorithm, configure multi-level data routing nodes between urban subsystems, and each node is equipped with a dedicated data processing unit and a short-term buffer;

[0094] First, using a geospatial distribution algorithm, according to the geographic information system (GIS) data of the city, configure multi-level data routing nodes for each urban subsystem (such as traffic management, public security monitoring, etc.). Each node is not only equipped with a dedicated data processing unit for processing the data transmitted to the node, but also has a short-term buffer to temporarily store data to cope with sudden large traffic, ensuring the continuity and stability of data processing. For example, in the urban traffic network, main routing nodes can be set up in areas with high traffic, supplemented by multiple auxiliary nodes to efficiently manage and schedule vehicle information.

[0095] Step S2: Based on predictive analysis technology, evaluate the type and scale of the data traffic to be processed, and dynamically adjust the data access points and routing paths of each subsystem;

[0096] Next, use predictive analysis technology, such as machine learning models, to evaluate the type and scale of the data traffic that the urban subsystems will process in the next period of time. Based on these prediction results, the system will dynamically adjust the data access points and routing paths of each subsystem, optimize the data flow direction, and reduce latency. This enables anticipating peak hours and adjusting the routing before that to avoid congestion.

[0097] Step S3: Segment and encapsulate the received data stream, assign an independent processing tag to each data segment, and map them to different data routing nodes;

[0098] Step S4: After the data encapsulation is completed, transmit the encapsulated data segments in parallel to the specified routing nodes for preliminary processing;

[0099] Step S5: After each routing node completes the preliminary data processing, re-aggregate the processing results according to the data tags, and transmit the aggregated data to the target subsystem;

[0100] As Figure 4 、 Figure 5 shown, after the data stream is received, the system segments and encapsulates it, assigns independent processing tags to each data segment and distributes them to the corresponding routing nodes. For example, a certain surveillance video stream may be divided into multiple segments, and each segment processes different sequences of video frames. Then, these data segments are transmitted in parallel to the specified nodes for preliminary processing, such as data compression, format conversion, etc. After the processing is completed, according to the tags of each segment, the system re-aggregates these processing results to ensure the integrity and consistency of the data.

[0101] Step S6: After the target subsystem receives the aggregated data, restore the data content and update the node processing status information;

[0102] Step S7: After the data is restored, the target subsystem sends a processing result confirmation signal to the data sender.

[0103] As Figure 6 shown, after the data is processed and aggregated, the target subsystem receives and restores the data content, and at the same time updates the node processing status information for subsequent data analysis or real-time response. After completing these, the target subsystem will send a processing result confirmation signal to the data sender to ensure the closed-loop control of data transmission and processing.

[0104] Through the above steps, the present invention can ensure the efficient and stable transmission and processing of data in the smart city, respond to various demands in urban operations, such as real-time traffic scheduling, emergency event response, etc., and significantly improve the intelligent and automated level of urban management.

[0105] In step S1, as Figure 2 shown, the specific steps for configuring multi-level data routing nodes among the subsystems of a city include:

[0106] First, analyze and determine the specific physical locations of each subsystem (such as traffic control centers, public security monitoring systems, etc.) and their data requirements in the city's Geographic Information System (GIS). This is to understand which areas require more data support and the types of data (such as videos, sensor data, etc.). Then, based on the geospatial distribution algorithm, divide the city into different regions, and set up a main routing node and multiple auxiliary routing nodes in each region according to the data requirements. For example, in a traffic-intensive area, set up a main node to handle a large amount of vehicle and traffic signal data. Select the optimal locations for each main routing node, usually the central areas with the highest data requirements, and deploy data processing units with high processing capabilities and short-term buffer areas at these locations to cope with data surges during peak hours. Set up multiple data access points according to the data traffic characteristics and communication frequencies of each subsystem, and ensure that these access points can effectively connect to the main routing nodes. This can reduce delays and losses during data transmission. Through network topology analysis, optimize the connection relationship between the main routing node and the auxiliary routing nodes to ensure that data can flow efficiently and stably between the nodes. Establish efficient data interaction channels between the main routing node and the auxiliary routing nodes, and these channels are the main paths for data to flow within each region of the city. Finally, uniformly number all the routing nodes, and record these numbers and the node configuration information in the central control system.

[0107] Through the above steps, it can be ensured that the data processing system of the smart city can not only meet the daily data processing requirements but also respond quickly in the face of emergencies, effectively manage and analyze a large amount of urban data, and improve the intelligent level of urban management.

[0108] In step S2, as Figure 3 shown, the steps of the method for dynamically adjusting the data access points and routing paths of the smart city based on predictive analysis technology include:

[0109] First, based on historical data and real-time monitoring data, such as past traffic flow records, weather conditions, large event arrangements, etc., establish a data traffic prediction model. The purpose of this model is to predict the data traffic of each subsystem (such as the traffic system, security monitoring system, etc.) in the future period by analyzing past trends and current situations. For example, if it is predicted that the traffic volume will increase significantly on weekends, the model will predict the corresponding increase in data traffic during that period. Then, use this model to regularly generate the predicted values of the future data traffic of each subsystem.

[0110] Based on the predicted data traffic, calculate the optimal data access point locations for each subsystem. These locations are the best points for data to enter the network, which can maximize the data processing speed and efficiency. Then, adjust the actual data access point settings according to these calculation results to ensure that data can enter the processing network through the optimal path during the predicted high-traffic periods. To ensure the accuracy of the prediction and the effectiveness of the model, the system continuously compares the deviation between historical data and predicted data and updates the parameters of the prediction model based on these deviations.

[0111] Finally, based on the geospatial distribution algorithm and combined with the latest data traffic prediction results, re-evaluate and optimize the connection relationship between the main routing node and the auxiliary routing nodes. If the data traffic in a certain area increases, the system can add more auxiliary routing nodes or adjust the connection method of the existing nodes to cope with the increased data processing requirements.

[0112] In step S2, for the method of dynamically adjusting the smart city data access points and routing paths based on prediction analysis technology, the specific algorithm calculation formula is as follows:

[0113] Step A: Use the exponential smoothing method to predict the future data traffic of each subsystem. The formula is:

[0114] F t+1 =αD t +(1-α)F t

[0115] Where, F t+1 is the predicted value for the next time step, D t is the observed value for the current time step, F t is the predicted value for the current time step, and α is the smoothing coefficient, with a value range of 0 < α < 1;

[0116] Step B: Calculate the optimal data access point locations for each subsystem and use the weighted shortest path algorithm to optimize the data transmission path. The formula is:

[0117]

[0118] Where, L opt is the optimal access point location, w i is the traffic weight from node p i to the predetermined access point p opt , and d(p i ,p opt ) is the distance from node pi to p opt ;

[0119] Step C: Compare the deviation between historical data and predicted data and use the mean square error to update the prediction model parameters. The formula is:

[0120]

[0121] Among them, D t is the actual observed value, F t is the corresponding predicted value, and N is the number of observation points;

[0122] Step D: Combine the prediction results and data traffic, and use the dynamic programming algorithm to evaluate and optimize the connection relationship between the main routing node and the auxiliary routing node.

[0123] Using the exponential smoothing method to predict data traffic can well adapt to the changes of data over time. Compared with traditional static or periodic predictions, it can reflect the trend changes of data traffic more real-time; the algorithm dynamically adapts to the weights of historical data and real-time data by adjusting the smoothing coefficient α, so that it can more accurately predict the upcoming data traffic, thereby adjusting the data routing strategy in advance and reducing data congestion and latency.

[0124] In this embodiment, the weighted shortest path algorithm not only finds the shortest distance, but also assigns weights in combination with the size of data traffic to ensure that data can be efficiently transmitted through the optimal path.

[0125] The mean square error (MSE) is used to evaluate the accuracy of the prediction model in real time. By adjusting the prediction parameters in real time, it is ensured that the prediction model is updated at any time to adapt to the changes of data traffic. Traditional data processing systems often rely on fixed parameter settings, while this method allows the system to dynamically adjust the prediction model according to the changes of actual data traffic. This not only enhances the adaptability of the system, but also reduces the prediction error and improves the accuracy of decision-making.

[0126] In addition, the connections of nodes in the traditional scheme are often preset and fixed, while this method allows the node connections to be dynamically adjusted according to the current network state and data traffic prediction, greatly improving the elasticity and stability of the network.

[0127] In the smart city data distributed processing and transmission method, by introducing transaction management and data consistency mechanisms, the reliability of data processing and the robustness of the system are improved. Specifically, a consistency verification module is deployed on each data node to record and monitor each data processing request and its response received, ensuring the transparency and traceability of data processing. In addition, a unique transaction identifier is generated for each data segment transmitted and processed, and this identifier is attached to the corresponding data segment, so as to track the flow and state changes of data during the entire data processing process.

[0128] Such as Figure 8As shown, before data transmission, the source node stores a copy of the data in a short-term buffer, retaining the initial state of the data change, which helps to roll back to the initial state in case of errors or data inconsistencies during processing. After the data segment is transmitted to the target node, the target node processes the data segment according to the transaction identifier and records the processing result in the transaction log in real time. By establishing a global transaction coordinator, it is possible to poll the transaction logs of each node, collect and compare the processing results of all nodes under the current transaction identifier, and ensure that the data processing status is consistent across all nodes.

[0129] If the global transaction coordinator detects an inconsistent data processing status during the comparison process, it notifies the relevant nodes to roll back to the initial state of the data and resynchronize the data segment. For transactions that have been completed and are consistent, the global transaction coordinator sends an acknowledgment signal to each node, instructing them to update the data status to the final determined value. Finally, after receiving the acknowledgment signal, each node writes the corresponding transaction identifier and the final data status to the permanent storage area for future data retrieval and verification.

[0130] As Figure 7 shown, when dealing with the states of nodes in the processing system. First, state monitoring agents are deployed on each node, and these agents are responsible for collecting and reporting the current state information of the nodes. Each monitoring agent is equipped with a unique identifier that associates it with the corresponding node to ensure the correct attribution and tracking of information.

[0131] The state monitoring agents periodically send heartbeat signals containing the unique identifier and the node state information to the central monitoring server. Such signals help the central monitoring system continuously track the health status of each node. The central monitoring server receives and parses these heartbeat signals, records the node state information, and analyzes this information according to predefined rules to identify nodes that exhibit potential anomalies.

[0132] Once potential anomalous nodes are identified, the central monitoring server sends detailed status query requests to these nodes to obtain more specific status information for further confirmation. After the anomalous nodes respond to the query requests, they feedback the detailed status information to the central monitoring server. The server conducts in-depth analysis based on these detailed feedbacks to determine whether the nodes actually have faults.

[0133] After confirming the fault, the central monitoring server notifies the relevant operation and maintenance personnel and generates a processing task for this fault. The operation and maintenance personnel perform diagnostic and repair operations on the faulty nodes based on the fault processing tasks provided by the server. Once the faulty nodes return to the normal state, the operation and maintenance personnel update the node state information on the central monitoring server and close the fault processing task, thus ensuring the efficient operation of the system and the continuous availability of services.

[0134] As Figures 9 - 11 , the data aggregation step includes a data synchronization algorithm. Sending a processing result confirmation signal includes applying a digital signature and a timestamp. It also includes using distributed ledger technology to record all data processing and transmission activities.

[0135] To further demonstrate the superiority of a method for distributed processing and transmission of smart city data provided by the present invention, the present invention also provides a system for distributed processing and transmission of smart city data, as Figure 12 shown, the system includes:

[0136] Multi-level data routing nodes, each node is configured with a dedicated data processing unit and a short-term buffer;

[0137] A geographic information system for determining the physical locations and data requirements of each subsystem;

[0138] A data traffic prediction module that establishes a data traffic prediction model based on historical data and real-time monitoring data;

[0139] A data encapsulation module for segmenting and encapsulating the received data stream and assigning a processing tag;

[0140] A data transmission module for parallelly transmitting the encapsulated data segments to a specified routing node for preliminary processing;

[0141] A data aggregation module for re-aggregating the processing results according to the data tags;

[0142] A data restoration module for restoring the data content after the target subsystem receives the aggregated data;

[0143] A status monitoring agent deployed on each node for collecting and reporting the current status information of the node;

[0144] A global transaction coordinator for coordinating consistency verification and fault handling among nodes.

[0145] The above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A distributed processing and transmission method for smart city data, characterized in that: The steps include: Using geospatial distribution algorithms, multi-level data routing nodes are configured between urban subsystems, with each node equipped with a dedicated data processing unit and short-term buffer area; Based on predictive analysis technology, the data traffic type and scale to be processed are evaluated, and the data access points and routing paths of each subsystem are dynamically adjusted; The received data stream is encapsulated in segments, each segment is given an independent processing tag, and mapped to different data routing nodes; After data encapsulation is completed, the encapsulated data segments are transmitted in parallel to the designated routing nodes for preliminary processing; After each routing node completes the preliminary data processing, it re-aggregates the processing results according to the data tags and transmits the aggregated data to the target subsystem; After receiving the aggregated data, the target subsystem restores the data content and updates the node processing status information; After the data is restored, the target subsystem sends a processing result confirmation signal to the data sender.

2. A method for distributed processing and transmission of smart city data according to claim 1, characterized in that: The specific steps for configuring multi-level data routing nodes between various subsystems in the city include: Determine the physical location and data requirements of each subsystem in the urban geographic information system; Different regions are divided based on the geographic spatial distribution algorithm, and each region has a main routing node and several auxiliary routing nodes; Select the optimal location for each main routing node and deploy a dedicated data processing unit and short-term buffer area at that location; According to the data flow characteristics and communication frequency of each subsystem, multiple data access points are set up and connected to the corresponding main routing nodes; Optimize the connection relationship between each main routing node and the auxiliary routing nodes it covers through network topology analysis; Establish an efficient data interaction channel between the main routing node and the auxiliary routing node to realize a multi-level data routing structure; All routing nodes are uniformly numbered and the numbering information is recorded in the central control system.

3. A method for distributed processing and transmission of smart city data according to claim 1, characterized in that: The method for dynamically adjusting smart city data access points and routing paths based on predictive analysis technology includes the following steps: Based on historical data and real-time monitoring data, establish a data flow prediction model for each subsystem of the city; Using the data traffic prediction model, regularly generating future data traffic prediction values ​​for each subsystem; Calculate the optimal data access point location for each subsystem based on the predicted future data traffic of each subsystem; Adjust the setting of the actual data access point according to the calculated optimal data access point position; Compare the deviation between historical data and predicted data, and update the parameters of the data flow prediction model; Based on the geographic space distribution algorithm and combined with the data traffic prediction results, the connection relationship between the main routing node and the auxiliary routing node is re-evaluated and selected.

4. A method for distributed processing and transmission of smart city data according to claim 3, characterized in that: A method for dynamically adjusting smart city data access points and routing paths based on predictive analysis technology. The specific algorithm calculation formula is as follows: Step A: Use the exponential smoothing method to predict the future data flow of each subsystem. The formula is: F t+1 =αD t +(1-α)F t Among them, F t+1 is the predicted value for the next time step, D t is the observed value at the current time step, F t is the predicted value of the current time step, α is the smoothing coefficient, and its value range is 0<α<1; Step B: Calculate the optimal data access point location for each subsystem and use the weighted shortest path algorithm to optimize the data transmission path. The formula is: Among them, L opt is the optimal access point location, w i is from node p i To the pre-determined access point p opt The traffic weight, d(p i ,p opt ) is the node pi to p opt distance; Step C: Compare the deviation between historical data and predicted data, and use the mean square error to update the prediction model parameters. The formula is: Among them, D t is the actual observed value, F t is the corresponding predicted value, N is the number of observation points; Step D: Combine the prediction results and data traffic, and use a dynamic programming algorithm to evaluate and optimize the connection relationship between the main routing node and the auxiliary routing node.

5. The method for distributed processing and transmission of smart city data according to claim 1, characterized in that: The method further comprises: Deploy a consistency verification module on each node to record and monitor received data processing requests and responses; Generate a unique transaction identifier for each transmitted and processed data segment and append the identifier to the corresponding data segment; Before the data segment is transmitted to the target node, the source node stores the data copy in a short-term buffer to preserve the initial state of the data change; After receiving the data segment, each target node processes the data segment according to the transaction identifier and records the processing status in the transaction log in real time; Establish a global transaction coordinator to collect the processing results of all nodes under the current transaction identifier by polling the transaction log of each node; The global transaction coordinator compares the processing status of each node based on the collected processing results to see if there is any inconsistency; When an inconsistency is detected, the global transaction coordinator notifies the relevant nodes to roll back to the initial state and resynchronize the data segment; For completed consistent transactions, the global transaction coordinator sends a confirmation signal to each node, instructing each node to update its data status to the final determined value; After receiving the confirmation signal, each node writes the corresponding transaction identifier and final data status into the permanent storage area for later retrieval and verification.

6. A method for distributed processing and transmission of smart city data according to claim 1, characterized in that: The method further includes monitoring the status of each node, and the step of monitoring the status of each node includes: Deploy a status monitoring agent on each node to collect and report the current status information of the node; Configure a unique identifier for each status monitoring agent and associate the identifier with the corresponding node; The status monitoring agent periodically sends a heartbeat signal containing a unique identifier and node status information to the central monitoring server; The central monitoring server receives and analyzes the heartbeat signals of each node and records the status information of the corresponding node; The central monitoring server analyzes the collected status information according to predefined rules and identifies potential abnormal nodes; Send detailed status query requests to potential abnormal nodes to obtain more specific status information for further confirmation; After the abnormal node responds to the status query request, it feeds back the detailed status information to the central monitoring server; The central monitoring server conducts in-depth analysis based on the detailed status information fed back to determine whether there is a node failure; After confirming the node failure, the central monitoring server notifies the relevant operation and maintenance personnel and generates a fault handling task; The operation and maintenance personnel diagnose and repair the faulty nodes according to the fault handling tasks provided by the central monitoring server; After the faulty node returns to normal, the operation and maintenance personnel update the node status information on the central monitoring server and close the fault handling task.

7. A method for distributed processing and transmission of smart city data according to claim 1, characterized in that: The data aggregation step includes a data synchronization algorithm.

8. The method for distributed processing and transmission of smart city data according to claim 1, characterized in that: Sending a processing result confirmation signal includes applying a digital signature and a timestamp.

9. A method for distributed processing and transmission of smart city data according to claim 1, characterized in that: It also includes the use of distributed ledger technology to record all data processing and transmission activities.

10. A distributed processing and transmission system for smart city data, characterized in that: include: Multi-level data routing nodes, each node is equipped with a dedicated data processing unit and short-term buffer area; Geographic Information System, used to determine the physical location and data requirements of each subsystem; Data traffic prediction module, which establishes a data traffic prediction model based on historical data and real-time monitoring data; The data encapsulation module is used to encapsulate the received data stream into segments and assign processing marks; A data transmission module for transmitting encapsulated data segments in parallel to a designated routing node for preliminary processing; A data aggregation module, used to re-aggregate processing results according to data tags; The data restoration module is used to restore the data content after the target subsystem receives the aggregated data; Status monitoring agent, deployed on each node, to collect and report the current status information of the node; The global transaction coordinator is used to coordinate consistency verification and fault handling between nodes.

Images