Smart city data transmission optimization system based on distributed architecture

Abstract

The invention provides a distributed architecture-based smart city data transmission optimization system, which comprises a data acquisition module, a distributed storage module, a transmission optimization module and an interaction control module, and is characterized in that the data acquisition module is used for acquiring smart city data, the distributed storage module is used for storing the smart city data, and the transmission optimization module is used for transmitting the smart city data to the interaction control module. The transmission optimization module is used for transmitting and scheduling smart city data among nodes, and the interaction control module is used for displaying a data state and controlling a transmission strategy; according to the system, bandwidth resources are controlled through the transmission optimization module, the circulation efficiency of data among the nodes can be effectively improved, and better operation of the smart city is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing, and in particular to a smart city data transmission optimization system based on a distributed architecture. Background Art

[0002] With the rapid development of smart city construction, the demand for real-time collection, storage and transmission of urban data is growing. Intelligent transportation, environmental monitoring, public safety and other fields rely on the efficient transmission of massive data to support intelligent decision-making and automated management. A large amount of data is generated in smart cities, so a transmission optimization system is needed to analyze data traffic in real time, intelligently adjust transmission strategies, reduce network congestion, improve bandwidth utilization, optimize data transmission paths and flow control, and ensure that key data arrives at the target node first, so as to better support the needs of smart city application scenarios.

[0003] The foregoing discussion of the background art is intended only to facilitate an understanding of the present invention. This discussion does not acknowledge or admit that any of the material referred to is part of the common general knowledge.

[0004] Now many transmission optimization systems have been developed. After a lot of searching and reference, it is found that the existing transmission optimization systems are such as the system disclosed in the publication number CN106961473B. These system methods generally include: using the vehicle terminal to collect vehicle data, transmitting it to the control center through the Internet when the network is normal, storing it in the data buffer when the network fails, and transmitting the data in the data buffer to the control center after recovery, and the scheduling transmitter controls the amount of data sent each time. Among them: the scheduling transmitter obtains the time used for each data transmission to calculate the average time T each time; and obtains the time t and data volume n used for the most recent data transmission; calculates the optimal data volume for the next transmission; determines whether the data in the data buffer is greater than m, if so, controls the next data volume sent to the Internet to be m, if not, all the data in the data buffer is sent to the Internet. However, this system needs to centralize the generated data to the control center for processing, there is a large amount of unnecessary data transmission, and there is a large room for improvement in data transmission efficiency. Summary of the invention

[0005] The purpose of the present invention is to propose a smart city data transmission optimization system based on a distributed architecture to address the existing deficiencies.

[0006] The present invention adopts the following technical solution:

[0007] A smart city data transmission optimization system based on a distributed architecture, comprising a data acquisition module, a distributed storage module, a transmission optimization module and an interactive control module;

[0008] The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategies;

[0009] The data acquisition module includes a sensor management unit, a data preprocessing unit and an edge computing unit. The sensor management unit is used to manage sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data.

[0010] The distributed storage module includes a storage node management unit, a data shard storage unit and a data reading management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store the smart city data of each node, and the data reading management unit controls and manages the reading of stored data;

[0011] The transmission optimization module includes an intelligent scheduling unit, a flow control unit and a compression coding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path. The flow control unit is used to monitor the data flow in real time and perform flow control. The compression coding unit is used to compress the smart city data to be transmitted.

[0012] The interactive control module includes a data visualization unit, a permission management unit and a remote control unit. The data visualization unit is used to visualize the data status of the smart city, the permission management unit is used to control and manage the operator's permissions, and the remote control unit is used to remotely configure transmission parameters.

[0013] Furthermore, the traffic control unit includes a bandwidth management processor, a congestion prediction processor and an allocation calculation processor, wherein the bandwidth management processor is used to manage the communication bandwidth between nodes, the congestion prediction processor is used to predict virtual congestion conditions, and the allocation calculation processor is used to allocate bandwidth resources to existing communication tasks.

[0014] Furthermore, the bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes and the idle bandwidth resources. When a communication task is completed, this part of the bandwidth resources is released and counted into the idle bandwidth resources, and a change signal is sent to the congestion prediction processor.

[0015] Further, after receiving the change signal, the congestion prediction processor predicts the congestion situation according to the following formula:

[0016]

[0017] Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i represents the number of tasks in the i-th window, m is the number of windows, and Y is the amount of predicted tasks;

[0018] The congestion prediction processor sets the necessary bandwidth resource Z0 based on the predicted task volume, and calculates the allocatable resource volume Z according to the following formula: a :

[0019] Z a =Z u -Z0;

[0020] Among them, Z u The amount of idle bandwidth resources.

[0021] Furthermore, the allocation calculation processor calculates the redistribution weight k of each current communication task according to the following formula:

[0022]

[0023] Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant;

[0024] The allocation calculation processor calculates the new bandwidth allocation amount ΔZ(i) of the i-th communication task according to the following formula:

[0025]

[0026] Among them, k i represents the redistribution weight of the i-th communication task, and n is the number of communication tasks.

[0027] The beneficial effects achieved by the present invention are:

[0028] This system adopts a distributed structure to transmit data directly between nodes, which greatly reduces the amount of data transmission. At the same time, during the data transmission process, by optimizing the allocation of bandwidth resources, it can effectively improve the waiting situation of communication tasks and provide a basis for the smooth operation of smart cities.

[0029] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall structural framework of the present invention;

[0031] Figure 2This is a schematic diagram of the data acquisition module of the present invention;

[0032] Figure 3 This is a schematic diagram of the distributed storage module of the present invention;

[0033] Figure 4 This is a schematic diagram of the transmission optimization module of the present invention;

[0034] Figure 5 This is a schematic diagram of the interactive control module of the present invention;

[0035] Figure 6 This is a comparative data table of the application effects of the present invention. DETAILED DESCRIPTION

[0036] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0037] Embodiment 1.

[0038] This embodiment provides a smart city data transmission optimization system based on a distributed architecture, combined with Figure 1 , including data acquisition module, distributed storage module, transmission optimization module and interactive control module;

[0039] The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategies;

[0040] The data acquisition module includes a sensor management unit, a data preprocessing unit and an edge computing unit. The sensor management unit is used to manage sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data.

[0041] The distributed storage module includes a storage node management unit, a data shard storage unit and a data reading management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store the smart city data of each node, and the data reading management unit controls and manages the reading of stored data;

[0042] The transmission optimization module includes an intelligent scheduling unit, a flow control unit and a compression coding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path. The flow control unit is used to monitor the data flow in real time and perform flow control. The compression coding unit is used to compress the smart city data to be transmitted.

[0043] The interactive control module includes a data visualization unit, a permission management unit and a remote control unit. The data visualization unit is used to visualize the data status of the smart city, the permission management unit is used to control and manage the operator's permissions, and the remote control unit is used to remotely configure transmission parameters.

[0044] The traffic control unit includes a bandwidth management processor, a congestion prediction processor and an allocation calculation processor. The bandwidth management processor is used to manage the communication bandwidth between nodes, the congestion prediction processor is used to predict virtual congestion conditions, and the allocation calculation processor is used to allocate bandwidth resources to existing communication tasks.

[0045] The bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes and the idle bandwidth resources. When a communication task is completed, this part of the bandwidth resources is released and counted into the idle bandwidth resources, and a change signal is sent to the congestion prediction processor.

[0046] After receiving the change signal, the congestion prediction processor predicts the congestion situation according to the following formula:

[0047]

[0048] Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i represents the number of tasks in the i-th window, m is the number of windows, and Y is the amount of predicted tasks;

[0049] The congestion prediction processor sets the necessary bandwidth resource Z0 based on the predicted task volume, and calculates the allocatable resource volume Z according to the following formula: a :

[0050] Z a =Z u -Z0;

[0051] Among them, Z uThe amount of idle bandwidth resources.

[0052] The allocation calculation processor calculates the redistribution weight k of each current communication task according to the following formula:

[0053]

[0054] Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant;

[0055] The allocation calculation processor calculates the new bandwidth allocation amount ΔZ(i) of the i-th communication task according to the following formula:

[0056]

[0057] Among them, k i represents the redistribution weight of the i-th communication task, and n is the number of communication tasks.

[0058] Embodiment 2.

[0059] This embodiment includes all the contents of the first embodiment, and provides a smart city data transmission optimization system based on a distributed architecture, including a data acquisition module, a distributed storage module, a transmission optimization module and an interactive control module;

[0060] The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategies;

[0061] Combination Figure 2 The data acquisition module includes a sensor management unit, a data preprocessing unit and an edge computing unit. The sensor management unit is used to manage sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data.

[0062] Combination Figure 3 The distributed storage module includes a storage node management unit, a data shard storage unit and a data reading management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store the smart city data of each node, and the data reading management unit controls and manages the reading of stored data;

[0063] Combination Figure 4The transmission optimization module includes an intelligent scheduling unit, a flow control unit and a compression coding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path. The flow control unit is used to monitor the data flow in real time and perform flow control. The compression coding unit is used to compress the smart city data to be transmitted.

[0064] Combination Figure 5 The interactive control module includes a data visualization unit, a permission management unit and a remote control unit. The data visualization unit is used to visualize the data status of the smart city, the permission management unit is used to control and manage the operator's permissions, and the remote control unit is used to remotely configure the transmission parameters;

[0065] The sensor management unit includes a heterogeneous protocol adapter, a dynamic energy consumption controller and a fault diagnosis processor, wherein the heterogeneous protocol processor is used to automatically identify and convert the communication protocol of the sensor, the dynamic energy consumption controller is used to adjust the sampling frequency of the sensor, and the fault diagnosis processor is used to detect the abnormal state of the sensor in real time and perform device switching;

[0066] The data preprocessing unit includes a stream cleaning processor, a format standardization processor and a metadata tagging processor, wherein the stream cleaning processor is used to filter noise data, the format standardization processor is used to unify data of different formats into the same format, and the metadata tagging processor is used to add tag information to the collected data;

[0067] The edge computing unit includes a feature extraction processor, a dynamic extraction controller and a cache management processor, wherein the feature extraction processor is used to process metadata to extract feature information, the dynamic extraction controller adaptively adjusts the feature extraction frequency based on the value density of the metadata, and the cache management processor is used to control and manage the metadata and feature information that need to be stored;

[0068] The storage node management unit includes a node information register, a communication connection processor and a status monitoring processor, wherein the node information register is used to store node information, the communication connection processor establishes a connection relationship between the sensor and the node, and between the nodes based on the node information, and the status monitoring processor is used to monitor the storage status and communication status of the node;

[0069] The data sharding storage unit includes an intelligent sharding decision maker, a cross-domain encryption processor and a sharding information register, wherein the intelligent sharding decision maker is used to shard the metadata and feature information to be stored, the cross-domain encryption processor is used to encrypt the sharded data, and the sharding information register is used to store the encrypted sharding information;

[0070] The data reading management unit includes a demand parsing processor, a content index processor and a data output processor, wherein the demand parsing processor is used to parse the data demand, the content index processor searches for data content based on the parsing result, and the data output processor is used to output the found data to the outside;

[0071] The intelligent scheduling unit includes a path retrieval processor, a multi-objective evaluation processor and a path execution processor, wherein the path retrieval processor is used to retrieve the communication path to the target node, the multi-objective evaluation processor is used to evaluate each path, and the path execution processor selects a path to execute data transmission based on the evaluation result;

[0072] The flow control unit includes a bandwidth management processor, a congestion prediction processor and an allocation calculation processor, wherein the bandwidth management processor is used to manage the communication bandwidth between nodes, the congestion prediction processor is used to predict the virtual congestion situation, and the allocation calculation processor is used to allocate bandwidth resources to existing communication tasks;

[0073] The bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes and the idle bandwidth resources. When a communication task is completed, this part of the bandwidth resources is released and counted into the idle bandwidth resources, and a change signal is sent to the congestion prediction processor at the same time;

[0074] After receiving the change signal, the congestion prediction processor predicts the congestion situation according to the following formula:

[0075]

[0076] Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i represents the number of tasks in the i-th window, m is the number of windows, and Y is the amount of predicted tasks;

[0077] The congestion prediction processor sets the necessary bandwidth resource Z0 based on the predicted task volume, and calculates the allocatable resource volume Z according to the following formula: a :

[0078] Z a =Z u -Z0;

[0079] Among them, Z u is the amount of idle bandwidth resources;

[0080] When Z a When Z is greater than 0, the congestion prediction processor a Send to the assigned computing processor;

[0081] The allocation calculation processor calculates the redistribution weight k of each current communication task according to the following formula:

[0082]

[0083] Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant;

[0084] The allocation calculation processor calculates the new bandwidth allocation amount ΔZ(i) of the i-th communication task according to the following formula:

[0085]

[0086] Among them, k i represents the redistribution weight of the i-th communication task, and n is the number of communication tasks;

[0087] The compression coding unit includes an intelligent coding selector, a lossless compression processor and a hierarchical compression controller, wherein the intelligent coding selector automatically matches the corresponding compression algorithm based on data features, the lossless compression processor is used to compress the data content to be transmitted, and the hierarchical compression controller is used to hierarchically process the data content to be transmitted;

[0088] The data visualization unit includes a three-dimensional rendering accelerator, an intelligent association analyzer and a picture fusion processor, wherein the three-dimensional rendering processor is used to dynamically render the data, the intelligent association processor is used to associate the displayed data content with factors, and the picture fusion processor is used to perform picture superposition processing on multiple display contents;

[0089] The authority management unit includes an authority registration processor, a user verification processor and a function opening processor, wherein the authority registration processor is used to record the authority information of each user, the user verification processor is used to authenticate the user, and the function opening processor opens the corresponding control function based on the user's authority information;

[0090] The remote control unit includes a parameter editing processor, an update configuration processor and a control feedback processor, wherein the parameter editing processor is used to edit communication parameters, the update configuration processor is used to send the communication parameters to the corresponding node, and the control feedback processor is used to feed back the update result;

[0091] The i appearing in the above text is an ordinal number used to indicate a sequence number and has no actual meaning.

[0092] Some code information of this system is as follows:

[0093]

[0094]

[0095]

[0096] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention specification and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. A smart city data transmission optimization system based on distributed architecture, characterized in that: It includes data acquisition module, distributed storage module, transmission optimization module and interactive control module; The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategies; The data acquisition module includes a sensor management unit, a data preprocessing unit and an edge computing unit. The sensor management unit is used to manage sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data. The distributed storage module includes a storage node management unit, a data shard storage unit and a data reading management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store the smart city data of each node, and the data reading management unit controls and manages the reading of stored data; The transmission optimization module includes an intelligent scheduling unit, a flow control unit and a compression coding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path. The flow control unit is used to monitor the data flow in real time and perform flow control. The compression coding unit is used to compress the smart city data to be transmitted. The interactive control module includes a data visualization unit, a permission management unit and a remote control unit. The data visualization unit is used to visualize the data status of the smart city, the permission management unit is used to control and manage the operator's permissions, and the remote control unit is used to remotely configure transmission parameters.

2. A distributed architecture-based smart city data transmission optimization system as claimed in claim 1, characterized in that: The traffic control unit includes a bandwidth management processor, a congestion prediction processor and an allocation calculation processor. The bandwidth management processor is used to manage the communication bandwidth between nodes, the congestion prediction processor is used to predict virtual congestion conditions, and the allocation calculation processor is used to allocate bandwidth resources to existing communication tasks.

3. A distributed architecture-based smart city data transmission optimization system as claimed in claim 2, characterized in that: The bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes and the idle bandwidth resources. When a communication task is completed, this part of the bandwidth resources is released and counted into the idle bandwidth resources, and a change signal is sent to the congestion prediction processor.

4. A distributed architecture-based smart city data transmission optimization system as claimed in claim 3, characterized in that: After receiving the change signal, the congestion prediction processor predicts the congestion situation according to the following formula: Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i represents the number of tasks in the i-th window, m is the number of windows, and Y is the amount of predicted tasks; The congestion prediction processor sets the necessary bandwidth resource Z0 based on the predicted task volume, and calculates the allocatable resource volume Z according to the following formula: a : WITH a =Z u -Z0; Among them, Z u The amount of idle bandwidth resources.

5. A distributed architecture-based smart city data transmission optimization system as claimed in claim 4, characterized in that: The allocation calculation processor calculates the redistribution weight k of each current communication task according to the following formula: Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant; The allocation calculation processor calculates the new bandwidth allocation amount ΔZ(i) of the i-th communication task according to the following formula: Among them, k i represents the redistribution weight of the i-th communication task, and n is the number of communication tasks.

Images