A smart city data transmission optimization system based on a distributed architecture

Abstract

This invention provides a smart city data transmission optimization system based on a distributed architecture, comprising a data acquisition module, a distributed storage module, a transmission optimization module, and an interactive control module. The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategies. This system, through the transmission optimization module, controls bandwidth resources, effectively improving the efficiency of data flow between nodes and facilitating better operation of the smart city.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and more specifically to a smart city data transmission optimization system based on a distributed architecture. Background Technology

[0002] With the rapid development of smart city construction, the demand for real-time collection, storage, and transmission of urban data is increasing. Fields such as intelligent transportation, environmental monitoring, and public safety rely on the efficient transmission of massive amounts of data to support intelligent decision-making and automated management. Smart cities generate a large amount of data, so a transmission optimization system is needed to analyze data traffic in real time, intelligently adjust transmission strategies, reduce network congestion, improve bandwidth utilization, optimize data transmission paths and traffic control, and ensure that key data arrives at target nodes first, thereby better supporting the needs of smart city application scenarios.

[0003] The foregoing description of the background art is intended only to facilitate understanding of the invention. This description does not endorse or acknowledge any common general knowledge in the materials mentioned.

[0004] Many transmission optimization systems have been developed. Extensive research and reference have revealed existing systems such as the one disclosed in publication number CN106961473B. These systems generally involve: using an onboard terminal to collect vehicle data; transmitting it to the control center via the internet when the network is normal; storing the data in a data buffer when the network fails; and transmitting the data from the buffer to the control center after recovery. A scheduling transmitter controls the amount of data sent each time. Specifically, the scheduling transmitter obtains the time taken for each data transmission to calculate the average time T; it also obtains the time t and data volume n of the most recent transmission; calculates the optimal data volume for the next transmission; and determines whether the data in the buffer is greater than m. If so, it controls the amount of data sent to the internet to be m; otherwise, it sends all data in the buffer to the internet. However, this system requires centralized data transmission to the control center for further processing, resulting in a large amount of unnecessary data transmission and significant room for improvement in data transmission efficiency. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings by proposing a smart city data transmission optimization system based on a distributed architecture.

[0006] The present invention adopts the following technical solution:

[0007] A smart city data transmission optimization system based on a distributed architecture includes a data acquisition module, a distributed storage module, a transmission optimization module, and an interactive control module.

[0008] The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategy.

[0009] The data acquisition module includes a sensor management unit, a data preprocessing unit, and an edge computing unit. The sensor management unit is used to manage the sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data.

[0010] The distributed storage module includes a storage node management unit, a data shard storage unit, and a data read management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store smart city data for each node, and the data read management unit controls and manages the reading of the stored data.

[0011] The transmission optimization module includes an intelligent scheduling unit, a traffic control unit, and a compression encoding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path, the traffic control unit is used to monitor data traffic in real time and perform traffic regulation, and the compression encoding unit is used to compress the smart city data that needs to be transmitted.

[0012] The interactive control module includes a data visualization unit, a permission management unit, and a remote control unit. The data visualization unit is used to visualize the status of smart city data, the permission management unit is used to control and manage the operator's permissions, and the remote control unit is used to remotely configure transmission parameters.

[0013] Furthermore, the traffic control unit includes a bandwidth management processor, a congestion prediction processor, and an allocation calculation processor. The bandwidth management processor is used to manage the communication bandwidth between nodes, the congestion prediction processor is used to predict virtual congestion, and the allocation calculation processor is used to allocate bandwidth resources to existing communication tasks.

[0014] Furthermore, the bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes as well as the idle bandwidth resources. When a communication task is completed, the bandwidth resources are released and included in the idle bandwidth resources, and a change signal is sent to the congestion prediction processor.

[0015] Furthermore, after receiving a change signal, the congestion prediction processor predicts the congestion situation according to the following formula:

[0016]

[0017] Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i Let m represent the number of tasks in the i-th window, m be the number of windows, and Y be the number of predicted tasks.

[0018] The congestion prediction processor sets the necessary bandwidth resource Z0 based on the prediction task volume, and calculates the allocable resource volume Z according to the following formula. a :

[0019] Z a =Z u -Z0;

[0020] Among them, Z u This represents the amount of idle bandwidth resources.

[0021] Furthermore, the allocation calculation processor calculates the redistribution weight k for each communication task according to the following formula:

[0022]

[0023] Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant.

[0024] The allocation calculation processor calculates the new bandwidth allocation ΔZ(i) for the i-th communication task according to the following formula:

[0025]

[0026] Where, k i This represents the redistribution weight of the i-th communication task, where n is the number of communication tasks.

[0027] The beneficial effects achieved by this invention are:

[0028] This system adopts a distributed structure to transmit data directly between nodes, which greatly reduces the amount of data transmitted. At the same time, by optimizing the allocation of bandwidth resources during data transmission, the waiting situation of communication tasks can be effectively improved, providing a foundation for the smooth operation of smart cities.

[0029] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall structural framework of the present invention;

[0031] Figure 2This is a schematic diagram of the data acquisition module of the present invention;

[0032] Figure 3 This is a schematic diagram of the distributed storage module of the present invention;

[0033] Figure 4 This is a schematic diagram of the transmission optimization module of the present invention;

[0034] Figure 5 This is a schematic diagram of the interactive control module of the present invention;

[0035] Figure 6 This is a comparison data table of the application effects of the present invention. Detailed Implementation

[0036] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0037] Example 1.

[0038] This embodiment provides a smart city data transmission optimization system based on a distributed architecture, combined with... Figure 1 It includes a data acquisition module, a distributed storage module, a transmission optimization module, and an interactive control module;

[0039] The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategy.

[0040] The data acquisition module includes a sensor management unit, a data preprocessing unit, and an edge computing unit. The sensor management unit is used to manage the sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data.

[0041] The distributed storage module includes a storage node management unit, a data shard storage unit, and a data read management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store smart city data for each node, and the data read management unit controls and manages the reading of the stored data.

[0042] The transmission optimization module includes an intelligent scheduling unit, a traffic control unit, and a compression encoding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path, the traffic control unit is used to monitor data traffic in real time and perform traffic regulation, and the compression encoding unit is used to compress the smart city data that needs to be transmitted.

[0043] The interactive control module includes a data visualization unit, a permission management unit, and a remote control unit. The data visualization unit is used to visualize the status of smart city data, the permission management unit is used to control and manage the operator's permissions, and the remote control unit is used to remotely configure transmission parameters.

[0044] The traffic control unit includes a bandwidth management processor, a congestion prediction processor, and an allocation calculation processor. The bandwidth management processor manages the communication bandwidth between nodes, the congestion prediction processor predicts virtual congestion, and the allocation calculation processor allocates bandwidth resources to existing communication tasks.

[0045] The bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes as well as the idle bandwidth resources. When a communication task is completed, it releases these bandwidth resources and adds them to the idle bandwidth resources, while sending a change signal to the congestion prediction processor.

[0046] Upon receiving a change signal, the congestion prediction processor predicts the congestion situation according to the following formula:

[0047]

[0048] Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i Let m represent the number of tasks in the i-th window, m be the number of windows, and Y be the number of predicted tasks.

[0049] The congestion prediction processor sets the necessary bandwidth resource Z0 based on the prediction task volume, and calculates the allocable resource volume Z according to the following formula. a :

[0050] Z a =Z u -Z0;

[0051] Among them, Z uThis represents the amount of idle bandwidth resources.

[0052] The allocation calculation processor calculates the redistribution weight k for each communication task according to the following formula:

[0053]

[0054] Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant.

[0055] The allocation calculation processor calculates the new bandwidth allocation ΔZ(i) for the i-th communication task according to the following formula:

[0056]

[0057] Where, k i This represents the redistribution weight of the i-th communication task, where n is the number of communication tasks.

[0058] Example 2.

[0059] This embodiment includes all the contents of Embodiment 1, and provides a smart city data transmission optimization system based on a distributed architecture, including a data acquisition module, a distributed storage module, a transmission optimization module, and an interactive control module;

[0060] The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategy.

[0061] Combination Figure 2 The data acquisition module includes a sensor management unit, a data preprocessing unit, and an edge computing unit. The sensor management unit is used to manage the sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data.

[0062] Combination Figure 3 The distributed storage module includes a storage node management unit, a data shard storage unit, and a data read management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store smart city data for each node, and the data read management unit controls and manages the reading of the stored data.

[0063] Combination Figure 4The transmission optimization module includes an intelligent scheduling unit, a traffic control unit, and a compression encoding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path, the traffic control unit is used to monitor data traffic in real time and perform traffic regulation, and the compression encoding unit is used to compress the smart city data that needs to be transmitted.

[0064] Combination Figure 5 The interactive control module includes a data visualization unit, a permission management unit, and a remote control unit. The data visualization unit is used to visualize the status of smart city data, the permission management unit is used to control and manage the permissions of the operator, and the remote control unit is used to remotely configure transmission parameters.

[0065] The sensor management unit includes a heterogeneous protocol adapter, a dynamic power consumption controller, and a fault diagnosis processor. The heterogeneous protocol processor is used to automatically identify and convert the communication protocol of the sensor. The dynamic power consumption controller is used to adjust the sampling frequency of the sensor. The fault diagnosis processor is used to detect abnormal states of the sensor in real time and switch devices accordingly.

[0066] The data preprocessing unit includes a streaming cleaner, a format standardization processor, and a metadata tagging processor. The streaming cleaner is used to filter noisy data, the format standardization processor is used to unify data of different formats into the same format, and the metadata tagging processor is used to add tag information to the collected data.

[0067] The edge computing unit includes a feature extraction processor, a dynamic extraction controller, and a cache management processor. The feature extraction processor is used to process metadata to extract feature information. The dynamic extraction controller adaptively adjusts the feature extraction frequency based on the value density of the metadata. The cache management processor is used to control and manage the metadata and feature information that need to be stored.

[0068] The storage node management unit includes a node information register, a communication connection processor, and a status monitoring processor. The node information register is used to store node information. The communication connection processor establishes connection relationships between sensors and nodes, and between nodes, based on the node information. The status monitoring processor is used to monitor the storage status and communication status of the nodes.

[0069] The data sharding storage unit includes an intelligent sharding decision-maker, a cross-domain encryption processor, and a shard information register. The intelligent sharding decision-maker is used to shard the metadata and feature information to be stored. The cross-domain encryption processor is used to encrypt the sharded data. The shard information register is used to store the encrypted shard information.

[0070] The data reading management unit includes a demand parsing processor, a content indexing processor, and a data output processor. The demand parsing processor is used to parse and process data demands. The content indexing processor searches for data content based on the parsing results. The data output processor is used to output the searched data to the outside.

[0071] The intelligent scheduling unit includes a path retrieval processor, a multi-target evaluation processor, and a path execution processor. The path retrieval processor is used to retrieve the communication path of the target node. The multi-target evaluation processor is used to evaluate each path. The path execution processor selects a path to execute data transmission based on the evaluation results.

[0072] The traffic control unit includes a bandwidth management processor, a congestion prediction processor, and an allocation calculation processor. The bandwidth management processor is used to manage the communication bandwidth between nodes, the congestion prediction processor is used to predict virtual congestion, and the allocation calculation processor is used to allocate bandwidth resources to existing communication tasks.

[0073] The bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes and the idle bandwidth resources. When a communication task is completed, it releases this part of the bandwidth resources and adds it to the idle bandwidth resources, and at the same time sends a change signal to the congestion prediction processor.

[0074] Upon receiving a change signal, the congestion prediction processor predicts the congestion situation according to the following formula:

[0075]

[0076] Where T represents the current time period, α(T) represents the task change coefficient within the time period T, and n i Let m represent the number of tasks in the i-th window, m be the number of windows, and Y be the number of predicted tasks.

[0077] The congestion prediction processor sets the necessary bandwidth resource Z0 based on the prediction task volume, and calculates the allocable resource volume Z according to the following formula. a :

[0078] Z a =Z u -Z0;

[0079] Among them, Z u This refers to the amount of idle bandwidth resources.

[0080] When Z a When the value is greater than 0, the congestion prediction processor will... a Send to the assigned computing processor;

[0081] The allocation calculation processor calculates the redistribution weight k for each communication task according to the following formula:

[0082]

[0083] Where t0 represents the completion time of the communication task, t is the running time of the communication task, d is the amount of bandwidth resources allocated to the communication task, and ε is the offset constant.

[0084] The allocation calculation processor calculates the new bandwidth allocation ΔZ(i) for the i-th communication task according to the following formula:

[0085]

[0086] Where, k i This represents the redistribution weight of the i-th communication task, where n is the number of communication tasks.

[0087] The compression encoding unit includes an intelligent encoding selector, a lossless compression processor, and a hierarchical compression controller. The intelligent encoding selector automatically matches the corresponding compression algorithm based on data characteristics. The lossless compression processor is used to compress the transmitted data content. The hierarchical compression controller is used to process the transmitted data content in hierarchical manner.

[0088] The data visualization unit includes a 3D rendering accelerator, an intelligent correlation analyzer, and a screen fusion processor. The 3D rendering processor is used for dynamic rendering of data, the intelligent correlation processor is used for associating the displayed data content with factors, and the screen fusion processor is used for overlaying multiple displayed contents.

[0089] The permission management unit includes a permission registration processor, a user verification processor, and a function opening processor. The permission registration processor is used to record the permission information of each user, the user verification processor is used to verify the identity of the user, and the function opening processor opens the corresponding control function based on the user's permission information.

[0090] The remote control unit includes a parameter editing processor, an update configuration processor, and a control feedback processor. The parameter editing processor is used to edit communication parameters, the update configuration processor is used to send the new communication parameters to the corresponding nodes, and the control feedback processor is used to provide feedback on the update results.

[0091] The 'i' mentioned above is an ordinal number used to represent the sequence number and has no actual meaning.

[0092] The following is a partial code description of this system:

[0093]

[0094]

[0095]

[0096] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A smart city data transmission optimization system based on a distributed architecture, characterized in that, It includes a data acquisition module, a distributed storage module, a transmission optimization module, and an interactive control module; The data acquisition module is used to collect smart city data, the distributed storage module is used to store smart city data, the transmission optimization module is used to schedule the transmission of smart city data between nodes, and the interactive control module is used to display data status and control transmission strategy. The data acquisition module includes a sensor management unit, a data preprocessing unit, and an edge computing unit. The sensor management unit is used to manage the sensors that collect smart city data. The data preprocessing unit is used to perform basic preprocessing on the collected data. The edge computing unit is used to analyze and process the collected data. The distributed storage module includes a storage node management unit, a data shard storage unit, and a data read management unit. The storage node management unit is used to manage distributed node information, the data shard storage unit is used to store smart city data for each node, and the data read management unit controls and manages the reading of the stored data. The transmission optimization module includes an intelligent scheduling unit, a traffic control unit, and a compression encoding unit. The intelligent scheduling unit is used to dynamically adjust the data transmission path, the traffic control unit is used to monitor data traffic in real time and perform traffic regulation, and the compression encoding unit is used to compress the smart city data that needs to be transmitted. The interactive control module includes a data visualization unit, a permission management unit, and a remote control unit. The data visualization unit is used to visualize the status of smart city data. The permission management unit is used to control and manage the permissions of operators. The remote control unit is used to remotely configure transmission parameters. The traffic control unit includes a bandwidth management processor, a congestion prediction processor, and an allocation calculation processor. The bandwidth management processor is used to manage the communication bandwidth between nodes. The congestion prediction processor is used to predict virtual congestion. The allocation calculation processor is used to allocate bandwidth resources to existing communication tasks. The bandwidth management processor records the bandwidth resources occupied by all communication tasks between nodes and the idle bandwidth resources. When a communication task is completed, it releases this portion of bandwidth resources and adds it to the idle bandwidth resources, while sending a change signal to the congestion prediction processor. After receiving the change signal, the congestion prediction processor predicts the congestion situation according to the following formula: ； Where T represents the current time period. n represents the task variation coefficient within time period T. i Let m represent the number of tasks in the i-th window, m be the number of windows, and Y be the number of predicted tasks. The congestion prediction processor sets the necessary bandwidth resource Z0 based on the prediction task volume, and calculates the allocable resource volume Z according to the following formula. a : ； Among them, Z u The amount of idle bandwidth resources; the allocation calculation processor calculates the reallocation weight k for each communication task according to the following formula: ； Where t0 represents the completion time of the communication task, t represents the running time of the communication task, and d represents the amount of bandwidth resources allocated to the communication task. It is an offset constant; The allocation calculation processor calculates the new bandwidth allocation for the i-th communication task according to the following formula. : ； Where, k i This represents the redistribution weight of the i-th communication task, where n is the number of communication tasks.

Images