Smart city monitoring big data intelligent management system
By designing a big data intelligent management system in the smart city monitoring system and using sequence length calculation and DTW algorithm for data management, the problems of data redundancy and insufficient computing capabilities in traditional systems are solved, and efficient, secure and flexible data storage and processing are achieved.
Patent Information
- Application Number
- CN202510135164.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-07
- Publication Date
- 2025-05-13
AI Technical Summary
In the data management process, traditional smart city monitoring systems have problems such as high data redundancy, limited computing power and insufficient dynamics, resulting in waste of storage space and uneven computing pressure.
A smart city monitoring big data intelligent management system was designed to realize flexible data management and storage through data acquisition, edge device synthesis, sequence length calculation, data processing and data loop storage modules. The system uses a sequence length calculation formula to adjust the periodic storage length of each device, trims data based on the DTW algorithm, and dynamically allocates tasks using edge device collaboration mechanisms to ensure the efficiency and security of data storage.
By optimizing storage policies, we can achieve optimal allocation of storage resources, reduce data redundancy, improve storage efficiency, balance computing pressure, extend equipment service life, and ensure data security and real-time.
Smart Images
Figure CN119989000A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to an intelligent management system for smart city monitoring big data. Background Art
[0002] With the rapid development of smart cities, urban management has put forward higher requirements for the real-time and accuracy of data collection and processing. As an important part of the smart city monitoring system, edge devices are responsible for the collection and preliminary processing of sensor data. However, traditional monitoring systems face the following problems in the data management process: high data redundancy, high similarity of sensor data at different time points, resulting in wasted storage space; computing power limitation: edge devices have limited computing power and it is difficult to store and process large amounts of complete data for a long time; lack of dynamics: existing storage strategies lack a flexible adjustment mechanism; in response to the above problems, the present invention designs an intelligent management system for smart city monitoring big data. Summary of the invention
[0003] In order to overcome the shortcoming of the existing storage strategy that lacks a flexible adjustment mechanism, the present invention provides a smart city monitoring big data intelligent management system.
[0004] The technical solution is as follows: A smart city monitoring big data intelligent management system, including: A data acquisition module is used to acquire relevant data corresponding to each edge device in the city, as well as regional relevant data corresponding to each edge device; The edge device integration module is used to establish a computing collaboration mechanism based on multiple surrounding edge devices to obtain the edge device capabilities of the target edge device; A sequence length calculation module, used to calculate the periodic storage length in each edge device using a sequence length calculation formula according to the relevant data corresponding to each edge device and the area-related data corresponding to each edge device; A data processing module, used to clip the part of the similarity data that is greater than or equal to a preset threshold to obtain data to be saved, and adjust the data to be saved and then put it into the edge device for storage; A data cycle storage module, used for storing the adjusted data to be stored according to the periodic storage length; The data extraction module is used to complete and display the adjusted data to be saved stored in the edge device.
[0005] Preferably, the data acquisition module is used to obtain relevant data corresponding to each edge device in the city, and area-related data corresponding to each edge device, including: obtaining relevant data corresponding to each edge device in the city, and area-related data corresponding to each edge device, and performing data cleaning, standardization and data preprocessing in a unified format.
[0006] Preferably, the edge device integration module is used to establish a computing collaboration mechanism based on multiple surrounding edge devices to obtain the edge device capabilities of the target edge device, including: monitoring the CPU load, memory usage and power data of the target edge device and multiple surrounding edge devices, and calculating the remaining capacity of each edge device, and performing weighted averaging based on the remaining capacity of the surrounding edge devices to obtain the edge device capabilities of the target edge device.
[0007] Preferably, the sequence length calculation module is used to calculate using a sequence length calculation formula according to the relevant data corresponding to each edge device and the area-related data corresponding to each edge device to obtain the periodic storage length in each edge device, including: obtaining the safety data of the area where the edge device is located, the real-time requirements of the monitoring data, and the frequency of regional accidents, and inputting the data into the sequence length calculation formula to obtain the periodic storage length in each edge device.
[0008] Preferably, the data is input into a sequence length calculation formula to obtain the periodic storage length in each edge device, including: wherein the sequence length calculation formula is: ; In the formula, The periodicity preservation length in each edge device; is the adjustment factor; is the normalized regional security; is the normalized edge device capability; To meet the real-time requirements of normalized monitoring data; is the normalized regional accident frequency.
[0009] Preferably, the data processing module is used to compare, crop and save the relevant data collected by the edge device to obtain the data to be saved, and put the data to be saved into the edge device for storage, including: using the DTW algorithm to perform similarity analysis on the continuous sensor data collected in the edge device to obtain similarity data, and comparing the similarity data with a preset threshold, cropping the part of the similarity data greater than or equal to the preset threshold to obtain the data to be saved, and adjusting the data to be saved and putting it into the edge device for storage.
[0010] Preferably, the step of clipping the portion of the similarity data that is greater than or equal to a preset threshold to obtain the data to be saved, and adjusting the data to be saved and then storing it in the edge device includes: adjusting the size of the data to be saved by using a clipping restriction formula to obtain a first adjusted size, and comparing the size of the data to be saved with the first adjusted size; If the size of the data to be saved is greater than or equal to the first adjusted size, the data to be saved is placed in the edge device for saving; If the size of the data to be saved is smaller than the first adjustment size, the data to be saved is adjusted to the first adjustment size and then put into the edge device for saving.
[0011] Preferably, the step of adjusting the size of the data to be stored by using a cropping restriction formula to obtain a first adjusted size includes: the cropping restriction formula is: ; In the formula, Resize for the first; is the absolute minimum length limit; is the initial minimum length limit; is the final minimum length limit; is the position of the current data in the sequence; The periodicity of the sequence is preserved; A parameter for adjusting the decay rate.
[0012] Preferably, the data circulation storage module is used to store the adjusted data to be saved according to the periodic storage length, including: saving the adjusted data to be saved in corresponding quantities according to the periodic storage length to obtain a corresponding periodic sequence; when the adjusted quantity of data to be saved reaches the periodic protection length, reusing the sequence length calculation formula to obtain the periodic storage length in each edge device, and reusing the clipping limit formula to adjust subsequent data to be saved and then store them.
[0013] Preferably, the data extraction module is used to complete and display the adjusted data to be saved stored in the edge device, including: obtaining the periodic sequence in which the adjusted data to be saved is located, and when extracting data from the periodic sequence, copying and completing the missing data of other adjusted data to be saved in the periodic sequence according to the adjusted data to be saved in the first position in the periodic sequence, and displaying the completed periodic sequence.
[0014] Beneficial effects of the present invention: 1. The present invention uses a sequence length calculation formula, combined with multi-dimensional data such as edge device capabilities and regional security, to flexibly adjust the periodic storage length of each device to achieve optimal allocation of storage resources; 2. Perform similarity analysis on sensor data based on the DTW algorithm, cut and save the parts with high similarity, reduce data redundancy and improve storage efficiency; 3. Utilize the collaborative mechanism of edge devices to dynamically allocate tasks when the device load is high, balance computing pressure, and extend the service life of the device; 4. Set the minimum storage size limit through the trimming limit formula to ensure that the data can still meet key requirements after trimming; 5. Automatically update the storage length and re-crop new data to make the storage strategy adapt to regional monitoring needs and equipment performance changes in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a flow chart of a smart city monitoring big data intelligent management system of the present invention. DETAILED DESCRIPTION
[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0017] Example 1: A smart city monitoring big data intelligent management system, such as Figure 1 As shown, including: A data acquisition module is used to acquire relevant data corresponding to each edge device in the city, as well as regional relevant data corresponding to each edge device; Obtain relevant data corresponding to each edge device in the city, as well as regional relevant data corresponding to each edge device, and pre-process the data to clean, standardize and unify the format.
[0018] It should be noted that the data of temperature and humidity sensors, air quality sensors, noise sensors and vibration sensors for security monitoring of edge devices in smart cities are obtained, and data cleaning is performed on the data, that is, missing data and abnormal data are processed; standardization is performed to convert different types of data into a unified scale for comparison. For example, the temperature and humidity values, PM2.5 concentrations and noise decibels are normalized to the [0,1] interval to facilitate subsequent algorithm analysis; the format is unified, and there are inconsistencies in the data formats uploaded by different devices. All data are converted into a unified JSON format structure through preprocessing.
[0019] The edge device integration module is used to establish a computing collaboration mechanism based on multiple surrounding edge devices to obtain the edge device capabilities of the target edge device; Monitor the CPU load, memory usage, and power data of the target edge device and multiple surrounding edge devices, calculate the remaining capacity of each edge device, and perform weighted average based on the remaining capacity of the surrounding edge devices to obtain the edge device capacity of the target edge device.
[0020] It should be noted that the calculation cooperation mechanism is that when a device is running under too high a load, it is necessary to transfer some tasks to nearby devices through the cooperation mechanism. Therefore, when calculating the edge device capacity, the comprehensive capacity of the target edge device should be calculated. At the same time, the ability of the surrounding edge devices to help the target edge device should be considered, and the remaining capacity of the target device should be calculated. According to the performance indicators of each device, the remaining capacity of a single device is calculated using the following formula: ;in, is the weight; the edge device capability of the target device is calculated using weighted average, for example: ; In the formula, Indicates the distance weight between the surrounding edge devices and the target mutation device. The closer the device is, the higher the weight. is the number of surrounding edge devices.
[0021] A sequence length calculation module, used to calculate the periodic storage length in each edge device using a sequence length calculation formula according to the relevant data corresponding to each edge device and the area-related data corresponding to each edge device; Obtain the safety data of the area where the edge device is located, the real-time requirements of the monitoring data, and the frequency of regional accidents, and input the data into the sequence length calculation formula to obtain the periodic storage length in each edge device.
[0022] The formula for calculating the sequence length is: ; In the formula, The periodicity preservation length in each edge device; is the adjustment factor; is the normalized regional security; is the normalized edge device capability; To meet the real-time requirements of normalized monitoring data; is the normalized regional accident frequency.
[0023] It should be noted that in a smart city, edge devices in different areas need to adjust the periodic storage length of saved data according to their regional characteristics and monitoring needs; taking an edge device in a commercial area as an example, the regional security data is calculated through historical crime rate statistics; the edge device capability is the edge device capability of the target device obtained as above; the real-time nature of the monitoring data requires monitoring of air quality, which requires high real-time performance; the frequency of regional accidents, that is, the number of accidents that occurred in the area within a preset time period, and normalized them. After the periodic storage length of each edge device is calculated through the above, the longer the periodic storage length, the longer the data length saved in a periodic sequence of the edge device. By adjusting the periodic sequence length of different edge devices at different times, the probability that the monitoring personnel cannot obtain accurate monitoring data when the edge device is monitored can be reduced due to the lack of the first data to complete it, thereby protecting data security to a certain extent.
[0024] A data processing module, used to clip the part of the similarity data that is greater than or equal to a preset threshold to obtain data to be saved, and adjust the data to be saved and then put it into the edge device for storage; The DTW algorithm is used to perform similarity analysis on the continuous sensor data collected in the edge device to obtain similarity data, and the similarity data is compared with a preset threshold. The part of the similarity data greater than or equal to the preset threshold is cropped to obtain the data to be saved, and the data to be saved is adjusted and put into the edge device for storage.
[0025] The size of the data to be saved is adjusted by using a cropping restriction formula to obtain a first adjusted size, and the size of the data to be saved is compared with the first adjusted size; If the size of the data to be saved is greater than or equal to the first adjusted size, the data to be saved is placed in the edge device for saving; If the size of the data to be saved is smaller than the first adjustment size, the data to be saved is adjusted to the first adjustment size and then put into the edge device for saving.
[0026] The clipping limit formula is: ; In the formula, Resize for the first; is the absolute minimum length limit; is the initial minimum length limit; is the final minimum length limit; is the position of the current data in the sequence; The periodicity of the sequence is preserved; A parameter for adjusting the decay rate.
[0027] It should be noted that the data processing module performs similarity analysis on the continuous sensor data collected by the edge device through the dynamic time warping (DTW) algorithm to obtain similarity data; then, the similarity data is compared with the preset threshold, and the part above or equal to the threshold is cropped and removed. Specifically, for the part above or equal to the threshold, it can be restored through the completion mechanism and has stability, so it is directly cropped and removed. For the part below the threshold, due to its significant dynamic characteristics, it needs to be retained, and the data to be saved is generated and put into the edge device for storage; in smart cities, edge devices often collect a large amount of sensor data, such as temperature, humidity, and air quality, which are continuously changing data. Through analysis and cropping, storage space can be effectively saved.
[0028] It should be noted that through the dynamic adjustment and storage mechanism of periodic sequences, the interference of edge devices to monitoring is improved to ensure data security, and the dynamic adjustment range is: the storage length of the data after trimming is flexibly adjusted according to the position of the periodic storage length sequence; the trimming strategy is optimized: more content tends to be retained at the initial position of the data sequence, and the closer to the end, the more attention is paid to data variability, protecting some key data from being easily captured and monitored.
[0029] A data cycle storage module, used for storing the adjusted data to be stored according to the periodic storage length; The adjusted data to be saved is saved in corresponding quantities according to the periodic storage length to obtain the corresponding periodic sequence. When the adjusted number of data to be saved reaches the periodic protection length, the sequence length calculation formula is reused to obtain the periodic storage length in each edge device, and the clipping limit formula is reused to adjust the subsequent data to be saved and then store them.
[0030] It should be noted that the system stores the adjusted data to be saved according to the periodic storage length. When the amount of data reaches the periodic storage length, the recalculation mechanism is triggered to generate a new periodic storage length. The new storage length will be adjusted according to the changes in regional accident frequency or edge device capabilities. The system reapplies the trimming limit formula under the new storage length to adjust subsequent data and store it.
[0031] The data extraction module is used to complete and display the adjusted data to be saved stored in the edge device.
[0032] The periodic sequence in which the adjusted data to be saved is located is obtained. When data is extracted from the periodic sequence, the missing data of other adjusted data to be saved in the periodic sequence are copied and completed according to the adjusted data to be saved at the first position in the periodic sequence, and the completed periodic sequence is displayed.
[0033] It should be noted that when it is necessary to display periodically saved data, since some data have been cropped in the early stage, in order to reduce data missing or errors during the display process, it is necessary to restore the cropped data through the completion mechanism to form a complete periodic sequence for display.
[0034] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A smart city monitoring big data intelligent management system, characterized in that: The following steps are involved: A data acquisition module is used to acquire relevant data corresponding to each edge device in the city, as well as regional relevant data corresponding to each edge device; The edge device integration module is used to establish a computing collaboration mechanism based on multiple surrounding edge devices to obtain the edge device capabilities of the target edge device; A sequence length calculation module, used to calculate the periodic storage length in each edge device using a sequence length calculation formula according to the relevant data corresponding to each edge device and the area-related data corresponding to each edge device; A data processing module, used to clip the part of the similarity data that is greater than or equal to a preset threshold to obtain data to be saved, and adjust the data to be saved and then put it into the edge device for storage; A data cycle storage module, used for storing the adjusted data to be stored according to the periodic storage length; The data extraction module is used to complete and display the adjusted data to be saved stored in the edge device.
2. According to claim 1, the smart city monitoring big data intelligent management system is characterized in that: The data acquisition module is used to obtain relevant data corresponding to each edge device in the city, as well as regional relevant data corresponding to each edge device, including: obtaining relevant data corresponding to each edge device in the city, as well as regional relevant data corresponding to each edge device, and performing data cleaning, standardization and data preprocessing in a unified format.
3. The smart city monitoring big data intelligent management system according to claim 1 is characterized in that: The edge device integration module is used to establish a computing collaboration mechanism based on multiple surrounding edge devices to obtain the edge device capabilities of the target edge device, including: monitoring the CPU load, memory occupancy and power data of the target edge device and multiple surrounding edge devices, and calculating the remaining capacity of each edge device, and performing weighted averaging based on the remaining capacity of the surrounding edge devices to obtain the edge device capabilities of the target edge device.
4. The smart city monitoring big data intelligent management system according to claim 1 is characterized in that: The sequence length calculation module is used to calculate the periodic storage length in each edge device using a sequence length calculation formula according to the relevant data corresponding to each edge device and the area related data corresponding to each edge device, including: obtaining the safety data of the area where the edge device is located, the real-time requirements of the monitoring data, and the frequency of regional accidents, and inputting the data into the sequence length calculation formula to obtain the periodic storage length in each edge device.
5. The smart city monitoring big data intelligent management system according to claim 4 is characterized in that: The data is input into a sequence length calculation formula to obtain the periodic storage length in each edge device, including: wherein the sequence length calculation formula is: ， In the formula, The periodicity preservation length in each edge device; is the adjustment factor; is the normalized regional security; is the normalized edge device capability; To meet the real-time requirements of normalized monitoring data; is the normalized regional accident frequency.
6. The smart city monitoring big data intelligent management system according to claim 1 is characterized in that: The data processing module is used to compare, crop and save the relevant data collected by the edge device, obtain the data to be saved, and put the data to be saved into the edge device for saving, including: using the DTW algorithm to perform similarity analysis on the continuous sensor data collected in the edge device to obtain similarity data, and compare the similarity data with a preset threshold, crop the part of the similarity data greater than or equal to the preset threshold to obtain the data to be saved, and adjust the data to be saved and put it into the edge device for saving.
7. The smart city monitoring big data intelligent management system according to claim 6 is characterized in that: The method of clipping the portion of the similarity data that is greater than or equal to a preset threshold to obtain the data to be saved, and adjusting the data to be saved and storing it in the edge device includes: adjusting the size of the data to be saved by using a clipping restriction formula to obtain a first adjusted size, and comparing the size of the data to be saved with the first adjusted size; If the size of the data to be saved is greater than or equal to the first adjusted size, the data to be saved is placed in the edge device for saving; If the size of the data to be saved is smaller than the first adjustment size, the data to be saved is adjusted to the first adjustment size and then put into the edge device for saving.
8. The smart city monitoring big data intelligent management system according to claim 7 is characterized in that: The step of adjusting the size of the data to be saved by using a clipping restriction formula to obtain a first adjusted size includes: the clipping restriction formula is: ； In the formula, Resize for the first; is the absolute minimum length limit; is the initial minimum length limit; is the final minimum length limit; is the position of the current data in the sequence; The periodicity of the sequence is preserved; A parameter for adjusting the decay rate.
9. The smart city monitoring big data intelligent management system according to claim 1 is characterized in that: The data circulation storage module is used to store the adjusted data to be saved according to the periodic storage length, including: saving the adjusted data to be saved in corresponding quantities according to the periodic storage length to obtain a corresponding periodic sequence; when the adjusted quantity of data to be saved reaches the periodic protection length, reusing the sequence length calculation formula to obtain the periodic storage length in each edge device, and reusing the clipping limit formula to adjust the subsequent data to be saved and then store them.
10. The smart city monitoring big data intelligent management system according to claim 1 is characterized in that: The data extraction module is used to complete and display the adjusted data to be saved stored in the edge device, including: obtaining the periodic sequence in which the adjusted data to be saved is located, and when extracting data from the periodic sequence, copying and completing the missing data of other adjusted data to be saved in the periodic sequence according to the adjusted data to be saved at the first position in the periodic sequence, and displaying the completed periodic sequence.