Smart city multi-source data multi-dimensional detection processing method and system

By acquiring, cleaning and classifying multi-source data, using ElasticSearch and exception detection technology, a multi-source database is built, which solves the difficulties in multi-source data processing and analysis in smart cities, and realizes efficient, accurate and secure storage and analysis of data. Provides comprehensive and accurate data support.

CN118939737APending Publication Date: 2024-11-12HENAN URBAN & RURAL PLANNING & DESIGN RESEARCH INSTITUTE CO LTD SHANGHAI BRANCH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410964605.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

There are difficulties in processing and analyzing multi-source data in smart cities. Traditional methods cannot fully utilize the richness and diversity of multi-source data, resulting in difficulty in reading, writing, storing and information optimization of data, and it is difficult to detect and process abnormal data efficiently and accurately. , ensure the reliability and security of data.

Method used

A smart city multi-source data detection and processing method is proposed, including obtaining multi-source raw data, cleaning and classification, storing and indexing optimization using ElasticSearch, which is divided into cold data and hot data for storage, and constructing through abnormal detection and data analysis. Multi-source database to achieve effective integration and optimization of data.

Benefits of technology

Improve data optimization efficiency and query performance, ensure data accuracy and reliability, promptly detect and repair errors or abnormal behaviors in the data, ensure data reliability and security, and provide comprehensive data optimization for smart cities. , accurate and timely support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939737A_ABST
    Figure CN118939737A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of smart cities, and particularly provides a smart city multi-source data multi-dimensional detection processing method and system, and the method comprises the steps: obtaining multi-source original data, cleaning the original data, classifying the cleaned original data into available data and unavailable data, screening out the unavailable data, and carrying out the multi-dimensional detection processing of the multi-source data. Dividing available data into conventional data and redundant data, and performing data compression operation on the redundant data; the conventional data is divided into cold data and hot data, and the cold data and the hot data are stored based on ElasticSearch; performing index optimization on the ElasticSearch, performing abnormal storage detection after data storage, and completing construction of a multi-source database; data analysis is carried out based on a multi-source database and is used for data information optimization of a smart city. Through the technical scheme, effective integration, cleaning, classification, storage and analysis of multi-source data are realized, and comprehensive, accurate and timely data support is provided for data information optimization of a smart city.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of smart cities, and specifically provides a smart city multi-source data multi-dimensional detection and processing method and system. Background Art

[0002] With the acceleration of urbanization and the rapid development of information technology, smart cities have become an important direction for future urban development. Smart cities integrate advanced information and communication technologies to achieve efficient, intelligent and green urban operations. However, in the construction and operation of smart cities, the processing and analysis of multi-source data has become an important technical challenge.

[0003] Traditional data processing methods are often limited to a single data source or simple data fusion, and cannot fully utilize the richness and diversity of multi-source data. In addition, due to the wide range of data sources, diverse formats, uneven quality, different structures, high redundancy and other issues, data reading, writing, storage and information optimization are difficult, and integration and analysis are more difficult. At the same time, with the continuous growth and complexity of data, how to efficiently and accurately detect and process abnormal data and ensure data reliability and security is also a technical problem that needs to be solved in the construction of smart cities. Summary of the invention

[0004] In order to overcome the above-mentioned defects, the present invention is proposed to solve the technical problem of the difficulty in reading, writing and storing multi-source data in existing smart cities.

[0005] A smart city multi-source data detection and processing method comprises the steps of:

[0006] Acquire raw data from multiple sources and clean the raw data, wherein the raw data from multiple sources includes environmental data, user behavior data, social public data, and IoT device data;

[0007] Classify the cleaned raw data into usable data and unusable data, filter out unusable data, divide the usable data into regular data and redundant data, and perform data compression on the redundant data;

[0008] Divide regular data into cold data and hot data, and store them based on ElasticSearch;

[0009] Optimize the ElasticSearch index, extract the fields to be searched to form a separate index data, write the regular data to HBase according to the Rowkey, and write the search data to ElasticSearch according to the Rowkey;

[0010] After data is stored, abnormal storage detection is performed to complete the construction of a multi-source database;

[0011] Data analysis based on multi-source databases is used for data information optimization in smart cities.

[0012] As a further optimization of this solution, the process of acquiring the original data includes acquiring environmental data through sensor detection, or retrieving the environmental data through a public database / commercial database, and the environmental data includes environmental temperature data, humidity data and light data of multiple locations;

[0013] Acquire user behavior data through sensors and applications built into mobile devices, or retrieve the user behavior data through public databases / commercial databases;

[0014] Obtaining social public data from governments, enterprises, and organizations through pre-built dedicated data communication paths or web crawlers, including but not limited to statistics bureau data, industry reports, and corporate annual reports;

[0015] Get real-time IoT device data from smart home devices and industrial automation equipment.

[0016] As a further optimization of this solution, the conventional data is divided into cold data and hot data, including the steps of:

[0017] All regular data are stored in the cold data area, and an identification ID is set for each piece of data, and the identification ID is initially 0;

[0018] Writing the time stamp and a specific string into the identification ID, wherein the specific string is a string of 32 characters generated by MD5 and Salt hashing;

[0019] Determine the data type of current regular data, and determine whether the regular data is cold data or hot data according to the data type;

[0020] Modify the ID of the hot data to 1;

[0021] The conventional data with the identification ID of 1 is stored in the hot data area, and then the data in the hot data area and the cold data area are respectively written into the storage areas of different nodes.

[0022] As a further optimization of this solution, the data compression operation on the redundant data includes the steps of:

[0023] Determine the compression algorithm and compression strategy based on the redundant data type and storage requirements;

[0024] Optimize compression algorithms and compression strategies;

[0025] Use optimized compression algorithms and compression strategies to compress redundant data;

[0026] The CRC is used to check the integrity of the compressed redundant data.

[0027] As a further optimization of this solution, the abnormal storage detection after data storage includes the steps of:

[0028] Using a monitoring tool to collect historical data of key performance indicators, the key performance indicators including at least one of average response time, throughput, I / O latency, CPU utilization, memory utilization, and disk space utilization;

[0029] Analyze historical data and calculate moving averages for each key performance indicator;

[0030] Obtain the currently stored key performance indicator data and compare whether the difference between the current data and the moving average exceeds the preset threshold;

[0031] If so, it is considered that there is abnormal storage and an abnormal report is generated.

[0032] As a further optimization of this solution, in the analysis of historical data, a function calculate_moving_average is defined to calculate the moving average. The parameters accepted by the function calculate_moving_average include data and window size. When the CPU determines that the function calculate_moving_average is in the execution state, a list moving_averages is created to store the average value of each data point, and the moving average values ​​are connected to obtain the moving average line.

[0033] Traverse the original data, if the index of the data point is less than the window size, subtract one, and add "None" to the list moving_averages, indicating that the current data point is not enough to fill the entire window. Otherwise, take the data within the window range, calculate the average value as the valid data, and add the result to the list moving_averages.

[0034] A function detect_anomalies is defined to detect anomalies between current data and a moving average. The parameters accepted by the function detect_anomalies include current data, a moving average, and a threshold. When the CPU determines that the function detect_anomalies is in an execution state, a list moving_averages is created to store the detected anomalies, each indicator of the current data is traversed, the difference between the current data and the corresponding moving average is calculated, and compared with a preset threshold. If the difference exceeds the threshold, the anomaly information is added to the list moving_averages.

[0035] As a further optimization of this solution, after the anomaly is identified and stored, the anomaly data is double-checked, and the initial anomaly data set is set to A{A1,A2,…,A k}, let the associated data array YCX, YCX ij Indicates the association between the i-th data entity and the j-th data entity,

[0036] Define a function check_association for checking the association between anomalies. The parameters received by the function check_association include the anomaly data set A, the associated data array YCX and the associated threshold. Traverse the data in the list moving_averages to determine the relationship between the set A processed by the refined association matrix and the associated threshold. If any pair of anomalies does not meet the association condition, the anomaly is determined to be a true anomaly; otherwise, it is a false anomaly.

[0037] A smart city multi-source data detection and processing system, comprising:

[0038] An acquisition unit, which acquires raw data from multiple sources and cleans the raw data;

[0039] The classification processing unit classifies the cleaned raw data and stores it based on ElasticSearch;

[0040] Index processing unit, which optimizes the index of ElasticSearch;

[0041] The storage detection unit performs abnormal storage detection after data is stored and completes the construction of a multi-source database;

[0042] The data analysis and optimization unit performs data analysis based on multi-source databases for data information optimization in smart cities.

[0043] Working principle and beneficial effects of the present invention:

[0044] In the technical solution of the present invention, data is collected through diversified data sources, and data cleaning is used to ensure the accuracy and reliability of the data. According to the type, source, purpose and other characteristics of the data, the data is classified and the appropriate storage technology is selected for storage, which helps to improve the optimization efficiency and query performance of the data and provide strong data support for the data information optimization of smart cities. ElasticSearch is used as a solution for data storage and query, and the retrieval efficiency and performance of data are improved through index optimization technology. After the data is stored, statistical analysis, machine learning algorithms or specific business rules are used to detect anomalies in the data. It helps to timely discover and repair errors, damage or abnormal behaviors in the data, and ensure the reliability and security of the data. The cleaned, classified and stored data are integrated into a multi-source database, and the data in the database is deeply analyzed using data mining, machine learning and other technologies to reveal the associations and laws between the data, and provide decision support and optimization suggestions for the data information optimization of smart cities.

[0045] This solution can effectively integrate, clean, classify, store and analyze multi-source data, and provide comprehensive, accurate and timely data support for the optimization of smart city data information. At the same time, this method is also efficient, flexible and scalable, and can adapt to the needs of the continuous development and changes of smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0047] Figure 1 It is a schematic flow chart of the main steps of a smart city multi-source data detection and processing method of the present invention;

[0048] Figure 2 It is a main structural block diagram of a smart city multi-source data detection and processing system of the present invention;

[0049] List of reference numerals:

[0050] 11: Acquisition unit, 12: Classification processing unit, 13: Index processing unit, 14: Storage detection unit, 15: Data analysis and optimization unit. DETAILED DESCRIPTION

[0051] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0052] Embodiment 1,

[0053] Figure 1 This is a schematic diagram of the main steps of a smart city multi-source data detection and processing method of the present invention. Figure 1 As shown, a smart city multi-source data detection and processing method in an embodiment of the present invention mainly includes the following steps S10 to S50.

[0054] Step S10: acquiring raw data from multiple sources and cleaning the raw data;

[0055] In this embodiment, data is collected from various sources, including environmental data, user behavior data, social public data, and IoT device data. These data cover various aspects of the city, such as air quality, crowd flow, energy consumption, public safety, etc. At the same time, the original data is cleaned to eliminate noise and outliers in the data.

[0056] Specifically, the steps of obtaining and cleaning are as follows:

[0057] 1. Environmental data acquisition and cleaning

[0058] Through sensors deployed in various corners of the city, such as temperature sensors, humidity sensors, light sensors, etc., environmental data such as ambient temperature, humidity, light, etc. at multiple locations can be detected and obtained in real time. The above data can usually also be obtained through public open databases, and are basically free.

[0059] Clean the acquired environmental data to remove outliers or invalid data caused by sensor failure, network delay, etc., to ensure the accuracy and reliability of the data.

[0060] 2. User behavior data acquisition and cleaning

[0061] Utilize the built-in sensors and applications of mobile devices, such as GPS positioning, acceleration sensors, and application usage records, to obtain user behavior data, such as user movement trajectories, travel habits, and consumption preferences. It is also possible to purchase and retrieve user behavior data (including user portraits and user tag data, etc.) through commercial databases that meet security and privacy standards.

[0062] User behavior data is anonymized to protect user privacy; at the same time, the data is cleaned to remove noise data, duplicate data, etc. to improve data quality.

[0063] 3. Acquisition and cleaning of social public data

[0064] Use web crawler technology to obtain social public data from official websites or data platforms of governments, enterprises, and organizations, such as statistics bureau data, industry reports, corporate annual reports, etc. Or, based on prior public communication, build dedicated data channels and ports for communication with relevant enterprises, institutions, or administrative units, and use dedicated channels for dedicated purposes to obtain social public data.

[0065] Clean the acquired social public data to remove irrelevant information, duplicate data, etc., while ensuring the accuracy and timeliness of the data.

[0066] 4. IoT device data acquisition and cleaning

[0067] Through IoT devices such as smart home devices and industrial automation equipment, real-time IoT device data such as equipment operating status, energy consumption data, production data, etc. can be obtained.

[0068] Clean IoT device data to remove outliers or invalid data caused by equipment failure, communication delays, etc., to ensure data accuracy and real-time performance.

[0069] Step S20: Classify and store the cleaned raw data based on ElasticSearch.

[0070] In this embodiment, refined data information optimization and optimized storage of multi-source data are achieved. First, by screening available data, unnecessary storage space is reduced; second, by data compression operations, the use of storage space is further reduced; finally, by partitioning and storing hot and cold data, the data access speed and storage efficiency are improved. These measures work together to make the data information optimization of smart cities more efficient, reliable and economical.

[0071] Specifically, step S20 further includes the following steps:

[0072] Step S201: The cleaned raw data is divided into usable data and unusable data, and the unusable data is filtered out.

[0073] The cleaned raw data is divided into usable data and unusable data. Unusable data usually includes invalid data, duplicate data, and data that does not meet the analysis requirements. By filtering out these unusable data, unnecessary storage space can be reduced and the efficiency of data storage can be improved.

[0074] Step S202: Divide the available data into regular data and redundant data, and perform data compression operation on the redundant data.

[0075] Further classify the available data that has been filtered out. The available data is divided into regular data and redundant data. Redundant data refers to repeated or highly similar parts in the data set, which will take up a lot of space when stored. Therefore, compressing redundant data can effectively reduce the use of storage space while maintaining the integrity and availability of the data. Data compression can use a variety of algorithms, such as lossless compression and lossy compression. The specific choice depends on the characteristics of the data and the analysis requirements.

[0076] In this embodiment, by compressing redundant data, the use of storage space can be greatly reduced, and storage costs can be reduced. The compressed data occupies less space, which can reduce I / O operations and memory usage, thereby improving data processing performance. Through integrity verification, it can be ensured that there are no errors or damage to the compressed data during transmission or storage, ensuring the integrity and recoverability of the data. These effects work together to make the data information optimization of smart city data more efficient, reliable and economical.

[0077] Specifically, step S202 further includes the following steps:

[0078] Step S2021: Determine the compression algorithm and compression strategy based on the redundant data type and storage requirements.

[0079] Different data types have different characteristics and require different compression algorithms to achieve the best results. At the same time, storage requirements and system performance are also factors that need to be considered when choosing a compression strategy.

[0080] Step S2022: Optimize the compression algorithm and compression strategy.

[0081] After determining the compression algorithm and compression strategy, you need to further optimize the compression effect by adjusting the algorithm parameters, including selecting the appropriate compression level, adjusting the buffer size, setting parallel processing parameters, etc.

[0082] Step S2023: compressing redundant data using the optimized compression algorithm and compression strategy;

[0083] This process is usually implemented programmatically and can be done using a variety of programming languages ​​and tools. The compression process should be as efficient and accurate as possible to ensure data integrity and recoverability.

[0084] Step S2024: Perform integrity check on the compressed redundant data through CRC to ensure data integrity and recoverability.

[0085] This is achieved by using technologies such as cyclic redundancy check (CRC). CRC is a commonly used data verification method that generates a check code by performing specific calculations on the data, and then stores or transmits this check code together with the original data. At the receiving end, the same calculation can be performed on the data again, and the generated check code can be compared with the received check code to see if they are consistent. If they are consistent, it means that there is no error in the data during transmission or storage; if they are inconsistent, it means that the data has been damaged or lost.

[0086] Step S203: divide the regular data into cold data and hot data for partitioned storage.

[0087] Regular data is divided into cold data and hot data, and stored in partitions. Cold data refers to data that is not frequently accessed or queried, such as historical data and backup data. Hot data refers to data that needs to be accessed or queried frequently, such as real-time data and monitoring data. By partitioning and storing cold data and hot data, the performance of the storage system can be optimized. Specifically, hot data can be stored on high-performance storage devices for fast access and query, and cold data can be stored on low-cost storage devices to save storage costs. This partitioned storage method not only improves the access speed of data, but also reduces the overall storage cost.

[0088] In this embodiment, storing hot data in a hot data area with higher performance can speed up the reading of data and improve the responsiveness of the system. Storing cold data in a cold data area with lower cost can save storage resources and reduce storage costs. By setting a unique identification ID and hash string for the data, the uniqueness and security of the data can be increased. At the same time, writing the data into the hot and cold data areas of different nodes can achieve redundant backup and load balancing of the data, further improving the reliability and stability of the system.

[0089] Specifically, step S203 further includes the following steps:

[0090] Step S203: All regular data are stored in the cold data area, and an identification ID is set for each piece of data, where the identification ID is initially 0.

[0091] Step S2031: Write the time TimeStamp and a specific character string into the identification ID; the specific character string is a character string consisting of 32 characters generated after MD5 and Salt hashing.

[0092] Step S2032: Determine the data type of the current regular data, and determine whether the regular data is cold data or hot data according to the data type.

[0093] Based on the data type of regular data (such as real-time monitoring data, historical record data, etc.) and business logic, determine whether the data is cold data or hot data. Hot data usually refers to data with high access frequency and high real-time requirements, while cold data refers to data with low access frequency and low real-time requirements.

[0094] Step S2033: modify the identification ID of the hot data to 1.

[0095] Step S2034: storing the conventional data with the identification ID of 1 in the hot data area, and then writing the data in the hot data area and the cold data area into storage areas of different nodes respectively.

[0096] Move the selected hot data from the cold data area to the hot data area for storage. At the same time, in order to ensure the security and availability of the data, these data can be written to the hot data areas of different nodes to achieve redundant backup and load balancing of the data. For regular data identified as cold data, it can continue to be stored in the cold data area. Similarly, this data can also be backed up to different nodes as needed.

[0097] Furthermore, after the conventional data is divided into cold data and hot data, a key pair is generated, the key pair includes an interactive public key and an interactive private key, the cold data and hot data are encrypted using the interactive public key, and the encrypted data is stored. At the same time, the index data corresponding to each cold data and hot data is encrypted using the interactive public key and stored together. When indexing, the interactive public key is obtained and the provided index data is encrypted, and then the search is performed to obtain the interactive private key and the corresponding cold data or hot data, and the cold data or hot data is decrypted using the interactive private key.

[0098] Step S30: Optimize the index of ElasticSearch;

[0099] Specifically, step S30 further includes the following steps.

[0100] Step S301: extract the fields to be searched to form a single index data;

[0101] Field analysis: First, you need to analyze business requirements and data characteristics to determine which fields need to be frequently retrieved. These fields may include user ID, location information, timestamp, keywords, etc.

[0102] Index design: Design separate indexes for the fields that need to be retrieved. In ElasticSearch, an index is a collection of one or more documents, each of which is a JSON object. Therefore, you can organize the fields that need to be retrieved into one or more JSON objects and create corresponding indexes.

[0103] Index mapping: Define the mapping of the index, and clarify the type of each field, whether it is searchable, whether it is indexable, etc. Reasonable mapping settings can improve the search efficiency and accuracy of the index.

[0104] Step S302: write regular data to HBase according to Rowkey, and write search data to ElasticSearch according to Rowkey.

[0105] HBase data writing: HBase is a highly reliable, high-performance, column-oriented, scalable distributed storage system that is particularly suitable for storing unstructured and semi-structured sparse data. By writing regular data to HBase based on Rowkey, you can use HBase's columnar storage characteristics and scalability to efficiently store and retrieve data.

[0106] ElasticSearch data writing: Write the data to be retrieved (usually in JSON format) to ElasticSearch based on Rowkey or other unique identifiers. Extract the retrieved data from the original data and convert it into a format suitable for ElasticSearch retrieval.

[0107] Data synchronization: Ensure that the data in HBase and ElasticSearch are synchronized. When the data in HBase changes, the index data in ElasticSearch needs to be updated accordingly. This can be achieved by listening to HBase change events or using data synchronization tools.

[0108] Step S40: After the data is stored, abnormal storage detection is performed to complete the construction of the multi-source database.

[0109] Specifically, step S40 further includes the following steps.

[0110] Step S401: using a monitoring tool to collect historical data of key performance indicators, where the key performance indicators include at least one of average response time, throughput, I / O latency, CPU utilization, memory utilization, and disk space utilization;

[0111] Step S402: Analyze historical data and calculate the moving average of each key performance indicator;

[0112] Step S403: Acquire the currently stored key performance indicator data, and compare whether the difference between the current data and the moving average exceeds a preset threshold;

[0113] Step S404: If yes, it is considered that there is abnormal storage and an abnormal report is generated.

[0114] Step S50: Perform data analysis based on multi-source databases for data information optimization of smart cities.

[0115] The analysis results will be applied to various aspects of urban data information optimization, such as intelligent transportation systems, public safety data information optimization, environmental protection, etc. Through data application, environmental protection measures can be formulated, the living environment of residents can be improved, and the level of high-quality urban development can be improved.

[0116] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0117] Embodiment 2,

[0118] After the data is stored, abnormal storage detection is performed, including the following steps:

[0119] Using a monitoring tool to collect historical data of key performance indicators, the key performance indicators including at least one of average response time, throughput, I / O latency, CPU utilization, memory utilization, and disk space utilization;

[0120] Analyze historical data and calculate the moving average of each key performance indicator. In the analysis of historical data, define a function calculate_moving_average to calculate the moving average. The parameters accepted by the function calculate_moving_average include data and window size. When the CPU determines that the function calculate_moving_average is in an execution state, create a list moving_averages to store the average value of each data point, and connect the moving average values ​​to obtain the moving average.

[0121] After the data is stored, abnormal storage detection is performed, including the steps of: using a monitoring tool to collect historical data of key performance indicators, the key performance indicators including at least one of average response time, throughput, I / O latency, CPU utilization, memory utilization, and disk space utilization; analyzing historical data and calculating the moving average of each key performance indicator; obtaining the currently stored key performance indicator data, and comparing whether the difference between the current data and the moving average exceeds a preset threshold; if so, it is considered that abnormal storage exists, and an abnormality report is generated.

[0122] The original data is traversed. If the index of the data point is less than the window size, it is reduced by one. "None" is added to the list moving_averages, indicating that the current data point is not enough to fill the entire window. Otherwise, the data within the window range is taken, the average value is calculated as the valid data, and the result is added to the list moving_averages. The function detect_anomal ies is defined to detect the anomaly between the current data and the moving average. The parameters accepted by the function detect_anomal ies include the current data, the moving average and the threshold. When the CPU determines that the function detect_anomal ies is in the execution state, a list moving_averages is created to store the detected anomaly, and each indicator of the current data is traversed. The difference between the current data and the corresponding moving average is calculated, and compared with the preset threshold. If the difference exceeds the threshold, the abnormal information is added to the list moving_averages.

[0123] Embodiment 3,

[0124] The current data is compared with the moving average of historical data, and it is considered abnormal when it exceeds the prediction; however, this algorithm is often prone to errors. The reason is that the initially constructed model is mainly based on a single comparison between the current value and the historical value, and the correlation between different data has not been considered. When constructing a smart city multi-source database model, when certain urban events occur (including conventional periodic events such as events that change with the seasons, and various random and accidental urban events, etc.), the data corresponding to these urban events in the smart city multi-source database model will change dramatically (for example, sudden changes in temperature and corresponding changes in electricity consumption caused by bad weather, as well as changes in traffic data, motor vehicle congestion, reduction in pedestrians and non-motor vehicles, and changes in various commercial and consumer behavior data), etc. This is not a data anomaly caused by a malfunction, but according to the algorithm before the improvement, data errors will still occur.

[0125] The improved data model is as follows: Step 1. First, we can construct a sub-data module to characterize the correlation between different data entities in the original smart city multi-source database model through data annotation, namely, the city event correlation sub-data module; Step 2. Then, use the technical content of item 9 in Part 2 before the improvement to perform data verification and output a data anomaly report; Step 3. Finally, perform a double test based on the sub-data module constructed in Step 1. In the double test, focus on whether the abnormal data output in Step 2 just meets the data correlation annotated by the city event correlation sub-data module in Step 1. Only data anomalies that do not meet this data correlation are considered to be real data anomalies.

[0126] After the anomaly is identified and stored, the abnormal data is double-checked. The initial abnormal data set is set to A{A1, A2, …, Ak}. The associated data array YCX, YCXij represents the association between the i-th data entity and the j-th data entity.

[0127] Define a function check_association for checking the association between anomalies. The parameters received by the function check_association include the anomaly data set A, the associated data array YCX and the associated threshold. Traverse the data in the list moving_averages to determine the relationship between the set A processed by the refined association matrix and the associated threshold. If any pair of anomalies does not meet the association condition, the anomaly is determined to be a true anomaly; otherwise, it is a false anomaly.

[0128] 1. Data collection and annotation

[0129] First, use monitoring tools to collect historical data of key performance indicators. Suppose these key performance indicators are: average response time Tavg; throughput Q; I / O latency LIO; CPU utilization UCPU; memory utilization Umem; disk space utilization Udisk; We represent the historical data of these indicators as time series: {Tavg(t i ), Q(t i ), LIO(t i ), UCPU(t i ), Umem(t i ), Udisk(t i )}i=1N. Where t i represents a time point, and N is the total number of historical data.

[0130] In order to construct the urban event correlation sub-data module, we need to annotate the correlation between data entities. Let the correlation data array be YCX, where YCX ij It represents the correlation between the i-th data entity and the j-th data entity. The correlation can be calculated by statistical methods such as correlation coefficient and mutual information.

[0131] 2. Moving average calculation

[0132] For each key performance indicator, calculate its moving average. Assume that the moving average window size is w, then the moving average It can be expressed as: Its mathematical expression is Where X represents any key performance indicator, such as Tavg, Q, etc.

[0133] 3. Current data acquisition and difference calculation

[0134] Get the key performance indicator data currently stored, recorded as: {Tavg(t current ),Q(t current ),LIO(t current ),UCPU(t current ),Umem(t current ),Udisk(t current )}; Calculate the difference between the current data and the moving average:

[0135] ΔX(t current )=X(t current )-X―(t current ).

[0136] 4. Preliminary anomaly detection

[0137] Set a preset threshold ∈X to perform preliminary anomaly detection on each key performance indicator. If the difference ΔX(t current ) exceeds the preset threshold ∈X, it is considered that there is preliminary abnormal storage. The mathematical expression is: Preliminary abnormal storage if|ΔX(t current )|>∈X, then the initial exception is stored.

[0138] 5. Double inspection

[0139] For the data stored in the preliminary abnormality, a double check is performed. According to the urban event correlation sub-data module, check whether the abnormal data meets the data correlation. Suppose the data stored in the preliminary abnormality is {X1, X2, …, X k}, the corresponding associated data array is YCX. If the correlation between these data meets certain conditions (such as the correlation coefficient is greater than a certain threshold), it is considered that the anomaly of these data is caused by urban events rather than failures. The mathematical expression is: Abnormality caused by urban events if YX ij >ρ, then the anomaly is caused by the city event. ρ is the threshold of correlation. If the above conditions are not met, it is considered to be a real data anomaly.

[0140] 6. Exception report generation

[0141] For the detected real abnormal storage, an abnormal report is generated. The report content includes: abnormal indicator name; current value X(t current ); moving average value X―(t current ); Difference ΔX(t current ); preset threshold ∈X; associated data array YCX. Through the above improved model and process, false positives can be effectively reduced to ensure the accuracy and reliability of data anomaly detection.

[0142] The data structure of this data algorithm is as follows:

[0143] A. First, import the NumPy library, and then define three functions: calculate_moving_average, detect_initial_anomalies, and check_association.

[0144] The `calculate_moving_average` function is used to calculate the moving average. It accepts two parameters, namely data (data) and window size (window_size). Then, it creates an empty list `moving_averages` to store the moving average of each data point. Next, it iterates over the data, and for each data point, if the index of the data point is less than the window size minus one, it adds `None` to the moving average list, indicating that the current data point is not enough to fill the entire window. Otherwise, it takes the data within the window range and calculates its average, and then adds the result to the moving average list. Finally, the moving average list is returned.

[0145] The `detect_initial_anomalies` function is used to detect initial anomalies between the current data and the moving average of historical data. It accepts three parameters, namely the current data (current_data), the moving average of historical data (moving_averages), and the threshold (thresholds). It creates an empty list `initial_anomalies` to store the detected initial anomalies. Then, it iterates over each indicator of the current data, calculates the difference between the current data and the corresponding moving average of historical data, and compares it with the pre-set threshold. If the difference exceeds the threshold, the anomaly information is added to the initial anomaly list. Finally, the initial anomaly list is returned

[0146] The `check_association` function is used to check the association between anomalies. It accepts three parameters, namely the anomaly list (anomalies), the anomaly association matrix (association_matrix), and the association threshold (threshold). It traverses each pair of anomalies in the anomaly list and determines whether there is an association between them by checking whether the corresponding values ​​in the association matrix are lower than the association threshold. If there is any pair of anomalies that does not meet the association condition, it returns False; otherwise it returns True.

[0147] B. In the application of algorithms, the following examples can be given:

[0148] In the example usage of the code, historical data (historical_data), current data (current_data), and thresholds (thresholds) can be first defined. Then, the window size (window_size) and anomaly association threshold (association_threshold) are set. Next, the moving average of the historical data is calculated by calling the `calculate_moving_average` function, and the result is stored in the `moving_averages` dictionary. Then, the `detect_initial_anomalies` function is called to detect the initial anomalies in the current data, and the results are stored in the `initial_anomal ies` list. Finally, if there is an initial anomaly, the association between the anomalies is checked by calling the `check_association` function, and the corresponding information is output based on the result.

[0149] If the initial anomaly exists and there is a correlation between anomalies, print "Anomalies are likely due to a city event." Otherwise, print the metric name, current value, moving average, difference, and threshold for each anomaly.

[0150] Example 4

[0151] The present invention also proposes a smart city multi-source data detection and processing system. Figure 2 This is the main structural block diagram of a smart city multi-source data detection and processing system of the present invention. Figure 2 As shown, a smart city multi-source data detection and processing system in this embodiment mainly includes:

[0152] An acquisition unit 11 acquires raw data from multiple sources and cleans the raw data;

[0153] A classification processing unit 12 classifies the cleaned raw data and stores it based on ElasticSearch;

[0154] An index processing unit 13 performs index optimization on ElasticSearch;

[0155] A storage detection unit 14 performs abnormal storage detection after data storage to complete the construction of a multi-source database;

[0156] The data analysis and optimization unit 15 performs data analysis based on a multi-source database for data information optimization of the smart city.

[0157] In some embodiments, one or more of the acquisition unit 11, the classification processing unit 12, the index processing unit 13, the storage detection unit 14 and the data analysis optimization unit 15 can be combined together into one module.

[0158] The above-mentioned smart city multi-source data detection and processing system is used to perform Figure 1 The embodiment of a smart city multi-source data detection and processing method shown in the figure has similar technical principles, technical problems solved and technical effects produced. Technical personnel in this technical field can clearly understand that for the convenience and conciseness of description, the specific working process and related instructions of a smart city multi-source data detection and processing system can refer to the contents described in the embodiment of a smart city multi-source data detection and processing method, which will not be repeated here.

[0159] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A smart city multi-source data detection and processing method, characterized in that: Includes steps: Acquire raw data from multiple sources and clean the raw data, wherein the raw data from multiple sources includes environmental data, user behavior data, social public data, and IoT device data; Classify the cleaned raw data into usable data and unusable data, filter out unusable data, divide the usable data into regular data and redundant data, and perform data compression on the redundant data; Divide regular data into cold data and hot data, and store them based on ElasticSearch; Optimize the ElasticSearch index, extract the fields to be searched to form a separate index data, write the regular data to HBase according to the Rowkey, and write the search data to ElasticSearch according to the Rowkey; After data is stored, abnormal storage detection is performed to complete the construction of a multi-source database; Data analysis based on multi-source databases is used for data information optimization in smart cities.

2. A smart city multi-source data detection and processing method according to claim 1, characterized in that: The process of acquiring the raw data includes acquiring environmental data through sensor detection, or retrieving the environmental data through a public database / commercial database, wherein the environmental data includes environmental temperature data, humidity data, and light data of multiple locations; Acquire user behavior data through sensors and applications built into mobile devices, or retrieve the user behavior data through public databases / commercial databases; Obtaining social public data from governments, enterprises, and organizations through pre-built dedicated data communication paths or web crawlers, including but not limited to statistics bureau data, industry reports, and corporate annual reports; Get real-time IoT device data from smart home devices and industrial automation equipment.

3. A smart city multi-source data detection and processing method according to claim 2, characterized in that: The method of dividing conventional data into cold data and hot data comprises the following steps: All regular data are stored in the cold data area, and an identification ID is set for each piece of data, and the identification ID is initially 0; Writing the time stamp and a specific string into the identification ID, wherein the specific string is a string of 32 characters generated by MD5 and Salt hashing; Determine the data type of current regular data, and determine whether the regular data is cold data or hot data according to the data type; Modify the ID of the hot data to 1; The conventional data with the identification ID of 1 is stored in the hot data area, and then the data in the hot data area and the cold data area are respectively written into the storage areas of different nodes.

4. The method for detecting and processing multi-source data in a smart city according to claim 1, characterized in that: The data compression operation on the redundant data comprises the steps of: Determine the compression algorithm and compression strategy based on the redundant data type and storage requirements; Optimize compression algorithms and compression strategies; Use optimized compression algorithms and compression strategies to compress redundant data; The CRC is used to check the integrity of the compressed redundant data.

5. The method for detecting and processing multi-source data in a smart city according to claim 1, characterized in that: The abnormal storage detection after data storage comprises the steps of: Using a monitoring tool to collect historical data of key performance indicators, the key performance indicators including at least one of average response time, throughput, I / O latency, CPU utilization, memory utilization, and disk space utilization; Analyze historical data and calculate moving averages for each key performance indicator; Obtain the currently stored key performance indicator data and compare whether the difference between the current data and the moving average exceeds the preset threshold; If so, it is considered that there is abnormal storage and an abnormal report is generated.

6. A smart city multi-source data detection and processing method according to claim 5, characterized in that: In the analysis of historical data, a function calculate_moving_average is defined to calculate the moving average. The parameters accepted by the function calculate_moving_average include data and window size. When the CPU determines that the function calculate_moving_average is in the execution state, a list moving_averages is created to store the average value of each data point, and the moving average values ​​are connected to obtain the moving average line. Traverse the original data, if the index of the data point is less than the window size, subtract one, and add "None" to the list moving_averages, indicating that the current data point is not enough to fill the entire window. Otherwise, take the data within the window range, calculate the average value as the valid data, and add the result to the list moving_averages. A function detect_anomalies is defined to detect anomalies between current data and a moving average. The parameters accepted by the function detect_anomalies include current data, a moving average, and a threshold. When the CPU determines that the function detect_anomalies is in an execution state, a list moving_averages is created to store the detected anomalies, each indicator of the current data is traversed, the difference between the current data and the corresponding moving average is calculated, and compared with a preset threshold. If the difference exceeds the threshold, the anomaly information is added to the list moving_averages.

7. A smart city multi-source data detection and processing method according to claim 6, characterized in that: After the anomaly is identified and stored, the anomaly data is double-checked and the initial anomaly data set is set as A{A1,A2,…,A k }, let the associated data array YCX, YCX ij Indicates the association between the i-th data entity and the j-th data entity, Define a function check_association for checking the association between anomalies. The parameters received by the function check_association include the anomaly data set A, the associated data array YCX and the associated threshold. Traverse the data in the list moving_averages to determine the relationship between the set A processed by the refined association matrix and the associated threshold. If any pair of anomalies does not meet the association condition, the anomaly is determined to be a true anomaly. Otherwise, it is a false exception.

8. A smart city multi-source data detection and processing system, characterized in that: include: An acquisition unit, which acquires raw data from multiple sources and cleans the raw data; The classification processing unit classifies the cleaned raw data and stores it based on ElasticSearch; Index processing unit, which optimizes the index of ElasticSearch; The storage detection unit performs abnormal storage detection after data is stored and completes the construction of a multi-source database; The data analysis and optimization unit performs data analysis based on multi-source databases for data information optimization in smart cities.