Intelligent data analysis method based on space-time big data

Abstract

The invention discloses an intelligent data analysis method based on spatio-temporal big data, and belongs to the technical field of spatio-temporal big data, the method comprises the step of obtaining original spatio-temporal data of different sources in real time, and the method comprises the steps of performing feature extraction on preprocessed spatio-temporal data, including unification of a time format, spatial coordinates and a data unit, deduplication fusion and the like, so as to obtain the spatio-temporal data of different sources. According to the method, multi-source spatio-temporal data is formed, a spatio-temporal data analysis model is constructed based on deep learning, spatio-temporal dynamic changes in data can be captured, modes in the spatial data can be effectively extracted through models such as a convolutional neural network, classification is performed through cross entropy loss, high-precision spatio-temporal data analysis is achieved, and the spatial data analysis efficiency is improved through the trained spatio-temporal data analysis model. According to the method, new spatio-temporal data are analyzed in real time, abnormal modes and trend changes in the data are rapidly identified, an anomaly detection threshold value is set, model output is compared with the set threshold value, comprehensive judgment is carried out in combination with context information, and the accuracy and efficiency of anomaly detection are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of spatiotemporal big data, and specifically refers to an intelligent data analysis method based on spatiotemporal big data. Background Art

[0002] With the development of information technology, spatiotemporal big data has been widely used in many fields such as urban planning, traffic management, and environmental monitoring. However, existing data analysis methods often ignore the comprehensive consideration of time and space dimensions, resulting in inaccurate analysis results that cannot meet the growing data analysis needs.

[0003] However, there are still some defects in the existing intelligent data analysis of spatiotemporal big data. The existing intelligent data analysis of spatiotemporal big data relies on timed collection or manual data upload, and cannot achieve real-time collection, resulting in poor timeliness of data and failure to reflect the latest spatiotemporal changes in a timely manner. It relies on a single data source, such as a fixed sensor network or a single public data platform, which lacks diversity and comprehensiveness and cannot cover a wide range of spatiotemporal information. Existing methods often rely on simple threshold settings in anomaly detection and lack the support of contextual information, resulting in low accuracy and reliability of anomaly detection. Simple moving average and other methods are usually used in trend analysis, which cannot effectively identify complex trend changes and periodic patterns, resulting in low prediction accuracy. When generating decision recommendations, they often rely on manual experience and simple rules, lack the support of intelligence and automation, and have low decision-making efficiency and accuracy. Therefore, an intelligent data analysis method based on spatiotemporal big data is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent data analysis method based on spatiotemporal big data to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: an intelligent data analysis method based on spatiotemporal big data, comprising the following steps:

[0006] S1. Real-time acquisition of raw spatiotemporal data from different sources;

[0007] S2. Perform preprocessing operations such as cleaning, denoising, and format unification based on the acquired spatiotemporal data;

[0008] S3, extracting the spatiotemporal features of the data based on the preprocessed spatiotemporal big data;

[0009] S4. Build a spatiotemporal data analysis model based on deep learning to capture the spatiotemporal dynamic changes in the data;

[0010] S5. Use the trained spatiotemporal data analysis model to perform real-time analysis on new spatiotemporal data to identify abnormal patterns and trend changes in the data;

[0011] S6. Make decision suggestions based on the analysis and prediction results, combined with business logic and rule engine;

[0012] S7. Visualize the analysis and prediction results to show the changing trend and spatial distribution pattern of spatiotemporal data.

[0013] Among them, S1 obtains original spatiotemporal data from different sources in real time; obtains data through sensors, remote sensing satellites and public data sources, collects data from data sources in real time by setting a frequency, marks a timestamp for each collected data point, and stores the acquired spatiotemporal data in a database.

[0014] Among them, the S2 performs preprocessing operations such as cleaning, denoising, and format unification based on the acquired spatiotemporal data; performs data cleaning based on the acquired spatiotemporal data, identifies missing values ​​in the data, and fills the missing values ​​through linear interpolation. If there are too many missing values ​​in a field and it cannot be judged, the corresponding field is deleted; outliers in data points are identified through statistical analysis, and abnormal data that deviates from the normal range are corrected through interpolation; the average value of the data is calculated through a sliding window, and high-frequency noise is removed.

[0015] Among them, the S3 unifies the data format according to the preprocessed spatiotemporal data, including the time format, spatial coordinates and data units, deduplicates the data after the unified format, deduplicates the data according to the timestamp and spatial coordinates, and performs weighted averaging of multiple data at the same time and location to fuse the multi-source data to form multi-source spatiotemporal data.

[0016] In which, S4 constructs a spatiotemporal data analysis model based on deep learning to capture the spatiotemporal dynamic changes in the data; constructs a convolutional neural network model, and the construction implementation formula is:

[0017] y=f(W·x+b),

[0018] In the formula, y represents the output of the convolution layer, W represents the convolution kernel, x represents the input data, b represents the bias term, and f represents the activation function.

[0019] Wherein, said S4, according to the analysis objectives, divides the data into a training set, a validation set and a test set, normalizes the numerical features; and captures patterns in the spatial data through a convolutional neural network model;

[0020] The spatiotemporal data is classified by cross entropy loss, and the implementation formula is:

[0021]

[0022] In the formula, yij represents the true label, Represents the predicted probability, n represents the number of samples, m represents the number of categories, the hyperparameters of the model are adjusted by the Bayesian optimization method, the model is trained by the training set, and the model performance is evaluated by the validation set.

[0023] Among them, the S5 performs real-time analysis on new spatiotemporal data through a trained spatiotemporal data analysis model to identify abnormal patterns and trend changes in the data; obtains new spatiotemporal data, including time series data and spatial location information, and performs data preprocessing, and the preprocessed data is input into the model for analysis, and a threshold for anomaly detection is set according to the statistical characteristics of the training data, and the model output is compared with the set threshold to identify data points that exceed the threshold, and a comprehensive judgment is made in combination with context information.

[0024] Wherein, said S5, identifies trend changes according to the data output by the model, assuming that the time series data is {x t}, time series analysis is performed through moving average, and the implementation formula is:

[0025]

[0026] In the formula, MA(x t ) represents the time series x t The moving average value, x t It represents the value of the time series at a point in time, and n represents the window size of the moving average.

[0027] Among them, S6 makes decision recommendations based on the analysis results and prediction results, combined with business logic and rule engine; by parsing the analysis results and prediction results, defining rules according to business needs, matching the parsed results with the defined rules, generating decision recommendations, and outputting the generated decision recommendations.

[0028] Among them, the S7 performs a visualization display based on the analysis and prediction results to display the changing trend and spatial distribution law of spatiotemporal data; obtains the analysis and prediction results for visualization, arranges the data points in timeline order, and displays the changing trend at different time granularities. For gridded data, it is displayed in the form of color blocks, each color block represents the value of a grid unit. For geographic data, a map background can be selected to cover heat maps of different areas to display the spatial distribution. The spatial data is gradually displayed according to the time dimension, and the data points are mapped to the geographic space through a 3D map. The spatial distribution of each time period is displayed on the timeline, allowing users to interactively select different time ranges, spatial areas, prediction models, etc., dynamically update the visualization results, and embed the visualization results into a centralized display platform to provide real-time data updates and feedback.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention collects data in real time at a set frequency through sensors, remote sensing satellites and public data sources to ensure the timeliness of the data, and performs pre-processing operations such as cleaning, denoising and format unification on the collected data to effectively identify and process missing values, outliers and high-frequency noise, thereby improving data quality;

[0031] 2. The present invention extracts features from the preprocessed spatiotemporal data, including unification of time format, spatial coordinates and data units, as well as deduplication and fusion, to form multi-source spatiotemporal data, and constructs a spatiotemporal data analysis model based on deep learning, which can capture the spatiotemporal dynamic changes in the data, effectively extract patterns in spatial data through models such as convolutional neural networks, and classify through cross entropy loss to achieve high-precision spatiotemporal data analysis;

[0032] 3. The present invention uses a trained spatiotemporal data analysis model to perform real-time analysis on new spatiotemporal data, quickly identify abnormal patterns and trend changes in the data, set anomaly detection thresholds, compare the model output with the set thresholds, and make a comprehensive judgment based on contextual information to improve the accuracy and efficiency of anomaly detection;

[0033] 4. The present invention generates intelligent decision-making suggestions based on the analysis results and prediction results, combined with business logic and rule engines. These suggestions can directly guide business operations and improve decision-making efficiency and accuracy. Through visualization technology, data points are arranged in timeline order to display the changing trends at different time granularities. For grid data and geographic data, color blocks and map backgrounds are used for display to intuitively present the changing trends and spatial distribution patterns of spatiotemporal data. At the same time, users are allowed to interactively select different time ranges, spatial areas, prediction models, etc., to dynamically update the visualization results and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The operation process of the intelligent data analysis method based on spatiotemporal big data of the present invention is as follows Figure 1 ;

[0035] Figure 2 The operation process of the intelligent data analysis method based on spatiotemporal big data of the present invention is as follows Figure 2 ;

[0036] Figure 3 The operation process of the intelligent data analysis method based on spatiotemporal big data of the present invention is as follows Figure 3 ;

[0037] Figure 4 The operation process of the intelligent data analysis method based on spatiotemporal big data of the present invention is as follows Figure 4. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example

[0040] See also Figure 1-Figure 2 As shown, the present invention provides a technical solution: comprising the following steps:

[0041] S1. Real-time acquisition of raw spatiotemporal data from different sources;

[0042] S2. Perform preprocessing operations such as cleaning, denoising, and format unification based on the acquired spatiotemporal data;

[0043] S3, extracting the spatiotemporal features of the data based on the preprocessed spatiotemporal big data;

[0044] S4. Build a spatiotemporal data analysis model based on deep learning to capture the spatiotemporal dynamic changes in the data;

[0045] S5. Use the trained spatiotemporal data analysis model to perform real-time analysis on new spatiotemporal data to identify abnormal patterns and trend changes in the data;

[0046] S6. Make decision suggestions based on the analysis and prediction results, combined with business logic and rule engine;

[0047] S7. Visualize the analysis and prediction results to show the changing trend and spatial distribution pattern of spatiotemporal data.

[0048] Among them, S1 obtains original spatiotemporal data from different sources in real time; obtains data through sensors, remote sensing satellites and public data sources, collects data from data sources in real time by setting a frequency, marks a timestamp for each collected data point, and stores the acquired spatiotemporal data in a database.

[0049] Among them, the S2 performs preprocessing operations such as cleaning, denoising, and format unification based on the acquired spatiotemporal data; performs data cleaning based on the acquired spatiotemporal data, identifies missing values ​​in the data, and fills the missing values ​​through linear interpolation. If there are too many missing values ​​in a field and it cannot be judged, the corresponding field is deleted; outliers in data points are identified through statistical analysis, and abnormal data that deviates from the normal range are corrected through interpolation; the average value of the data is calculated through a sliding window, and high-frequency noise is removed.

[0050] Among them, the S3 unifies the data format according to the preprocessed spatiotemporal data, including the time format, spatial coordinates and data units, deduplicates the data after the unified format, deduplicates the data according to the timestamp and spatial coordinates, and performs weighted averaging of multiple data at the same time and location to fuse the multi-source data to form multi-source spatiotemporal data.

[0051] In which, S4 constructs a spatiotemporal data analysis model based on deep learning to capture the spatiotemporal dynamic changes in the data; constructs a convolutional neural network model, and the construction implementation formula is:

[0052] y=f(W·x+b),

[0053] In the formula, y represents the output of the convolution layer, W represents the convolution kernel, x represents the input data, b represents the bias term, and f represents the activation function.

[0054] Wherein, said S4, according to the analysis objectives, divides the data into a training set, a validation set and a test set, normalizes the numerical features; and captures patterns in the spatial data through a convolutional neural network model;

[0055] The spatiotemporal data is classified by cross entropy loss, and the implementation formula is:

[0056]

[0057] In the formula, y ij represents the true label, Represents the predicted probability, n represents the number of samples, m represents the number of categories, the hyperparameters of the model are adjusted by the Bayesian optimization method, the model is trained by the training set, and the model performance is evaluated by the validation set.

[0058] Among them, the S5 performs real-time analysis on new spatiotemporal data through a trained spatiotemporal data analysis model to identify abnormal patterns and trend changes in the data; obtains new spatiotemporal data, including time series data and spatial location information, and performs data preprocessing, and the preprocessed data is input into the model for analysis, and a threshold for anomaly detection is set according to the statistical characteristics of the training data, and the model output is compared with the set threshold to identify data points that exceed the threshold, and a comprehensive judgment is made in combination with context information.

[0059] Wherein, said S5, identifies trend changes according to the data output by the model, assuming that the time series data is {x t}, time series analysis is performed through moving average, and the implementation formula is:

[0060]

[0061] In the formula, MA(x t ) represents the time series x t The moving average value, x t It represents the value of the time series at a point in time, and n represents the window size of the moving average.

[0062] Among them, S6 makes decision recommendations based on the analysis results and prediction results, combined with business logic and rule engine; by parsing the analysis results and prediction results, defining rules according to business needs, matching the parsed results with the defined rules, generating decision recommendations, and outputting the generated decision recommendations.

[0063] Among them, the S7 performs a visualization display based on the analysis and prediction results to display the changing trend and spatial distribution law of spatiotemporal data; obtains the analysis and prediction results for visualization, arranges the data points in timeline order, and displays the changing trend at different time granularities. For gridded data, it is displayed in the form of color blocks, each color block represents the value of a grid unit. For geographic data, a map background can be selected to cover heat maps of different areas to display the spatial distribution. The spatial data is gradually displayed according to the time dimension, and the data points are mapped to the geographic space through a 3D map. The spatial distribution of each time period is displayed on the timeline, allowing users to interactively select different time ranges, spatial areas, prediction models, etc., dynamically update the visualization results, and embed the visualization results into a centralized display platform to provide real-time data updates and feedback.

[0064] Working principle: Real-time spatiotemporal data is acquired through sensors, remote sensing satellites and public data sources. Timestamps are marked for each collected data point, and the acquired spatiotemporal data is stored in the database. Missing values ​​in the data are identified and filled in through methods such as linear interpolation. If there are too many missing values ​​in a field and it cannot be determined, the corresponding field is deleted. Outliers in the data points are identified through statistical analysis, and interpolation is used to correct the abnormal data that deviates from the normal range. At the same time, the average value of the data is calculated through a sliding window, high-frequency noise is removed, and the time format, spatial coordinates and data units are unified.

[0065] By removing duplicates based on timestamps and spatial coordinates, multi-source data is fused by weighted averaging of multiple data at the same time and location to form multi-source spatiotemporal data, and time-space related features are extracted from the preprocessed data. A spatiotemporal data analysis model is built based on deep learning, and the data is divided into training set, validation set and test set, and the numerical features are normalized; the convolutional neural network model is used to capture patterns in spatial data; the spatiotemporal data is classified through cross entropy loss; the hyperparameters of the model are adjusted through the Bayesian optimization method, the model is trained using the training set, the model performance is evaluated through the validation set, and the newly acquired spatiotemporal data is preprocessed, including data cleaning, denoising and format unification. The preprocessed data is input into the model for analysis, and the threshold for anomaly detection is set according to the statistical characteristics of the training data. The model output is compared with the set It compares the results with the specified threshold, identifies the data points that exceed the threshold, and makes a comprehensive judgment based on the context information. It performs time series analysis through methods such as moving average to predict the changing trend of future spatiotemporal data. It defines rules according to business needs, matches the parsed analysis results and prediction results with the defined rules, generates decision suggestions based on the matching results, and outputs the generated decision suggestions. It arranges the data points in timeline order to show the changing trends at different time granularities. For gridded data, it uses color blocks for display. For geographic data, you can choose to cover the map background with heat maps of different areas to show the spatial distribution. It allows users to interactively select different time ranges, spatial areas, prediction models, etc., dynamically update the visualization results, and embed the visualization results into a centralized display platform to provide real-time data updates and feedback.

[0066] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0067] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. An intelligent data analysis method based on spatiotemporal big data, characterized in that: The following steps are involved: S1. Real-time acquisition of raw spatiotemporal data from different sources; S2. Perform preprocessing operations such as cleaning, denoising, and format unification based on the acquired spatiotemporal data; S3, extracting the spatiotemporal features of the data based on the preprocessed spatiotemporal big data; S4. Build a spatiotemporal data analysis model based on deep learning to capture the spatiotemporal dynamic changes in the data; S5. Use the trained spatiotemporal data analysis model to perform real-time analysis on new spatiotemporal data to identify abnormal patterns and trend changes in the data; S6. Make decision suggestions based on the analysis and prediction results, combined with business logic and rule engine; S7. Visualize the analysis and prediction results to show the changing trend and spatial distribution pattern of spatiotemporal data.

2. The intelligent data analysis method based on spatiotemporal big data according to claim 1 is characterized in that: The S1 acquires original spatiotemporal data from different sources in real time; acquires data through sensors, remote sensing satellites and public data sources, collects data from data sources in real time by setting a frequency, marks a timestamp on each collected data point, and stores the acquired spatiotemporal data in a database.

3. The intelligent data analysis method based on spatiotemporal big data according to claim 1 is characterized in that: The S2 performs preprocessing operations such as cleaning, denoising, and format unification based on the acquired spatiotemporal data; performs data cleaning based on the acquired spatiotemporal data, identifies missing values ​​in the data, and fills the missing values ​​through linear interpolation. If there are too many missing values ​​in a field and it cannot be determined, the corresponding field is deleted; outliers in data points are identified through statistical analysis, and based on the identified abnormal data that deviates from the normal range, corrections are made through interpolation; the average value of the data is calculated through a sliding window, and high-frequency noise is removed.

4. The intelligent data analysis method based on spatiotemporal big data according to claim 3 is characterized by: The S3 unifies the data format according to the preprocessed spatiotemporal data, including the time format, spatial coordinates and data units, deduplicates the data after the unified format, deduplicates the data according to the timestamp and spatial coordinates, performs weighted averaging on multiple data at the same time and location, and fuses the multi-source data to form multi-source spatiotemporal data.

5. The intelligent data analysis method based on spatiotemporal big data according to claim 1 is characterized in that: S4, based on deep learning, constructs a spatiotemporal data analysis model to capture the spatiotemporal dynamic changes in the data; constructs a convolutional neural network model, and the construction implementation formula is: y=f(W·x+b), In the formula, y represents the output of the convolution layer, W represents the convolution kernel, x represents the input data, b represents the bias term, and f represents the activation function.

6. The intelligent data analysis method based on spatiotemporal big data according to claim 5 is characterized in that: S4, according to the analysis objectives, divides the data into a training set, a validation set and a test set, normalizes the numerical features; and captures patterns in the spatial data through a convolutional neural network model; The spatiotemporal data is classified by cross entropy loss, and the implementation formula is: In the formula, y ij represents the true label, Represents the predicted probability, n represents the number of samples, m represents the number of categories, the hyperparameters of the model are adjusted by the Bayesian optimization method, the model is trained by the training set, and the model performance is evaluated by the validation set.

7. The intelligent data analysis method based on spatiotemporal big data according to claim 1 is characterized in that: The S5 performs real-time analysis on new spatiotemporal data through a trained spatiotemporal data analysis model to identify abnormal patterns and trend changes in the data; obtains new spatiotemporal data, including time series data and spatial location information, and performs data preprocessing, and inputs the preprocessed data into the model for analysis, sets a threshold for anomaly detection based on the statistical characteristics of the training data, compares the model output with the set threshold, identifies data points that exceed the threshold, and makes a comprehensive judgment based on contextual information.

8. The intelligent data analysis method based on spatiotemporal big data according to claim 7 is characterized in that: S5, according to the data output by the model, identifies the trend change, assuming that the time series data is {x t }, time series analysis is performed through moving average, and the implementation formula is: In the formula, MA(x t ) represents the time series x t The moving average value, x t It represents the value of the time series at a point in time, and n represents the window size of the moving average.

9. The intelligent data analysis method based on spatiotemporal big data according to claim 1 is characterized in that: The S6 makes decision recommendations based on the analysis results and prediction results in combination with the business logic and rule engine; by parsing the analysis results and prediction results, defining rules according to business needs, matching the parsed results with the defined rules, generating decision recommendations, and outputting the generated decision recommendations.

10. The intelligent data analysis method based on spatiotemporal big data according to claim 1, characterized in that: The S7 performs a visualization based on the analysis and prediction results to display the changing trend and spatial distribution law of spatiotemporal data; obtains the analysis and prediction results for visualization, arranges the data points in timeline order, and displays the changing trend at different time granularities. For gridded data, it is displayed in the form of color blocks, each color block represents the value of a grid unit. For geographic data, a map background can be selected to cover heat maps of different areas to display the spatial distribution. The spatial data is gradually displayed according to the time dimension, and the data points are mapped to the geographic space through a 3D map. The spatial distribution of each time period is displayed on the timeline, allowing users to interactively select different time ranges, spatial areas, prediction models, etc., dynamically update the visualization results, and embed the visualization results into a centralized display platform to provide real-time data updates and feedback.

Images