Real-time data analysis and visualization system and method based on big data
By using a real-time data analysis and visualization system based on big data, the problems of data latency, adaptation, and accuracy in existing technologies have been solved. This system enables low-latency, flexible, and highly accurate data analysis and visualization, meeting the needs of enterprise-level real-time decision-making and rapid response.
Patent Information
- Application Number
- CN202511300033.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-12
- Publication Date
- 2025-12-12
AI Technical Summary
Existing technologies for real-time data analysis and visualization systems in big data environments suffer from problems such as data acquisition-computation-presentation delays, limited data source adaptation, high scalability complexity, and inaccurate analysis results due to improper handling of outliers, failing to meet the needs of enterprise-level real-time decision-making and rapid response.
We employ a real-time data analysis and visualization system based on big data, including a data acquisition module, a data transmission module, a data preprocessing module, a real-time analysis module, and a visualization module. Through breakthroughs in real-time performance, upgrades in scalability, optimization of accuracy, and reductions in ease of use, combined with a 'CDC+ lightweight collector', a layered decoupling and distributed elastic architecture, and multi-algorithm fusion optimization, we achieve real-time data processing and highly accurate data display.
It achieves a low-latency closed loop from data generation to visualization, with flexible data source adaptation and expansion, and analysis result error controlled within 1%. It supports enterprise-level real-time decision-making and rapid response, meeting the needs of multi-scenario adaptation and autonomous business operation.
Smart Images

Figure CN121117092A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology, specifically relating to a real-time data analysis and visualization system and method based on big data. Background Technology
[0002] With the rapid development of society and the national economy, the number of enterprises in each region is increasing, which also increases the difficulty of overall enterprise management. Monitoring and effectively managing each enterprise from a macro perspective is becoming increasingly complex. Enterprise activity is an important indicator for measuring the current state of enterprise development and can effectively provide managers in a target region with information on the operational status of their enterprises. Currently, the era of big data has arrived, and big data has become a new focus of attention for governments and societies worldwide. The big data industry, following cloud computing, the Internet of Things, and mobile internet, is a product of the deep integration of next-generation information technology with new business applications such as e-commerce and intelligent manufacturing. It is becoming a new hotspot in information technology and a new direction for industrial development, which will have a significant impact on human production and life and bring profound changes to economic and social development. Grasping the development direction of big data, promoting big data development and application, and developing the big data service industry are conducive to promoting the innovative development and transformation and upgrading of Wuhan Iron and Steel Group's high-tech industrial sector.
[0003] For example, the invention disclosed in patent application CN119396997B, a real-time data analysis and visualization method and system in a big data environment, relates to the field of data processing technology. This includes acquiring and preprocessing documents to be processed; performing hierarchical structure analysis and semantic understanding analysis; establishing a semantic index mapping; receiving natural language processing instructions input by the user; performing intent understanding and semantic parsing; matching semantic understanding tasks; processing corresponding semantic fragment units; obtaining the target document and displaying it visually. This solution can achieve intelligent processing and analysis of different types of documents, improving data processing efficiency.
[0004] However, the above solutions have the following shortcomings: 1. There is a delay in the "data collection-calculation-presentation" process, which can only meet non-real-time needs such as "next-day reports"; 2. There are problems of "limited data source adaptation" or "high complexity of expansion", which makes it difficult to cope with the "surge in data volume + new data sources" brought about by business growth; 3. The above systems do not handle outliers, which leads to statistical bias and low accuracy of analysis results, making it impossible to support accurate decision-making. Therefore, we propose a real-time data analysis and visualization system and method based on big data. Summary of the Invention
[0005] The purpose of this invention is to provide a real-time data analysis and visualization system and method based on big data. Through a comprehensive design that includes "real-time breakthrough, scalability upgrade, accuracy optimization, ease of use reduction, reliability assurance, and cost control", it solves the pain points of existing technologies in terms of "low latency requirements, multi-scenario adaptation, and autonomous business operation", and can better meet the core needs of enterprises for "real-time decision-making, rapid response, and compliance and security".
[0006] To achieve the above objectives, the present invention provides the following technical solution: a real-time data analysis and visualization system based on big data, comprising a data acquisition module, a data transmission module, a data preprocessing module, a real-time analysis module, a data storage module, and a visualization module; The data acquisition module is connected to the data transmission module, the data transmission module is connected to the data preprocessing module, the data preprocessing module is connected to the real-time architecture module, and the real-time analysis module is connected to the data storage module and the visualization module.
[0007] Preferably, the data acquisition module is used to capture and access raw data from multiple heterogeneous data sources in real time to ensure data integrity and timeliness, and the data transmission module is used to forward, buffer and control the flow of the collected raw data in real time to avoid overload of downstream processing pressure.
[0008] Preferably, the data preprocessing module is used to clean, transform, and integrate the raw data to solve data quality problems and provide standardized data for the analysis module.
[0009] Preferably, the real-time analysis module is used to perform real-time calculations (statistical analysis, feature extraction, predictive inference) on the preprocessed data to generate analysis results.
[0010] Preferably, the data storage module is used to store raw data, preprocessed data, and analysis results respectively to meet different query needs.
[0011] Preferably, the visualization module is used to display the analysis results to the user in an intuitive form and to provide interactive functions and application interfaces.
[0012] Real-time data analysis and visualization methods based on big data include the following steps: S1. The data acquisition module determines the data source type (such as application logs, database changes, IoT sensor data, API interface data) based on business requirements, and selects the corresponding acquisition tool (such as Filebeat for logs and Debezium for database changes). At the same time, it configures the parameters of the acquisition tool. The acquisition tool encapsulates the raw data into messages (such as Kafka's ProducerRecord) and sends them to the distributed message queue according to the business topic (such as "user_behavior" and "order_payment"). The distributed message queue is then transmitted to the data transmission module. S2. The data transmission module transmits the distributed message queue to the data preprocessing module; S3, the data preprocessing module cleans, transforms, maps, desensitizes, and integrates the data, and then sends the processed data to the real-time analysis module; S4. The real-time analysis module determines the analysis type based on business needs, which is divided into statistical analysis, feature extraction, and predictive inference. It calls the aggregation algorithm for statistical analysis and the time series prediction algorithm for predictive inference to perform calculations and generate intermediate results. S5. Deduplicatize and merge intermediate results (take the average of multiple calculation results in the same dimension), and store them in the real-time database (such as Redis for high-frequency queries) or time-series database in the data storage module according to the data popularity. S6, the visualization module, transforms the analysis results into intuitive visual charts for users to view and interact with.
[0013] Preferably, the data cleaning in S3 specifically includes: S30. Deduplication: Based on unique identifiers (such as user ID + operation time) or data fingerprints (such as MD5 hash), duplicate data is removed using Flink's distinct() operator or Redis's Set structure. S31. Outlier handling: Outlier detection algorithms are used to identify outliers. For example, if the order amount exceeds 1 million, it is considered an outlier and the median of the field is used to fill the gap. S32. Missing value handling: Fill key fields (such as payment amount) with the mean / median, and mark non-key fields (such as remarks) with "NULL".
[0014] Preferably, the aggregation algorithm in S4 has the following specific steps: In Flink, group by business dimension (such as product ID, region) using keyBy(); call window() to specify the window type (such as TumblingProcessingTimeWindows.of(Time.minutes(1))); call aggregation functions (such as sum("amount"), count()) to calculate the window result, and achieve global aggregation through windowAll().
[0015] Preferably, the time series prediction algorithm in S4 is as follows: First smoothing value: ; Secondary smoothing value: ; Prediction formula: ; Where, t: current time (timestamp), representing the current time point being processed, T: prediction duration (step size), representing the time interval to be predicted from the current time t, α: smoothing coefficient, with a value range of (0,1), usually 0.1~0.3, St(1): first exponential smoothing value at time t, xt: original data value at time t, St-1(1): first exponential smoothing value at time t-1, St(2): second exponential smoothing value at time t, St-1(2): second exponential smoothing value at time t-1, ct: horizontal estimate (intercept term) at time t, bt: trend estimate (slope term) at time t. : The predicted value at time t+T.
[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) Adopt the combination of "CDC (Change Data Capture) + Lightweight Collector" (such as Debezium + FlinkCDC) to avoid the minute-level delay of traditional Sqoop "timed pull" and realize the millisecond-level access of database change data.
[0017] (2) Through layered decoupling and distributed elastic architecture, flexible expansion of “data source-computing node-storage capacity” is achieved.
[0018] (3) By combining full-process data quality control with multi-algorithm fusion optimization, the error of the analysis results is controlled within 1%. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the framework of a real-time data analysis and visualization system based on big data. Figure 2 This is a flowchart illustrating a real-time data analysis and visualization method based on big data. Figure 3 This is a schematic diagram of the data cleaning process in this invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Please see Figure 1 The present invention provides a technical solution: a real-time data analysis and visualization system based on big data, including a data acquisition module, a data transmission module, a data preprocessing module, a real-time analysis module, a data storage module, and a visualization module; The data acquisition module is connected to the data transmission module, the data transmission module is connected to the data preprocessing module, the data preprocessing module is connected to the real-time architecture module, and the real-time analysis module is connected to the data storage module and the visualization module.
[0022] Furthermore, the data acquisition module is used to capture and access raw data from multiple heterogeneous data sources in real time to ensure data integrity and timeliness. The data transmission module is used to forward, buffer, and control the flow of the collected raw data in real time to avoid overload of downstream processing pressure.
[0023] Furthermore, the data preprocessing module is used to clean, transform, and integrate the raw data to solve data quality problems and provide standardized data for the analysis module.
[0024] Furthermore, the real-time analysis module is used to perform real-time calculations (statistical analysis, feature extraction, and predictive inference) on the preprocessed data to generate analysis results.
[0025] Furthermore, the data storage module is used to store raw data, preprocessed data, and analysis results respectively to meet different query needs.
[0026] Preferably, the visualization module is used to display the analysis results to the user in an intuitive form and to provide interactive functions and application interfaces.
[0027] Please refer to Figure 2 Real-time data analysis and visualization methods based on big data include the following steps: Step 1: The data acquisition module determines the data source type (such as application logs, database changes, IoT sensor data, API interface data) based on business requirements, and selects the corresponding acquisition tool (such as Filebeat for logs and Debezium for database changes). At the same time, the acquisition tool configures the parameters. The acquisition tool encapsulates the raw data into messages (such as Kafka's ProducerRecord) and sends them to the distributed message queue according to the business topic (such as "user_behavior" and "order_payment"). The distributed message queue is then transmitted to the data transmission module. Data source identification and adaptation: Determine the data source type based on business needs (such as application logs, database changes, IoT sensor data, API interface data), and select the corresponding collection tool (such as Filebeat for logs and Debezium for database changes). Data collection configuration and startup: Configure parameters for the data collection tools (such as collection frequency, data format, target address). For example, configure the log file path and Kafka topic for Filebeat, and configure the database connection information and table-level synchronization rules for Debezium. Data is transmitted to the message queue in real time: The acquisition tool encapsulates the raw data into messages (such as Kafka's ProducerRecord) and sends them to the distributed message queue according to business topics (such as "user_behavior" and "order_payment") to achieve data buffering and decoupling.
[0028] Step 2: The data transmission module transmits the distributed message queue to the data preprocessing module; Step 3: The data preprocessing module cleans, transforms, maps, de-identifies, and integrates the data, and then sends the processed data to the real-time analysis module. Please see Figure 3 The data cleaning in step 3 specifically involves: Step 30, Deduplication: Based on a unique identifier (such as user ID + operation time) or data fingerprint (such as MD5 hash), duplicate data is deleted using Flink's distinct() operator or Redis's Set structure; Data deduplication algorithm – Bloom filter based on Redis Algorithm principle: A Bloom filter is a space-efficient probabilistic data structure that maps data to multiple bits of a binary vector using multiple hash functions to determine whether the data exists (if there is a false positive rate, there is no false negative rate), and can quickly filter out duplicate data.
[0029] Calculation steps: Initialize the Bloom filter: Set the binary vector length m and the number of hash functions k (calculated according to the false positive rate requirement, such as k=10 when the false positive rate is 0.1%). Insert data: For each data e, calculate k positions using k hash functions, and set the corresponding positions in the binary vector to 1; Deduplication determination: For new data e, if all k hash positions are 1, it is determined to be duplicate data and filtered directly; otherwise, it is retained and inserted into a Bloom filter.
[0030] Application scenarios: Real-time deduplication in high-throughput scenarios (such as log data, user click data), reducing memory usage by more than 90% compared to traditional database deduplication.
[0031] Implementation tool: RedisBloom module (providing BF.ADD and BF.EXISTS commands), combined with Flink's RichMapFunction to call the Redis interface. Step 31, Outlier Handling: Use statistical methods (such as the 3σ principle, IQR quartile method) to identify outliers. For example, if the order amount exceeds 1 million, it is considered an outlier and the median of the field is used to fill it. Calculation steps: Calculate the mean μ of the target field (e.g., order amount) using the formula μ = (x1 + x2 + ... + x...). n ) / n; xᵢ: The i-th data value in the target field (e.g., the amount of the i-th order), where i ranges from 1 to n (i=1,2,...,n), n: the number of samples in the target field (e.g., the total number of orders), x1, x2,..., x n : These represent the 1st, 2nd, ... nth data value (e.g., a list of all order amounts). Calculate the standard deviation ; Judgment rule: If xᵢ < μ-3σ or xᵢ > μ+3σ, then xᵢ is an outlier.
[0032] Step 32, missing value handling: Fill key fields (such as payment amount) with the mean / median, and mark non-key fields (such as remarks) with "NULL"; Data transformation: Format conversion: Parse JSON / CSV format data into structured data (such as Flink's Row type) and improve transmission efficiency through Avro serialization; Field mapping: Align source data fields with the standard schema (e.g., map "user_name" to "username", "pay_amt" to "payment_amount"); Data anonymization: Sensitive fields (such as mobile phone numbers and ID card numbers) are encrypted using algorithms (such as AES) or masked. Data integration: Merging multi-source data by association key (such as user ID), for example, merging "user behavior logs" with "user basic information" and supplementing with dimensions such as user age and region. Step 4: The real-time analysis module determines the analysis type based on business needs, which is divided into statistical analysis, feature extraction, and predictive inference. It calls the aggregation algorithm for statistical analysis and the time series prediction algorithm for predictive inference to perform calculations and generate intermediate results. The aggregation algorithm has the following specific steps: In Flink, group by business dimension (such as product ID, region) using keyBy(); call window() to specify the window type (such as TumblingProcessingTimeWindows.of(Time.minutes(1))); call aggregation functions (such as sum("amount"), count()) to calculate the window result, and use windowAll() to achieve global aggregation; Window types and application scenarios are shown in the table below: Calculation steps: Initialize a min-heap of size K (e.g., K=10, to find the "Top 10 products with real-time sales"). Iterate through each data record in the window (such as product sales records) and extract key fields (such as sales volume). If the heap is not full, insert the data directly into the heap; if the heap is full, compare the current data with the data at the top of the heap. If the current sales volume is greater than the sales volume at the top of the heap, delete the top of the heap, insert the current data, and adjust the heap to a min-heap using the "sink" operation; Otherwise, skip the current data; When the window ends, all elements in the heap constitute the Top-K result, which can be output in descending order of sales volume.
[0033] Application scenarios: Real-time hot topic analysis (such as "Top 10 hot search terms" and "Top 5 best-selling products"), supporting the rapid output of sorting results within a second-level window.
[0034] The time series prediction algorithm in S4 Specifically as follows: First smoothing value: ; Secondary smoothing value: ; Prediction formula: ; Where, t: current time (timestamp), representing the current time point being processed, T: prediction duration (step size), representing the time interval from the current time t forward, α: smoothing coefficient, with a value range of (0, 1), usually 0.1~0.3, St(1): first exponential smoothing value at time t, xt: original data value at time t, St-1(1): first exponential smoothing value at time t-1, St(2): second exponential smoothing value at time t, St-1(2): second exponential smoothing value at time t-1, ct: horizontal estimate (intercept term) at time t, bt: trend estimate (slope term) at time t. : The predicted value at time t+T.
[0035] Step 5: Deduplicatize and merge intermediate results (take the average of multiple calculation results in the same dimension), and store them in the real-time database (such as Redis for high-frequency queries) or time-series database in the data storage module according to the data popularity. Step 6: The visualization module converts the analysis results into intuitive visual charts for users to view and interact with.
[0036] Therefore, this application has the following advantages: 1. Real-time performance: low-latency closed loop from "T+ hours" to "T+ seconds". This system achieves end-to-end latency control within 7 seconds (T+0 to T+7 seconds) from data generation to visualization presentation through end-to-end low-latency design. The core optimization points and advantages are as follows: Low latency at the data acquisition layer: The combination of "CDC (Change Data Capture) + lightweight collector" (such as Debezium + FlinkCDC) is adopted to avoid the minute-level latency of traditional Sqoop "timed pull" and achieve "millisecond-level access" of database change data; Low latency at the computation layer: Based on Flink's "incremental computation + state backend optimization" (such as RocksDB state backend), it supports window computation (such as 5-second scrolling window) with a result output latency of <2 seconds, far exceeding Spark Streaming (based on micro-batch processing, with a minimum latency of typically >10 seconds). Non-blocking transport layer: Through Kafka's partitioned replication mechanism and flow control (such as rate limiting on the Producer side and back pressure on the Consumer side), transmission delays caused by data backlog are avoided, ensuring non-blocking forwarding of millions of messages per second; Application layer real-time synchronization: WebSocket bidirectional communication is used to replace the traditional visualization solution of "timed page refresh". After the analysis results are updated, they can be synchronized to the front-end chart within 1 second, and users can get the latest data without manually refreshing.
[0037] 2. Scalability: Multi-source adaptation and dynamic expansion to cope with business growth. This system achieves flexible expansion of "data source - compute node - storage capacity" through layered decoupling and distributed elastic architecture, with the following specific advantages: Multi-source heterogeneous data adaptation capability: The acquisition layer supports four types of data sources: "structured (MySQL / Oracle), semi-structured (JSON / XML), unstructured (log / image metadata), and streaming data (IoTMQTT messages)". The format is standardized through a unified schema registry (such as ConfluentSchemaRegistry), so there is no need to reconstruct the underlying architecture for new data sources. Dynamic scaling of compute nodes: Based on the K8s containerized deployment of the stream computing engine (Flink / SparkStreaming), it supports automatically adjusting the number of TaskManagers according to the data throughput (e.g., when the number of messages per second increases from 100,000 to 1 million, the number of compute nodes is automatically increased from 3 to 15), and the scaling process does not interrupt the running analysis tasks. Storage tiered elastic scaling: The storage tier adopts a tiered architecture of "hot data (Redis) + warm data (ClickHouse) + cold data (HDFS / S3)". Hot data can be horizontally scaled through the Redis cluster (by adding slave nodes) to improve read performance, while cold data can be infinitely expanded in capacity through object storage (such as S3), balancing performance and cost. Functional module plug-in expansion: The analysis layer and visualization layer adopt a plug-in design (for example, when adding a "user profile analysis" function, only the corresponding Flink calculation plugin and ECharts visualization plugin need to be developed, without modifying the existing process), and the function expansion cycle is shortened from the traditional "weekly level" to "dayly level".
[0038] 3. Accuracy: Algorithm fusion and data quality control reduce analytical errors. This system controls the error of analysis results to within 1% through full-process data quality control and multi-algorithm fusion optimization. Its core advantages are as follows: Data preprocessing stage: Multi-dimensional quality control: Compared to the simple cleaning of traditional systems that only "deduplicates", this system integrates a four-fold mechanism of "3σ principle outlier handling + Bloom filter deduplication + sensitive field desensitization + schema validation". For example, for the "order amount" field, the 3σ principle is first used to filter out "1 million fake orders", and then the Bloom filter is used to remove duplicate orders, ensuring that the data input to the calculation layer is "free of anomalies, duplications and errors". Introduce "data quality monitoring indicators" (such as missing value rate, outlier rate, and duplication rate). If the missing value rate of a data source exceeds 5%, an alarm will be automatically triggered and a backup data source will be switched to avoid "dirty data" affecting the calculation results. Analysis Phase: Algorithm Adaptation to Scenario-Specific Requirements For time series forecasting scenarios, compared to traditional systems that use a fixed "moving average method" (which cannot handle trend changes), this system dynamically selects an algorithm: if the data has no obvious trend (such as user clicks), it uses "simple exponential smoothing"; if the data has a linear trend (such as sales during major promotions), it uses "quadratic exponential smoothing"; if the data has seasonality (such as daily sales fluctuations in fresh produce), it integrates the Prophet algorithm to support periodic adjustments, reducing the forecast error from the traditional 8% to less than 2%. For "anomaly detection" scenarios, compared to the traditional system's "static threshold judgment" (such as fixing "order quantity < 10 triggers an alert" and ignoring the differences in benchmark values at different times), the false judgment rate has been reduced from the traditional 15% to below 3%.
[0039] In summary, through a comprehensive design that addresses the pain points of existing technologies in areas such as "low latency requirements, multi-scenario adaptation, and autonomous business operation," including "real-time breakthroughs, scalability upgrades, accuracy optimization, ease of use reduction, reliability assurance, and cost control," this approach better meets the core enterprise-level needs for "real-time decision-making, rapid response, and compliance and security." Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A real-time data analysis and visualization system based on big data, characterized in that, It includes a data acquisition module, a data transmission module, a data preprocessing module, a real-time analysis module, a data storage module, and a visualization module; The data acquisition module is connected to the data transmission module, the data transmission module is connected to the data preprocessing module, the data preprocessing module is connected to the real-time architecture module, and the real-time analysis module is connected to the data storage module and the visualization module.
2. The real-time data analysis and visualization system based on big data according to claim 1, characterized in that, The data acquisition module is used to capture and access raw data from multiple heterogeneous data sources in real time to ensure data integrity and timeliness. The data transmission module is used to forward, buffer, and control the flow of the collected raw data in real time to avoid overload of downstream processing pressure.
3. The real-time data analysis and visualization system based on big data according to claim 1, characterized in that, The data preprocessing module is used to clean, transform, and integrate the raw data, solve data quality problems, and provide standardized data for the analysis module.
4. The real-time data analysis and visualization system based on big data according to claim 1, characterized in that, The real-time analysis module is used to perform real-time calculations on the preprocessed data and generate analysis results.
5. The real-time data analysis and visualization system based on big data according to claim 1, characterized in that, The data storage module is used to store raw data, preprocessed data, and analysis results respectively to meet different query needs.
6. The real-time data analysis and visualization system based on big data according to claim 1, characterized in that, The visualization module is used to present the analysis results to users in an intuitive form and provides interactive functions and application interfaces.
7. The real-time data analysis and visualization method based on big data according to any one of claims 1-6, characterized in that, Includes the following steps: S1. The data acquisition module determines the data source type according to business requirements, selects the corresponding acquisition tool, and configures the parameters of the acquisition tool. The acquisition tool encapsulates the raw data into a message and sends it to the distributed message queue, and then transmits the distributed message queue to the data transmission module. S2. The data transmission module transmits the distributed message queue to the data preprocessing module; S3, the data preprocessing module cleans, transforms, maps, desensitizes, and integrates the data, and then sends the processed data to the real-time analysis module; S4. The real-time analysis module determines the analysis type based on business needs, which is divided into statistical analysis, feature extraction, and predictive inference. It calls the aggregation algorithm for statistical analysis and the time series prediction algorithm for predictive inference to perform calculations and generate intermediate results. S5. Deduplicatize and merge intermediate results, and store them in the real-time database or time-series database of the data storage module according to the data popularity. S6, the visualization module, transforms the analysis results into intuitive visual charts for users to view and interact with.
8. The real-time data analysis and visualization method based on big data according to claim 7, characterized in that, The data cleaning in S3 specifically involves: S30. Deduplication: Based on unique identifiers or data fingerprints, duplicate data is removed using Flink's distinct() operator or Redis's Set structure. S31. Outlier handling: Outlier detection algorithms are used to identify outliers; S32. Missing value handling: Fill key fields with the mean / median, and mark non-key fields with "NULL".
9. The real-time data analysis and visualization method based on big data according to claim 7, characterized in that: The time series prediction algorithm in S4 is as follows: First smoothing value: ; Secondary smoothing value: ; Prediction formula: ; Where, t: current time (timestamp), representing the current time point being processed, T: prediction duration (step size), representing the time interval from the current time t forward, α: smoothing coefficient, with a value range of (0, 1), usually 0.1~0.3, St(1): first exponential smoothing value at time t, xt: original data value at time t, St-1(1): first exponential smoothing value at time t-1, St(2): second exponential smoothing value at time t, St-1(2): second exponential smoothing value at time t-1, ct: horizontal estimate (intercept term) at time t, bt: trend estimate (slope term) at time t. : The predicted value at time t+T.
Citation Information
Patent Citations
Real-time data analysis and visualization method and system in big data environment
CN119396997B