Efficient data visualization software system for big data processing

Abstract

The invention discloses an efficient data visualization software system oriented to big data processing, which efficiently receives original data in a big data source through a data receiving and conversion module, and converts the original data into a data array to provide convenience for subsequent processing. The data entropy calculation and data duty ratio analysis module rapidly evaluates the complexity, the information amount, the sparsity and the integrity of the data, provides support for data optimization and screening, can accurately sort and screen the data, retains a data subset with high information amount and integrity, improves the data quality and availability, reduces redundancy and noise, and improves the reliability of the data. The method comprises the following steps: optimizing data, improving analysis accuracy and efficiency, converting optimized data into various visual charts by a visual engine module, visually displaying data distribution, trend and association, allowing a user interaction interface to customize views and screen data, meeting diversified analysis requirements, and supporting interactive exploration of visual results, data entropy and duty ratio information, so as to improve the analysis accuracy and efficiency. And understanding and cognition of the user on the data are deepened.

Description

Technical Field

[0001] The present invention relates to the technical field of data visualization, and in particular to an efficient data visualization software system for big data processing. Background Art

[0002] With the rapid development of information technology, big data has become an important resource in today's era. Big data not only contains massive amounts of data, but also involves multiple data types and complex data structures. In order to effectively mine and utilize the value of big data, data visualization technology has emerged. Data visualization refers to the intuitive presentation of data information and relationships through graphics, charts, maps, etc., so that people can understand and analyze them.

[0003] However, traditional data visualization technology faces many challenges when processing big data. First, the scale and complexity of big data make data preprocessing particularly important. Raw data often needs to be cleaned, integrated and converted to ensure data quality and consistency. Second, the amount of information contained in big data is huge. How to effectively evaluate the complexity and amount of information of the data to guide data optimization and screening is a key issue in big data processing. In addition, the sparsity and completeness of big data are also important factors affecting the data visualization effect. If there are a large number of missing values ​​or invalid data in the data, the visualization results may be misleading. Summary of the invention

[0004] In view of this, the present invention proposes an efficient data visualization software system for big data processing, which can effectively solve the defects of the prior art in effectively evaluating the complexity and information volume of data, and the defects that the visualization results may be misleading.

[0005] The technical solution of the present invention is achieved in this way:

[0006] An efficient data visualization software system for big data processing, including:

[0007] The data receiving and conversion module is used to receive the original data from the big data source and convert it into a data array form;

[0008] A data entropy calculation module, connected to the data receiving and conversion module, for calculating the data entropy of each data subset in the data array and evaluating the complexity and information content of the data;

[0009] Data duty cycle analysis module, used to analyze the ratio of valid data to total data in the data array and evaluate the sparsity and integrity of the data;

[0010] The data optimization and screening module optimizes the sorting and screening of data in the data array according to the data entropy calculation results and data duty cycle analysis results, and retains the data subset with high information content and integrity;

[0011] A visualization engine module, connected to the data optimization and screening module, for converting the optimized data array into a visualization chart or graphical interface;

[0012] The user interaction interface module interacts with the visualization engine module to provide users with functions of customizing view settings, inputting data screening conditions, interactively exploring visualization results, and viewing data entropy and data duty cycle information.

[0013] As a further optional solution of the efficient data visualization software system for big data processing, the data receiving and conversion module receives the original data in the big data source and converts it into a data array form, specifically including:

[0014] Perform data cleaning on raw data from big data sources to identify and correct errors, omissions or inconsistencies in the data;

[0015] Convert the cleaned data to obtain data that meets the data array format requirements;

[0016] The converted data are arranged in a preset order and assigned corresponding indexes or identifiers to obtain a data array.

[0017] As a further optional solution of the efficient data visualization software system for big data processing, the data entropy calculation module calculates the data entropy of each data subset in the data array, specifically including:

[0018] In a data array, data is divided into different subsets based on different attributes or characteristics;

[0019] For each data subset, count the frequency of each type of event;

[0020] Calculate the probability of each event occurring based on the statistical frequency of each event;

[0021] According to the entropy formula and the probability of occurrence of various events, the data entropy of each data subset is calculated.

[0022] As a further optional solution of the efficient data visualization software system for big data processing, the specific formula for calculating the probability of occurrence of various events is:

[0023] P(x)=n(x)N;

[0024] Where n(x) represents the frequency of event x, and N represents the total number of samples in the data set;

[0025] The specific formula for calculating the data entropy of each data subset is:

[0026] H(X) = -∑P(x)logP(x);

[0027] Among them, P(x) represents the probability of event x occurring, and Σ represents the sum of all possible events x.

[0028] As a further optional solution of the efficient data visualization software system for big data processing, the data duty cycle analysis module analyzes the ratio of valid data to total data in the data array, specifically including:

[0029] Define valid data and total data, wherein the valid data is data in the data array that meets preset standards or conditions, and the total data is all data in the data array, including valid data and invalid data;

[0030] Traverse the data array and count the number of data that meet the definition of valid data;

[0031] Calculate the total number of data in the data array;

[0032] Divide the number of valid data by the total number of data to get the ratio of valid data to total data, and thus get the data duty cycle.

[0033] As a further optional solution of the efficient data visualization software system for big data processing, the data optimization and screening module optimizes the sorting and screening of the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, specifically including:

[0034] According to the calculation results of data entropy, the data with higher information content or greater uncertainty are screened out;

[0035] According to the analysis results of data duty cycle, select data sets or data subsets with a higher proportion of valid data;

[0036] Select the corresponding sorting algorithm and optimization strategy based on the data size and characteristics of the screening results;

[0037] According to the selected sorting algorithm and optimization strategy, the screening results are sorted to obtain an optimized data array.

[0038] As a further optional solution of the efficient data visualization software system for big data processing, the system also includes a data security management module for protecting the security and privacy of big data, including data encryption, access control, audit log and data backup functions.

[0039] An efficient data visualization method for big data processing, specifically including:

[0040] Data receiving and conversion step: receiving raw data from a big data source and converting the raw data into a data array form;

[0041] Data entropy calculation step: calculating the data entropy of each data subset in the data array to evaluate the complexity and information content of the data;

[0042] Data duty cycle analysis step: analyzing the ratio of valid data to total data in the data array to evaluate data sparsity and completeness;

[0043] Data optimization and screening step: optimizing and sorting the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, and retaining a data subset with high information content and integrity;

[0044] Visual chart generation step: converting the optimized data array into a visual chart or graphical interface;

[0045] User interaction step: The user interaction interface provides users with the functions of customizing view settings, inputting data filtering conditions, interactively exploring visualization results, and viewing data entropy and data duty cycle information.

[0046] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned efficient data visualization method for big data processing are implemented.

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned efficient data visualization method for big data processing.

[0048] The beneficial effects of the present invention are as follows: through the data receiving and conversion module, the system can efficiently receive the original data in the big data source and quickly convert it into the form of data array, which facilitates the subsequent data processing and analysis; the data entropy calculation module and the data duty cycle analysis module can quickly evaluate the complexity and information content of the data, as well as the sparsity and integrity of the data, and provide strong support for data optimization and screening; the data optimization and screening module can accurately optimize the sorting and screening of the data in the data array according to the analysis results of the data entropy and the data duty cycle, and retain the data subset with high information content and integrity, thereby improving the quality and availability of the data; this precise data optimization and screening method helps to reduce data redundancy and noise, and improve the accuracy and efficiency of data analysis; the visualization engine module can convert the optimized data array into various types of visualization charts or graphical interfaces, so that users can intuitively understand the distribution, trend and correlation of the data; the user interaction interface module provides a wealth of custom view settings and data screening condition input functions, so that users can customize visualization charts according to their own needs to meet different analysis needs; at the same time, users can also interactively explore the visualization results, view the data entropy and data duty cycle information, and further deepen their understanding and cognition of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 This is a schematic diagram of the composition of an efficient data visualization software system for big data processing according to the present invention;

[0051] Figure 2 A schematic diagram of a process flow of an efficient data visualization method for big data processing according to the present invention;

[0052] Figure 3 The figure is a schematic diagram of the composition of a computing device according to the present invention. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] refer to Figures 1 to 3, an efficient data visualization software system for big data processing, including:

[0055] The data receiving and conversion module is used to receive the original data from the big data source and convert it into a data array form;

[0056] A data entropy calculation module, connected to the data receiving and conversion module, for calculating the data entropy of each data subset in the data array and evaluating the complexity and information content of the data;

[0057] Data duty cycle analysis module, used to analyze the ratio of valid data to total data in the data array and evaluate the sparsity and integrity of the data;

[0058] The data optimization and screening module optimizes the sorting and screening of data in the data array according to the data entropy calculation results and data duty cycle analysis results, and retains the data subset with high information content and integrity;

[0059] A visualization engine module, connected to the data optimization and screening module, for converting the optimized data array into a visualization chart or graphical interface;

[0060] The user interaction interface module interacts with the visualization engine module to provide users with functions of customizing view settings, inputting data screening conditions, interactively exploring visualization results, and viewing data entropy and data duty cycle information.

[0061] In this embodiment, through the data receiving and conversion module, the system can efficiently receive the original data in the big data source and quickly convert it into the form of data array, which facilitates the subsequent data processing and analysis. The data entropy calculation module and the data duty cycle analysis module can quickly evaluate the complexity and information content of the data, as well as the sparsity and integrity of the data, and provide strong support for data optimization and screening. The data optimization and screening module can accurately optimize the sorting and screening of the data in the data array according to the analysis results of the data entropy and the data duty cycle, and retain the data subset with high information content and integrity, thereby improving the quality and availability of the data. This precise data optimization and screening method helps to reduce data redundancy and noise, and improve the accuracy and efficiency of data analysis. The visualization engine module can convert the optimized data array into various types of visualization charts or graphical interfaces, so that users can intuitively understand the distribution, trend and correlation of the data. The user interaction interface module provides a wealth of custom view settings and data screening condition input functions, so that users can customize visualization charts according to their needs to meet different analysis needs. At the same time, users can also interactively explore the visualization results, view data entropy and data duty cycle information, and further deepen their understanding and cognition of the data.

[0062] Preferably, the data receiving and converting module receives the original data from the big data source and converts it into a data array form, specifically including:

[0063] Perform data cleaning on raw data from big data sources to identify and correct errors, omissions or inconsistencies in the data;

[0064] Convert the cleaned data to obtain data that meets the data array format requirements;

[0065] The converted data are arranged in a preset order and assigned corresponding indexes or identifiers to obtain a data array.

[0066] In this embodiment, the data cleaning step can identify and correct errors, omissions or inconsistencies in the original data, which is a key link in data preprocessing. Through cleaning, it can be ensured that the data for subsequent analysis is accurate, complete and consistent, thereby improving the accuracy and reliability of data analysis; the cleaned data is converted to obtain data that meets the data array format requirements. This standardized processing makes the data easier to manage and operate in the subsequent processing and analysis process, and also facilitates data exchange and sharing between different systems or platforms; the converted data is arranged in a preset order and assigned corresponding indexes or identifiers to obtain a data array. This orderly data organization method makes the data more intuitive and easy to understand during visualization, analysis and mining. At the same time, the introduction of indexes or identifiers also facilitates rapid retrieval and positioning of data; the data receiving and conversion module greatly reduces the time and cost of manual intervention through automated data cleaning, conversion and arrangement processes. This efficient data processing method enables the system to respond to new data in the big data source more quickly, improving the real-time and efficiency of data processing.

[0067] Preferably, the data entropy calculation module calculates the data entropy of each data subset in the data array, specifically including:

[0068] In a data array, data is divided into different subsets based on different attributes or characteristics;

[0069] For each data subset, count the frequency of each type of event;

[0070] Calculate the probability of each event occurring based on the statistical frequency of each event;

[0071] According to the entropy formula and the probability of occurrence of various events, the data entropy of each data subset is calculated.

[0072] In this embodiment, the data entropy calculation module can accurately quantify the complexity of each data subset by dividing the data subsets and counting the frequencies of various events. The data entropy is an indicator for measuring data uncertainty and complexity. The higher the value, the richer the information content of the data subset, and also means that the distribution of the data is more uniform or difficult to predict. The module allows data to be divided into different subsets according to different attributes or characteristics, which means that the data can be analyzed from multiple dimensions. For example, the data can be divided according to characteristics such as time, location, and user behavior, and then the data entropy of each dimension is calculated, so as to have a more comprehensive understanding of the distribution and characteristics of the data. By comparing the data entropy of different data subsets, data subsets with low information content or redundancy can be identified, so as to perform targeted optimization or deletion, which helps to reduce the burden of data processing and storage and improve the efficiency of data processing. The calculation process of data entropy involves counting the frequency and probability of various events, which increases the interpretability of the data. By viewing the frequency and probability distribution of various events, the composition and characteristics of the data can be more intuitively understood, which helps to discover potential laws and patterns in the data.

[0073] Preferably, the specific formula for calculating the probability of occurrence of various events is:

[0074] P(x)=n(x)N;

[0075] Where n(x) represents the frequency of event x, and N represents the total number of samples in the data set;

[0076] The specific formula for calculating the data entropy of each data subset is:

[0077] H(X) = -∑P(x)logP(x);

[0078] Among them, P(x) represents the probability of event x occurring, and Σ represents the sum of all possible events x.

[0079] Preferably, the data duty cycle analysis module analyzes the ratio of valid data to total data in the data array, specifically including:

[0080] Define valid data and total data, wherein the valid data is data in the data array that meets preset standards or conditions, and the total data is all data in the data array, including valid data and invalid data;

[0081] Traverse the data array and count the number of data that meet the definition of valid data;

[0082] Calculate the total number of data in the data array;

[0083] Divide the number of valid data by the total number of data to get the ratio of valid data to total data, and thus get the data duty cycle.

[0084] In this embodiment, the data duty cycle analysis module can accurately evaluate the quality of the data by defining valid data and total data and calculating the ratio between them. A high data duty cycle means that there is a large amount of valid data in the data array that meets the preset standards or conditions, which helps to ensure the accuracy and reliability of subsequent data analysis and visualization; by calculating the data duty cycle, the distribution of valid data and invalid data in the data array can be intuitively understood. A low data duty cycle may indicate that the data has a sparsity problem, that is, the data contains a large amount of invalid or missing information, which helps to timely discover and deal with data sparsity and improve data integrity and availability; the analysis results of the data duty cycle can provide a basis for optimizing data storage and processing strategies. For example, for data arrays with extremely low data duty cycles, compressed storage or deletion of redundant data can be considered to reduce storage space usage and improve data processing efficiency; in the visualization process, the color, size or transparency of the visualization chart can be adjusted according to the data duty cycle to more intuitively display the distribution and characteristics of the data. At the same time, the analysis results of the data duty cycle can also help explain abnormal values ​​or outliers in the data and improve the credibility of data interpretation.

[0085] Preferably, the data optimization and screening module optimizes the sorting and screening of the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, specifically including:

[0086] According to the calculation results of data entropy, the data with higher information content or greater uncertainty are screened out;

[0087] According to the analysis results of data duty cycle, select data sets or data subsets with a higher proportion of valid data;

[0088] Select the corresponding sorting algorithm and optimization strategy based on the data size and characteristics of the screening results;

[0089] According to the selected sorting algorithm and optimization strategy, the screening results are sorted to obtain an optimized data array.

[0090] In this embodiment, the data optimization and screening module screens out data with high information content or large uncertainty through the calculation results of data entropy. These data often contain more useful information and potential rules, which help to improve the accuracy and depth of data analysis and mining, and improve the value density of the overall data by removing data with low information content or redundant data; according to the analysis results of the data duty cycle, the module can screen out data sets or data subsets with a high proportion of valid data, which helps to reduce the impact of invalid or missing data on the analysis results, improve the integrity and reliability of the data, and the optimized data array will be more representative and persuasive in the subsequent analysis and visualization process; the module selects data based on the data scale and characteristics of the screening results. Select the corresponding sorting algorithm and optimization strategy. This targeted selection helps to improve the efficiency of data processing while ensuring data quality. For example, for large-scale data sets, you can choose an efficient sorting algorithm to reduce computing time. For specific types of data, you can choose a suitable optimization strategy to reduce storage space occupancy. The optimized data array will be more readable and easier to understand during data visualization. Data sets with high information content and a high proportion of valid data can more clearly show the distribution and trend of data in charts, which helps users quickly understand data and make decisions. At the same time, the optimized data array is also easier to perform data mining and pattern recognition, which helps to discover potential rules and trends in the data.

[0091] Preferably, the system also includes a data security management module for protecting the security and privacy of big data, including functions of data encryption, access control, audit log and data backup.

[0092] In this embodiment, the data security management module ensures the confidentiality of big data during storage and transmission through the data encryption function. Even if the data is illegally obtained, its content cannot be directly read, thereby effectively preventing the leakage of sensitive information; the access control function allows the system administrator to perform fine-grained control on the access to big data according to the user's identity, role or permission level, which can prevent unauthorized users from accessing sensitive data and ensure the security and integrity of the data; the audit log function can record all operations on the data, including data reading, writing, and deletion. These log information is crucial for tracking and investigating security issues such as data leakage and illegal access, and helps to timely discover and correct potential security risks; the data backup function can copy big data to a secure storage medium regularly or on demand to prevent data loss or damage. In the event of a data disaster, data can be quickly restored through data backup to ensure business continuity and data integrity; the implementation of the data security management module can significantly improve the overall security of the system. Through the comprehensive use of functions such as data encryption, access control, audit logs and data backup, a multi-level and comprehensive data security protection system can be built to effectively resist various internal and external security threats.

[0093] An efficient data visualization method for big data processing, specifically including:

[0094] Data receiving and conversion step: receiving raw data from a big data source and converting the raw data into a data array form;

[0095] Data entropy calculation step: calculating the data entropy of each data subset in the data array to evaluate the complexity and information content of the data;

[0096] Data duty cycle analysis step: analyzing the ratio of valid data to total data in the data array to evaluate data sparsity and completeness;

[0097] Data optimization and screening step: optimizing and sorting the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, and retaining a data subset with high information content and integrity;

[0098] Visual chart generation step: converting the optimized data array into a visual chart or graphical interface;

[0099] User interaction step: The user interaction interface provides users with the functions of customizing view settings, inputting data filtering conditions, interactively exploring visualization results, and viewing data entropy and data duty cycle information.

[0100] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned efficient data visualization method for big data processing are implemented.

[0101] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned efficient data visualization method for big data processing.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient data visualization software system for big data processing, characterized in that: include: The data receiving and conversion module is used to receive the original data from the big data source and convert it into a data array form; A data entropy calculation module, connected to the data receiving and conversion module, for calculating the data entropy of each data subset in the data array and evaluating the complexity and information content of the data; Data duty cycle analysis module, used to analyze the ratio of valid data to total data in the data array and evaluate the sparsity and integrity of the data; The data optimization and screening module optimizes the sorting and screening of the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, and retains the data subset with high information content and integrity; A visualization engine module, connected to the data optimization and screening module, for converting the optimized data array into a visualization chart or graphical interface; The user interaction interface module interacts with the visualization engine module to provide users with functions of customizing view settings, inputting data screening conditions, interactively exploring visualization results, and viewing data entropy and data duty cycle information.

2. The efficient data visualization software system for big data processing according to claim 1, characterized in that: The data receiving and conversion module receives the original data from the big data source and converts it into a data array form, specifically including: Perform data cleaning on raw data from big data sources to identify and correct errors, omissions or inconsistencies in the data; Convert the cleaned data to obtain data that meets the data array format requirements; The converted data are arranged in a preset order and assigned corresponding indexes or identifiers to obtain a data array.

3. The efficient data visualization software system for big data processing according to claim 2 is characterized in that: The data entropy calculation module calculates the data entropy of each data subset in the data array, specifically including: In a data array, data is divided into different subsets based on different attributes or characteristics; For each data subset, count the frequency of each type of event; Calculate the probability of each event occurring based on the statistical frequency of each event; According to the entropy formula and the probability of occurrence of various events, the data entropy of each data subset is calculated.

4. The efficient data visualization software system for big data processing according to claim 3 is characterized in that: The specific formula for calculating the probability of occurrence of various events is: P(x)=n(x)N; Where n(x) represents the frequency of event x, and N represents the total number of samples in the data set; The specific formula for calculating the data entropy of each data subset is: H(X)=-∑P(x)logP(x) ； Among them, P(x) represents the probability of event x occurring, and Σ represents the sum of all possible events x.

5. The efficient data visualization software system for big data processing according to claim 4 is characterized in that: The data duty cycle analysis module analyzes the ratio of valid data to total data in the data array, specifically including: Define valid data and total data, wherein the valid data is data in the data array that meets preset standards or conditions, and the total data is all data in the data array, including valid data and invalid data; Traverse the data array and count the number of data that meet the definition of valid data; Calculate the total number of data in the data array; Divide the number of valid data by the total number of data to get the ratio of valid data to total data, and thus get the data duty cycle.

6. The efficient data visualization software system for big data processing according to claim 5, characterized in that: The data optimization and screening module optimizes the sorting and screening of the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, specifically including: According to the calculation results of data entropy, the data with higher information content or greater uncertainty are screened out; According to the analysis results of data duty cycle, select data sets or data subsets with a higher proportion of valid data; Select the corresponding sorting algorithm and optimization strategy based on the data size and characteristics of the screening results; According to the selected sorting algorithm and optimization strategy, the screening results are sorted to obtain an optimized data array.

7. The efficient data visualization software system for big data processing according to claim 6 is characterized in that: The system also includes a data security management module for protecting the security and privacy of big data, including functions of data encryption, access control, audit log and data backup.

8. An efficient data visualization method for big data processing, characterized in that: Specifically include: Data receiving and conversion step: receiving raw data from a big data source and converting the raw data into a data array form; Data entropy calculation step: calculating the data entropy of each data subset in the data array to evaluate the complexity and information content of the data; Data duty cycle analysis step: analyzing the ratio of valid data to total data in the data array to evaluate data sparsity and completeness; Data optimization and screening step: optimizing and sorting the data in the data array according to the data entropy calculation results and the data duty cycle analysis results, and retaining a data subset with high information content and integrity; Visual chart generation step: converting the optimized data array into a visual chart or graphical interface; User interaction step: The user interaction interface provides users with the functions of customizing view settings, inputting data filtering conditions, interactively exploring visualization results, and viewing data entropy and data duty cycle information.

9. A computing device, characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the efficient data visualization method for big data processing as claimed in claim 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the efficient data visualization method for big data processing as described in claim 8.

Images