Method and system for data visualization in large data volume scenarios

Abstract

The present invention discloses a method and system for data visualization for large data volume scenarios, including: preprocessing the original data of the large data volume scenario into first data; compressing and sampling the first data into second data; feature extracting the main feature pattern of the second data as third data; selecting a visualization model, training the visualization model, taking the third data representing the main feature pattern as input, and outputting it as fourth data representing the visualization pattern; and visually displaying the fourth data. The present invention realizes efficient, accurate and intuitive visualization of large data volumes, thereby helping users better understand and analyze data and improving decision-making efficiency and effectiveness.

Description

Technical Field

[0001] The present invention relates to the field of data visualization technology, and in particular to a method and system for data visualization for large data volume scenarios. Background Art

[0002] In the current information age, the amount of data is growing exponentially, especially in the fields of Internet of Things, smart cities, financial technology, etc., the processing and analysis of big data has become a major challenge. Existing big data visualization technology mainly focuses on the display of static data, lacking effective processing and display methods for continuous and dynamic big data. Faced with a large amount of data, traditional visualization methods often cannot meet the requirements of real-time and accuracy, resulting in information overload or important information being submerged. In addition, existing technologies have deficiencies in data feature analysis and intelligent selection of the optimal visualization method, and cannot adaptively select the optimal visualization solution to display data according to the different characteristics of the data. Summary of the invention

[0003] In view of the above defects, the technical problem solved by the present invention is to provide a method, by streamlining and feature analyzing big data, the system can intelligently select the most suitable visualization method to achieve efficient, accurate and intuitive visualization of large amounts of data, thereby helping users to better understand and analyze data and improve decision-making efficiency and effectiveness. The problems of poor real-time performance, low accuracy and inability to intelligently select the optimal visualization method in existing big data visualization technologies are solved.

[0004] The first aspect of the present invention provides a data visualization method for a large data volume scenario, comprising: S101: preprocessing the original data of the large data volume scenario into first data. S102: compressing and sampling the first data into second data. S103: feature extracting the main feature pattern of the second data as third data. S104: selecting a visualization model, training the visualization model, taking the third data representing the main feature pattern as input, and outputting it as fourth data representing the visualization pattern. S105: visually displaying the fourth data;

[0005] The main characteristic pattern of step S103 includes at least one of the following characteristic patterns: time series characteristics, statistical characteristics and frequency domain characteristics;

[0006] Step S104 specifically includes the following steps:

[0007] Feature input, inputting the main feature pattern;

[0008] Machine learning model training: Use a training data set composed of historical data to train the machine learning model; the training data set should contain various data feature patterns and their corresponding optimal visualization method labels; perform feature engineering on the training data set, including using a classifier to input the feature pattern vector and determine the category features; use an embedding matrix to embed the one-hot encoding vector representing the category features into a continuous vector space to obtain the embedded continuous feature vector; train the machine learning model through the continuous feature vectors of the feature pattern corresponding to each training data set and its corresponding optimal visualization method label; the machine learning model includes: a data set feature encoder and a visualization feature encoder; the data set feature encoder inputs the continuous feature vector of the feature pattern of each training data set and outputs the data set feature encoding; the visualization feature encoder inputs the visualization label text corresponding to each training data set and outputs the visualization label feature encoding; initialize the data set feature encoder and the visualization feature encoder, use the continuous feature vector of each training data set and the visualization label corresponding to the training data set as positive samples, use the continuous feature vector of the training data set and the visualization label not corresponding to the training data set as negative samples, and construct a loss function including both the data set feature encoding and the visualization label feature encoding; optimize the loss function through training to form the trained data set feature encoder and visualization feature encoder;

[0009] In actual applications after the training is completed, feature engineering is performed on the main feature patterns to form a continuous feature vector, which is then used as the feature input of the data set feature encoder to obtain the data set feature encoding vector. The label of the alternative best visualization method is input into the trained visualization feature encoder to obtain the visualization feature encoding. According to the similarity between the data set feature encoding vector and the visualization feature encoding, the most suitable visualization method is predicted by the trained machine learning model as the predicted visualization mode, and the predicted visualization mode is output.

[0010] According to an embodiment of the present invention, the step S101 preprocesses the original data of the large data volume scenario into first data, including: cleaning, deduplication and format conversion of the original data into the first data.

[0011] According to an embodiment of the present invention, compressing and sampling the first data into second data in S102 includes: compressing the first data into second data by using a lossless compression algorithm and a lossy compression algorithm.

[0012] According to an embodiment of the present invention, the feature extraction of the second data as the third data in S103 includes: the feature extraction includes time series feature extraction, statistical feature extraction and frequency domain feature extraction.

[0013] According to an embodiment of the present invention, in S105, the fourth data is visualized and displayed in the form of a line graph, a bar graph, and a scatter graph.

[0014] According to an embodiment of the present invention, the S105 further includes: visual display of the fourth data satisfying different screening conditions. Visual display of the fourth data satisfying different viewing conditions for data details or overall trends.

[0015] The fourth data satisfies the visual display of adding data standards to the original line chart, bar chart and scatter chart.

[0016] A second aspect of the present invention provides a data visualization system for a large data volume scenario, the system comprising:

[0017] The data preprocessing module is used to preprocess the original data of the large data volume scene into first data.

[0018] The compression sampling module is used to compress and sample the data of the data preprocessing module to generate second data.

[0019] The analysis module is used for feature extraction of the second data of the compression sampling module as the third data.

[0020] The selection module is used to select a model, take the third data of the analysis module as input, perform model training, and output the fourth data. The display module is used to visually display the fourth data of the selection module.

[0021] The third aspect of the present invention provides an intelligent device, including a transmitter, a receiver, a memory and a processor; the memory is used to store computer instructions; the processor is used to run the computer instructions stored in the memory to implement the above method of data visualization for large data volume scenarios.

[0022] A fourth aspect of the present invention provides a storage medium, comprising: a readable storage medium and computer instructions, wherein the computer instructions are stored in the readable storage medium; the computer instructions are used to implement the above method for data visualization for large data volume scenarios.

[0023] The beneficial effects provided by the present invention are as follows: by streamlining and feature analyzing big data, the most suitable visualization method can be intelligently selected to achieve efficient, accurate and intuitive visualization of large amounts of data, while the interactive display of data can meet the user's usage needs in a diversified manner, laying a foundation for users to better analyze data, thereby helping users to better understand and analyze data and improve decision-making efficiency and effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] Figure 1 A flow chart of a method for data visualization for large data volume scenarios disclosed in an embodiment of the present invention;

[0026] Figure 2 A system block diagram of data visualization for large data volume scenarios disclosed in an embodiment of the present invention.

[0027] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] The first aspect of the present invention provides a method for data visualization for large data volume scenarios, such as Figure 1 As shown. Includes:

[0030] S101: Preprocessing the original data of the large data volume scene into first data.

[0031] The data preprocessing module executes step S101, which is the basic link in the entire big data visualization method. It cleans, removes duplicates and converts the format of the original data to ensure data quality. It lays the foundation for subsequent data processing and analysis. Its main functions include data cleaning, deduplication and format conversion. The specific implementation steps are as follows.

[0032] Data cleaning: Data cleaning refers to the use of a series of technical means to identify and correct or delete errors, missing values ​​and outliers in the data. Specific steps: (1) Identify missing values: Identify missing data by analyzing null values ​​or special markers in the data set. Handle missing values: You can delete records containing missing values, fill missing values ​​with statistical values ​​(such as mean, median), or use interpolation to estimate. (2) Identify outliers: Identify outliers in the data through statistical analysis or machine learning methods. Common methods include box plots, Z-scores, etc. Handle outliers: Correct or delete outliers to ensure the accuracy and consistency of the data.

[0033] Data deduplication: Data deduplication refers to identifying and deleting duplicate records in a data set to reduce redundant data. Specific steps: (1) Identify duplicate records: Define a primary key or composite key to uniquely identify each record, and use a hash algorithm or other matching algorithm to identify duplicate records. (2) Delete duplicate records: Keep the first or last record that appears and delete the rest of the duplicate records.

[0034] Format conversion: Format conversion refers to converting raw data into a unified format for subsequent processing. Specific steps: (1) Data standardization: Convert data into a unified unit and format, such as date format, numerical range, etc. (2) Data type conversion: Convert data types into types suitable for subsequent processing, such as converting string type values ​​into numeric types. (3) Data encoding conversion: Ensure that the character encoding of the data is consistent, such as converting all text data into UTF-8 encoding.

[0035] Suppose there is a large data set containing customer transaction records, where the data fields include customer ID, transaction date, transaction amount, etc. The following are the specific implementation steps: (1) Data cleaning: Identify missing values: Found that there are several missing values ​​in the transaction amount field. (2) Handle missing values: Fill these missing values ​​with the average transaction amount of the customer. (3) Identify outliers: Use the box plot method to find several records with abnormally high transaction amounts. (4) Handle outliers: Set these abnormally high transaction amounts to the maximum normal transaction amount of the customer. (5) Data deduplication: Identify duplicate records: Found several records with exactly the same customer ID and transaction date. (6) Delete duplicate records: Keep the first record in each group of duplicate records and delete the remaining duplicate records. (7) Format conversion: Data standardization: Convert all transaction dates to a unified "YYYY-MM-DD" format. (8) Data type conversion: Convert the transaction amount field from string type to floating point type. (9) Data encoding conversion: Ensure that all text fields use UTF-8 encoding.

[0036] Through the above steps, the raw data is cleaned, deduplicated and format converted, ultimately generating high-quality preprocessed data, which provides a reliable foundation for subsequent data reduction and feature analysis.

[0037] S102: compress and sample the first data into second data.

[0038] The data reduction module executes step S102, which plays a key role in the big data visualization method. It uses data compression algorithms and sampling techniques to reduce the amount of big data and reduce data redundancy. The large amount of data is compressed and sampled by various technical means, thereby reducing data redundancy and improving the efficiency of data processing and analysis. The specific implementation steps are as follows.

[0039] Data compression: Data compression refers to reducing the amount of data through algorithms while keeping its information content as unchanged as possible. Commonly used data compression algorithms include lossless compression and lossy compression. (1) Lossless compression: Lossless compression algorithms do not lose any information during the compression and decompression process, and are suitable for scenarios with high data accuracy requirements. Commonly used lossless compression algorithms include Huffman coding, LZW (Lempel-Ziv-Welch), etc. Huffman coding: By constructing a Huffman tree, codes are generated according to the frequency of character occurrence. Frequent characters use shorter codes, and low-frequency characters use longer codes, thereby achieving compression. LZW algorithm: Based on the principle of dictionary coding, repeated character strings are represented by shorter codes, thereby reducing the amount of data. (2) Lossy compression: Lossy compression algorithms will lose some information during the compression process, but can significantly reduce the amount of data, and are suitable for scenarios with low data accuracy requirements. Commonly used lossy compression algorithms include JPEG (for images), MP3 (for audio), etc.

[0040] Data sampling: Data sampling refers to extracting representative small data sets from a large data set to retain the main features of the data while reducing the amount of data. Common data sampling methods include random sampling and stratified sampling. (1) Random sampling: Randomly extract a certain proportion of data from the entire data set to ensure the randomness and representativeness of the sample. The specific method of data sampling is at least one of the following: Simple random sampling: Randomly extract samples directly from the data set. Systematic random sampling: Extract samples from the data set at fixed intervals, for example, extract one record every 10 records. (2) Stratified sampling: First stratify the data set according to a certain feature (such as category, interval), and then perform random sampling within each layer to ensure the representativeness of each layer. Specific stratification methods include: Stratification by category: For example, stratify by customer type (VIP customers, ordinary customers), and then perform random sampling in each category. Stratification by interval: For example, stratify by transaction amount interval (high, medium, low), and then perform random sampling in each interval.

[0041] Suppose there is a large data set containing millions of customer transaction records, with data fields including customer ID, transaction date, transaction amount, etc. The following are the specific implementation steps: (1) Data compression: Use Huffman coding to compress the text fields in the transaction records. First, count the frequency of each character and build a Huffman tree. Then generate codes based on the tree, representing high-frequency characters with shorter codes and low-frequency characters with longer codes. Finally, generate a compressed data file. (2) Use the LZW algorithm to compress the entire transaction record. Repeated strings are represented by shorter codes to generate a compressed data file. (3) Data sampling: Perform simple random sampling: randomly select 10% of the transaction records from the data set to generate a small data set to ensure the randomness and representativeness of the sample. Perform stratified sampling: stratify by transaction amount range (less than 100 yuan, 100-1000 yuan, and more than 1000 yuan), and then randomly select 10% of the transaction records in each range to ensure the representativeness of each range.

[0042] Through the above steps, the original data set is compressed and sampled, and the resulting streamlined data set not only reduces the data volume but also retains the main features, providing an efficient data basis for subsequent feature analysis and visualization.

[0043] S103: Feature extracting main feature patterns of the second data as third data.

[0044] Step S103 is executed by the feature analysis module to extract and analyze features of the streamlined data and identify the main feature patterns of the data.

[0045] The feature analysis module is crucial in big data visualization methods. Its main purpose is to perform in-depth feature extraction and analysis on the streamlined data, identify the main feature patterns of the data, and provide a basis for the subsequent selection of intelligent visualization methods. The specific implementation steps are as follows, including data normalization, feature extraction, and feature selection.

[0046] Data normalization: converting data to a uniform scale to eliminate the impact of dimension. The normalization methods that can be adopted include at least one of the following:

[0047] (1) Min-Max normalization: linearly transform the data into a specific range (such as [0, 1]), as shown in formula (1).

[0048] Formula (1)

[0049] Among them, X is the original data, X min and X max are the minimum and maximum values ​​of the data respectively.

[0050] (2) Z-score standardization: The data is standardized with a mean of 0 and a standard deviation of 1, as shown in formula (2).

[0051] Formula (2)

[0052] in, is the mean of the data, is the standard deviation of the data.

[0053] Feature extraction: Feature extraction is to extract features that are significant for a specific task or problem from the data to form a feature pattern for subsequent feature analysis and pattern recognition. Specific feature extraction methods include at least one of the following:

[0054] (1) Time series feature extraction: Extract features from time series data, such as trends, periodicity, seasonality, etc. Also, perform trend analysis: Use moving average or exponential smoothing to identify long-term trends in data. For example, periodicity analysis: Use FFT (Fast Fourier Transform) to extract periodic features of data. Seasonal analysis: Use STL (Seasonal-Trend Decomposition) to identify seasonal components of data.

[0055] (2) Statistical feature extraction: Calculate basic statistics of the data, such as mean, variance, skewness, kurtosis, etc.

[0056] Specifically, the mean : The average value of the data, as shown in formula (3).

[0057] Formula (3)

[0058] Where N is the number of data samples, is the ith data point.

[0059] variance : The degree of data dispersion, as shown in formula (4).

[0060] Formula (4)

[0061] Skewness : The symmetry of data distribution, as shown in formula (5).

[0062] Formula (5)

[0063] Kurtosis : The peak degree of data distribution, as shown in formula (6).

[0064] Formula (6)

[0065] (3) Frequency domain feature extraction: Perform frequency domain analysis on the data and extract frequency domain features. Use FFT to calculate the spectrum of the data and identify the main frequency components, as shown in formula (7).

[0066] Formula (7)

[0067] Among them, X(f) is the frequency domain signal, x(n) is the time domain signal, and N is the number of data points.

[0068] Feature selection: Feature selection is to select the most useful features for the task or problem from the extracted feature patterns as the main feature patterns, thereby reducing the feature dimension and improving the performance and interpretability of the model. The feature selection method adopts at least one of the following:

[0069] (1) Filter Method: Select features based on their correlation with the target variable.

[0070] (2) Pearson correlation coefficient: Calculate the linear correlation between the feature and the target variable, as shown in formula (8).

[0071] Formula (8)

[0072] Among them, X and Y are the feature and target variables respectively. and are their means respectively.

[0073] (3) Chi-square test: Test the independence of the classification feature and the target variable, as shown in formula (9).

[0074] Formula (9)

[0075] in, is the observed frequency, is the expected frequency.

[0076] (4) Embedded Method: Select features during model training.

[0077] (5) L1 regularization (Lasso regression): By penalizing unimportant features, its coefficients tend to zero, as shown in formula (10).

[0078] Formula (10)

[0079] in, is the target variable, Characterized by is the characteristic coefficient, is the regularization parameter.

[0080] Suppose we have a streamlined data set containing customer transaction records, including fields such as customer ID, transaction date, transaction amount, and transaction type. The following are the specific implementation steps: (1) Data normalization: Perform Min-Max normalization on the transaction amount data and convert the data to the range of [0,1]. Assuming that the minimum transaction amount is 10 yuan and the maximum is 1,000 yuan, the normalized transaction amount calculation formula is as shown in formula (11):

[0081] Formula (11)

[0082] (2) Feature extraction: Time series feature extraction: Perform trend analysis on transaction dates, use the moving average method to calculate the average transaction amount per month, and identify long-term trends. (3) Use FFT to perform periodic analysis on transaction amounts and identify the main periodic components as the main feature patterns.

[0083] S104: Select a visualization model, perform visualization model training, use the third data representing the main feature pattern as input, and output as fourth data representing the visualization pattern.

[0084] Step S104 is performed by the intelligent selection module: based on the feature analysis result, a machine learning algorithm is used to intelligently select a visualization method that best suits the main feature pattern of the current data.

[0085] The intelligent selection module is the core part of the big data visualization method. Its main task is to use machine learning algorithms to intelligently select the visualization method that best suits the current data features based on the main feature patterns of the data extracted and analyzed by the feature analysis module. The specific implementation steps are as follows, including feature input, machine learning model training, and visualization method prediction.

[0086] Feature input: The main feature patterns extracted by the feature analysis module are input into the intelligent selection module. These features may include but are not limited to: time series features (such as trend, periodicity, seasonality), statistical features (such as mean, variance, skewness, kurtosis) and frequency domain features (such as main frequency components).

[0087] Machine learning model training: Use a training dataset consisting of historical data and the best known visualization method to train the machine learning model. The training dataset should contain various data features and their corresponding labels for the best visualization method. Specifically, it includes:

[0088] (1) Dataset preparation: Collect and organize training datasets with different feature patterns, and annotate the best visualization method for each training dataset. For example, time series data can be annotated as a line chart, and categorical data can be annotated as a bar chart, etc.

[0089] (2) Feature engineering: Perform feature engineering on the training data set to ensure the quality and diversity of the feature patterns. Specifically, first perform data preprocessing, data reduction (compression or sampling), and feature extraction on each training data set in the same way as above to obtain the feature pattern of each training data set. The feature pattern includes the time series feature T, statistical feature S, and frequency domain feature F of each training data set. Encode the extracted feature patterns to form a feature pattern vector X:

[0090] Formula (12)

[0091] Use the classifier to input the feature pattern vector and determine the category features:

[0092] Formula (13)

[0093] function represents the classifier function, Represents the category feature, which is encoded in one-hot format. That is, if the category feature There are K possible values, which represents the one-hot encoding vector of the category feature for:

[0094] Formula (14)

[0095] Among them and category characteristics The corresponding one-hot encoding vector The kth element of The value is 1, and the rest of the elements are 0. For the one-hot encoding vector representing the category feature , using the embedding matrix Embed it into a continuous vector space to obtain the embedded continuous feature vector :

[0096] Formula (15)

[0097] The dimension of this continuous feature vector is D.

[0098] (3) Model training: The machine learning model is trained through the continuous feature vectors of the feature patterns corresponding to each training data set and their corresponding optimal visualization method labels.

[0099] The machine learning model includes: a data set feature encoder and a visualization feature encoder. The data set feature encoder inputs a continuous feature vector of a feature pattern of each training data set and outputs a data set feature code; the visualization feature encoder inputs a visualization label text corresponding to each training data set and outputs a visualization label feature code.

[0100] The dataset feature encoder adopts the ResNet network structure, which can be expressed as:

[0101] Formula (16)

[0102] in is the continuous feature vector after embedding the feature pattern of each training data set, The feature encoding vector of the dataset formed by this encoder.

[0103] The visual feature encoder adopts a Transformer structure and can be expressed as:

[0104] Formula (17)

[0105] in is the label of the best visualization method corresponding to each training dataset, Visualize the label feature encoding output by the encoder.

[0106] During the training process, the dataset feature encoder and the visualization feature encoder are initialized, the continuous feature vectors of each training dataset and the visualization label corresponding to the training dataset are used as positive samples, the continuous feature vectors of the training dataset and the visualization label not corresponding to the training dataset are used as negative samples, and the loss function including both dataset feature encoding and visualization label feature encoding is constructed. as follows:

[0107] Formula (18)

[0108] in is the total number of training data sets used for training samples, is the continuous feature vector of the training data set of the i-th positive sample, is the visualization label corresponding to the positive sample, is the visual label of the negative sample that does not correspond to the i-th positive sample, the function Represents the similarity function between two feature codes. By training and optimizing the loss function, a trained dataset feature encoder and a visualization feature encoder are formed.

[0109] In actual applications after the training is completed, the intelligent selection module receives the third data representing the main feature pattern, performs feature engineering enhancement on the main feature pattern to form a continuous feature vector, which is then used as the feature input of the data set feature encoder to obtain the data set feature encoding vector, and the alternative best visualization method label is input into the trained visualization feature encoder to obtain the visualization feature encoding. According to the similarity between the data set feature encoding vector and the visualization feature encoding, the most suitable visualization method is predicted by the trained machine learning model as the predicted visualization mode, and the predicted visualization mode is output for subsequent data visualization display.

[0110] Suppose we have a customer transaction data set that has been feature analyzed, where the extracted features include the trend of monthly transaction amount, the mean and variance of transaction amount, etc. The specific implementation steps are as follows: (1) Input the main feature patterns extracted by the feature analysis module, such as the trend of monthly transaction amount, the mean and variance of transaction amount, etc., into the intelligent selection module. (2) Perform feature engineering on the main feature patterns to ensure the quality of the input continuous feature vector. (3) Visualization method prediction: Based on the input continuous feature vector, predict the most suitable visualization pattern. For example, the model may predict that a line chart is most suitable for displaying the trend of monthly transaction amount. (4) Visualization method output: Output a visualization label (such as a line chart label) representing the predicted visualization pattern for subsequent data visualization.

[0111] Through the above steps, the intelligent selection module can intelligently select the most suitable visualization method according to the data characteristics to ensure efficient visualization of large amounts of data.

[0112] S105: Select a visualization script according to the fourth data representing the visualization mode, and perform visualization display.

[0113] The visualization display module executes step S105, which selects the corresponding visualization script according to the visualization mode to display the data, supports multiple visualization forms, such as line charts, bar charts, scatter plots, etc., and provides interactive functions to facilitate users to conduct in-depth analysis.

[0114] The visualization module is the display part of the big data visualization method. Its main task is to display the data using the visualization method selected by the intelligent selection module. This module supports a variety of visualization forms, such as line charts, bar charts, scatter plots, etc., and provides rich interactive functions to facilitate users to conduct in-depth analysis.

[0115] Implementation of interactive functions: In order to facilitate users to conduct in-depth analysis, the visualization module also needs to provide rich interactive functions. Common interactive functions include: (1) Data filtering: Users can filter data by selecting specific conditions to view the visualization results within a specific data range. (2) Data zooming: Users can zoom in or out of the chart to view the details or overall trends of the data. (3) Data annotation: Users can add data annotations to the chart to facilitate the explanation of data characteristics and trends.

[0116] Suppose we have a customer transaction data set recommended by the intelligent selection module, and recommend the use of line charts for visualization. The following are the specific implementation steps: (1) Selection of visualization method: Based on the output of the intelligent selection module, select line chart as the visualization method. (2) Data preprocessing: Preprocess the customer transaction data. Ensure the continuity and consistency of timestamps, and convert the data format to a format suitable for drawing line charts. (3) Generate visualization charts: Line chart: Set the horizontal axis to time (such as month) and the vertical axis to transaction amount. Draw the transaction amount data points for each month and connect them to form a line chart. Different customers or transaction types can be distinguished by different colors or styles. (4) Interactive function implementation, including: providing filtering conditions to allow users to select transaction data for specific customers or within a specific time range; implementing chart zooming function, allowing users to zoom in or out of the chart using the mouse wheel or touch gestures to view the details or overall trends of the data; data annotation: allowing users to add annotations to specific data points, such as annotating the transaction peak of a certain month, to facilitate the interpretation of data characteristics and trends.

[0117] Through the above steps, users can use the selected visualization method to display large amounts of data and conduct in-depth analysis through rich interactive functions to better understand and utilize the data.

[0118] A second aspect of the present invention provides a data visualization system for large data volume scenarios, such as Figure 2 As shown, the system comprises:

[0119] The data preprocessing module 201 is used to preprocess the original data of the large data volume scene into first data.

[0120] The compression sampling module 202 is used to compress and sample the data of the data preprocessing module to generate second data.

[0121] The analysis module 203 is used for feature extraction of the second data of the compression sampling module as third data.

[0122] The selection module 204 is used to select a model, take the third data of the analysis module as input, perform model training, and output the fourth data.

[0123] The display module 205 is used to visually display the fourth data of the selection module.

[0124] The third aspect of the present invention provides an intelligent device, including a transmitter, a receiver, a memory and a processor; the memory is used to store computer instructions; the processor is used to run the computer instructions stored in the memory to implement the above method of data visualization for large data volume scenarios.

[0125] A fourth aspect of the present invention provides a storage medium, comprising: a readable storage medium and computer instructions, wherein the computer instructions are stored in the readable storage medium; the computer instructions are used to implement the above method for data visualization for large data volume scenarios.

[0126] The beneficial effects provided by the present invention are as follows: by streamlining and feature analyzing big data, the most suitable visualization method can be intelligently selected to achieve efficient, accurate and intuitive visualization of large amounts of data, while the interactive display of data can meet the user's usage needs in a diversified manner, laying a foundation for users to better analyze data, thereby helping users to better understand and analyze data and improve decision-making efficiency and effectiveness.

[0127] Obviously, the above specific implementation cases are only examples for illustrating the application of the present method, rather than limiting the implementation methods. For those skilled in the art, other different forms of changes and modifications can be made on the basis of the above description to study other related issues. Therefore, the protection scope of the present invention should be the protection scope of the claims.

[0128] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0129] The electronic device and other embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0132] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0133] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for data visualization for large data volume scenarios, characterized in that: The method comprises: S101: preprocessing the original data of the large data volume scenario into first data; the original data of the large data volume scenario includes a large data set of customer transaction records, and the data fields of the large data set of customer transaction records include customer ID, transaction date and transaction amount; S102: compressing and sampling the first data into second data; S103: feature extracting a main feature pattern of the second data as third data; S104: selecting a visualization model, and after training the visualization model, taking the third data representing the main feature pattern as input, and outputting fourth data representing the visualization pattern; S105: Visually displaying the fourth data; The main characteristic pattern of step S103 includes at least one of the following characteristic patterns: time series characteristics, statistical characteristics and frequency domain characteristics; Step S104 specifically includes the following steps: inputting the main feature pattern; The machine learning model is trained using a training data set composed of historical data; the training data set should contain various data feature patterns and their corresponding optimal visualization method labels; the training data set is enhanced by feature engineering, including using a classifier to input the feature pattern vector to determine the category feature; the one-hot encoding vector representing the category feature is embedded into a continuous vector space using an embedding matrix to obtain the embedded continuous feature vector; the machine learning model is trained using the continuous feature vector of the feature pattern corresponding to each training data set and its corresponding optimal visualization method label; the machine learning model includes: a data set feature encoder and a visualization feature encoder; the data set feature encoder inputs the continuous feature vector of the feature pattern of each training data set and outputs the data set feature encoding; the visualization feature encoder inputs the visualization label text corresponding to each training data set and outputs the visualization label feature encoding; the data set feature encoder and the visualization feature encoder are initialized, the continuous feature vector of each training data set and the visualization label corresponding to the training data set are used as positive samples, the continuous feature vector of the training data set and the visualization label not corresponding to the training data set are used as negative samples, and a loss function including both the data set feature encoding and the visualization label feature encoding is constructed; the loss function is optimized by training to form a trained data set feature encoder and a visualization feature encoder; Perform feature engineering enhancement on the main feature patterns to form a continuous feature vector, which is then used as the feature input of the dataset feature encoder to obtain the dataset feature encoding vector. Input the best alternative visualization method label into the trained visualization feature encoder to obtain the visualization feature encoding. According to the similarity between the dataset feature encoding vector and the visualization feature encoding, the most suitable visualization method is predicted by the trained machine learning model as the predicted visualization mode, and then the predicted visualization mode is output. The step S101 preprocesses the original data of the large data volume scenario into first data, including: cleaning, deduplication and format conversion of the original data into first data; The feature extraction of the second data into third data in S103 includes: The feature extraction includes time series feature extraction, statistical feature extraction and frequency domain feature extraction; In S105, the fourth data is visualized and displayed in the form of a line graph, a bar graph, and a scatter graph; The S105 further includes: The fourth data satisfies different conditions for screening and is displayed visually; The fourth data satisfies the visual display of different viewing data details or overall trends; The fourth data satisfies the visual display of adding data standards to the original line chart, bar chart and scatter chart.

2. The method according to claim 1, characterized in that The step of compressing and sampling the first data into second data in S102 includes: The first data is compressed into second data by using a lossless compression algorithm and a lossy compression algorithm.

3. A system for data visualization in large data volume scenarios, characterized in that: The system comprises: A data preprocessing module, used for preprocessing the raw data of the large data volume scenario into first data; the raw data of the large data volume scenario includes a large data set of customer transaction records, and the data fields of the large data set of customer transaction records include customer ID, transaction date and transaction amount; A compression sampling module, used for compressing and sampling the data of the data preprocessing module into second data; An analysis module, used for feature extraction of the second data of the compression sampling module as third data; A selection module, used for selecting a model, taking the third data of the analysis module as input, performing model training, and outputting fourth data; A display module, used for visually displaying the fourth data of the selection module; The data preprocessing module preprocesses the original data of the large data volume scenario into first data, including: cleaning, deduplication and format conversion of the original data into first data; The feature extraction of the second data into third data in the analysis module includes: The feature extraction includes time series feature extraction, statistical feature extraction and frequency domain feature extraction; The display module visualizes the fourth data and displays the fourth data in a line graph, a bar graph and a scatter graph; The display module also includes: The fourth data satisfies different conditions for screening and is displayed visually; The fourth data satisfies the visual display of different viewing data details or overall trends; The fourth data satisfies the visual display of adding data standards to the original line chart, bar chart and scatter chart.

4. A smart device, characterized in that: include: transmitter, receiver, memory and processor; The memory is used to store computer instructions; The processor is used to execute the computer instructions stored in the memory to implement the method for data visualization for large data volume scenarios as described in claim 1 or 2.

5. A storage medium, characterized in that: include: A readable storage medium and computer instructions, wherein the computer instructions are stored in the readable storage medium; The computer instructions are used to implement the method for data visualization for large data volume scenarios as described in claim 1 or 2.

Images