A big data-based data visualization processing method and system

Abstract

This invention discloses a data visualization processing method and system based on big data. The method includes: constructing a feature library according to a feature data structure; extracting features from source data to obtain feature data and adding feature data classification labels; storing the feature data in the feature library; and statistically analyzing the display frequency of the feature data in real time; receiving a data visualization processing request; parsing the data visualization processing request according to the feature data structure; and determining the user name, data category, and descriptive data based on the parsing result; determining the data storage location based on the user name and data category; and obtaining visualization feature data from the feature library based on the data storage location, the feature data structure, and the descriptive data; and sending the visualization feature data to a visualization interface for display through an interface, thereby improving the performance and efficiency of large-scale data visualization processing.

Description

Technical Field

[0001] This application relates to the fields of database technology and data visualization, and more specifically, to a data visualization processing method and system based on big data. Background Technology

[0002] In the field of big data visualization, existing technologies use a field-based table approach to process and store big data. When displaying the data, this approach is used to retrieve the data from the storage space based on the fields. However, as users' requirements for the real-time nature of data visualization continue to increase, and as the amount of data grows rapidly, existing technologies require a lot of manpower to analyze and classify the data generated in business operations and to store this data using more and more fields. This leads to a significant reduction in the complexity and efficiency of data processing, while still failing to show the true meaning represented by the data.

[0003] Therefore, it is necessary to introduce a new method and system to extract data features from large-scale data, classify and store large-scale data based on the extracted features, and simultaneously count the display frequency of feature data in real time. Based on the display frequency and feature data structure, the system can quickly respond to data visualization display requests, thereby solving the technical problems of complexity and low display efficiency in existing large-scale data visualization processing. This will meet users' ever-increasing real-time requirements for data visualization display and the business needs of large-scale data visualization processing with rapidly growing data volumes, and improve the performance, accuracy and efficiency of large-scale data visualization processing. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a data visualization processing method and system based on big data. By constructing a feature library based on characteristic data structures, and then visualizing and displaying large-scale data based on these structures and the feature library, it achieves data feature extraction for large-scale data, large-scale classification and storage based on the extracted features, real-time statistics of the display frequency of feature data, and rapid response to data visualization display requests based on the display frequency and feature data structure. This solves the technical problems of complexity and low display efficiency in existing large-scale data visualization processing, meets users' continuously improving real-time requirements for data visualization display, and satisfies the business needs of large-scale data visualization processing with rapidly growing data volumes, thereby improving the performance, accuracy, and efficiency of large-scale data visualization processing.

[0005] This invention provides a data visualization processing method based on big data, which is applied in a big data visualization data processing platform. The method includes:

[0006] S101, Construct a feature library based on the feature data structure, extract features from the source data to obtain feature data, add feature data classification labels, store the feature data in the feature library, and count the display frequency of the feature data in real time; S102, Receive a data visualization processing request, parse the data visualization processing request according to the feature data structure, and determine the user name, data category, and description data based on the parsing result; S103, Determine the data storage location based on the user name and data category, and obtain the visualization feature data from the feature library based on the data storage location, the feature data structure, and the description data; S104, Publish the visualization feature data... The data is sent to a visualization interface via an interface; the feature data structure includes: feature library name, user name, data source attributes, display frequency, display threshold, data storage location, data attributes, data feature attributes, feature data classification tags, and event attributes; the data source attributes include: data source name, data source type, port number, and IP address; the data attributes include: data storage type and data block size; the data feature attributes include: static features and dynamic features; the event attributes include: event name, event window length, event start time, and event end time; the feature data classification tags are determined based on user habits and browsing history data analysis.

[0007] As described above, in step S01, before storing the feature data in the feature library, the user data needs to be classified. Specifically, the steps are as follows: Based on the data source attributes, obtain the data source name, data source type, port number, and IP address; call the data source connector based on the data source type, and connect to the data source through the port number and IP address; obtain the source data from the data source based on the user name and event attributes; classify the source data according to the data feature attributes, extract the classified source data, add feature data classification tags, and obtain the feature data; initialize the display frequency of the feature data, setting the initial value of the display frequency of the feature data to zero; wherein, the feature data includes: static feature data and dynamic feature data.

[0008] As described above, in step S01, when storing the feature data in the feature library, it is also necessary to classify and store the feature data. The specific steps are as follows: classify the feature library according to the data feature attributes to form a static feature library and a dynamic feature library, and initially allocate high-speed storage space and low-speed storage space; according to the feature data classification label, store the feature data in the low-speed storage space of the corresponding feature library; wherein, static feature data is stored in the low-speed storage space of the static feature library, and dynamic feature data is stored in the low-speed storage space of the dynamic feature library; the high-speed storage space and the low-speed storage space are allocated according to a preset ratio, the preset ratio is set according to the amount of feature data, and the size of the low-speed storage space is 1.5 times the amount of feature data.

[0009] As described above, step S101, after storing the feature data in the feature library, further includes a step of real-time statistics and dynamic storage of the feature data's display frequency. Specifically, this involves: calculating the display frequency of the feature data in real time based on the number of times the feature data is read by the data visualization processing request; incrementing the display frequency by "1" when the feature data is read once by the data visualization processing request; comparing the display frequency of the feature data with the display threshold in real time, and dynamically storing the feature data based on the comparison result; if the display frequency is greater than or equal to the display threshold, storing the feature data in high-speed storage space, deleting it from low-speed storage space, and updating the data storage location of the feature data; if the display frequency is less than the display threshold, storing the feature data in low-speed storage space, deleting it from high-speed storage space, and updating the data storage location of the feature data; wherein, the display threshold is set by the system administrator according to the needs of big data visualization display.

[0010] As described above, step S102 specifically involves: the big data visualization data processing platform receiving and parsing the data visualization processing request to obtain a parsing result, the parsing result including the user name, the feature library name, the data category, and the descriptive data; wherein, the data category corresponds one-to-one with the data feature attributes, and the descriptive data includes the data source attributes and the event attributes.

[0011] As described above, step 103 specifically involves: determining the data feature attributes corresponding to the data visualization processing request based on the user name and the data classification; determining the feature data classification label of the feature data based on the data feature attributes corresponding to the data visualization processing request, and determining the data storage location of the feature data based on the feature data classification label; determining the event name, event window length, event start time, and event end time corresponding to the data visualization processing request based on the description data; and obtaining visualization feature data from the feature library based on the feature data structure, the data storage location of the feature data, and the event name, event window length, event start time, and event end time corresponding to the data visualization processing request.

[0012] Preferably, the data visualization processing request corresponds to one or more of the user names, each user name corresponds to one or more events, and the event window length of the event is determined by the event start time and the event end time.

[0013] Event window length = Event end time - Event start time;

[0014] The event is invalid if the event window length is "0".

[0015] Preferably, the visualization feature data consists of static visualization feature data and dynamic visualization feature data, wherein the dynamic visualization feature data corresponds to the dynamic feature data;

[0016] If, after processing the data visualization processing request, another data visualization processing request identical to the previous one is received, then based on the feature data structure, according to the data storage location of the feature data, and the event name, event window length, event start time, and event end time corresponding to the data visualization processing request, new dynamic visualization feature data is obtained from the dynamic feature data in the feature library, and the visualization feature data is updated based on the new dynamic visualization feature data.

[0017] Accordingly, the present invention also provides a data visualization processing system based on big data, the system comprising a feature library construction module, a request processing module, a visualization feature data acquisition module, and a visualization feature data display module;

[0018] The feature library construction module is used to construct a feature library based on the feature data structure, extract features from the source data to obtain feature data, add feature data classification labels, store the feature data in the feature library, and count the display frequency of the feature data in real time.

[0019] The request processing module is used to receive data visualization processing requests, parse the data visualization processing requests according to the feature data structure, and determine the user name, data category and description data according to the parsing results.

[0020] The visualization feature data acquisition module is used to determine the data storage location based on the user name and the data classification, and to acquire visualization feature data from the feature library based on the data storage location, the feature data structure, and the description data.

[0021] The visualization feature data display module is used to send the visualization feature data to the visualization interface for display via an interface;

[0022] The feature data structure includes: feature library name, user name, data source attribute, display frequency, display threshold, data storage location, data attribute, data feature attribute, feature data classification label, and event attribute; the data source attribute includes: data source name, data source type, port number, and IP address; the data attribute includes: data storage type and data block size; the data feature attribute includes: static features and dynamic features; the event attribute includes: event name, event window length, event start time, and event end time; the feature data classification label is determined based on user usage habits and browsing history data analysis.

[0023] This invention, by applying the above technical solutions, realizes the construction of a feature library based on the feature data structure, and the visualization processing and display of large-scale data based on the feature data structure and the feature library. It completes the extraction of data features for large-scale data, classifies and stores large-scale data based on the extracted data features, counts the display frequency of feature data in real time, and quickly responds to data visualization display requests based on the display frequency and feature data structure. It solves the technical problems of complexity and low display efficiency of large-scale data visualization processing in the prior art, meets users' requirements for the real-time performance of data visualization display and the business needs of large-scale data visualization processing with rapidly growing data volume, and thus improves the performance, accuracy and efficiency of large-scale data visualization processing. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a data visualization processing method based on big data proposed in an embodiment of the present invention is shown.

[0026] Figure 2 A schematic diagram of the structure of a data visualization processing system based on big data proposed in an embodiment of the present invention is shown. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] This invention provides a data visualization processing method based on big data, such as... Figure 1 As shown, the method includes the following steps:

[0029] S101, construct a feature library based on the feature data structure, extract features from the source data to obtain feature data, add feature data classification labels, store the feature data in the feature library, and count the display frequency of the feature data in real time.

[0030] The feature data structure includes: feature library name, user name, data source attributes, display frequency, display threshold, data storage location, data attributes, data feature attributes, feature data classification tags, and event attributes; the data source attributes include: data source name, data source type, port number, and IP address; the data attributes include: data storage type and data block size; the data feature attributes include: static features and dynamic features; the event attributes include: event name, event window length, event start time, and event end time; the feature data classification tags are determined based on user habits and browsing history data analysis.

[0031] To better classify user data, in this embodiment, before storing the feature data in the feature database, step S01 requires further classification processing of the user data. The specific steps are as follows:

[0032] Based on the data source attributes, obtain the data source name, data source type, port number, and IP address, and call the data source connector based on the data source type to connect to the data source through the port number and IP address;

[0033] Based on the user name and the event attribute, obtain the source data from the data source;

[0034] The source data is classified according to the data feature attributes, and the classified source data is extracted, feature data classification labels are added, and the feature data is obtained.

[0035] Initialize the display frequency of the feature data by setting the initial value of the display frequency of the feature data to zero.

[0036] The feature data includes static feature data and dynamic feature data. The static feature data includes, for example, age, gender, region, occupation, etc., and the dynamic feature data includes, for example, the name of the purchased goods, the name of the street, the call duration, etc.

[0037] To efficiently store feature data, in this embodiment, step S01, when storing the feature data in the feature library, also requires classifying and storing the feature data. The specific steps are as follows:

[0038] Based on the data feature attributes, the feature library is classified to form a static feature library and a dynamic feature library, and high-speed storage space and low-speed storage space are initially allocated.

[0039] Based on the feature data classification labels, the feature data are stored in the low-speed storage space of the corresponding feature library;

[0040] Among them, static feature data is stored in the low-speed storage space of the static feature library, and dynamic feature data is stored in the low-speed storage space of the dynamic feature library.

[0041] The high-speed storage space and the low-speed storage space are allocated according to a preset ratio, which is set based on the amount of data of the feature data. The size of the low-speed storage space is 1.5 times the amount of data of the feature data.

[0042] To effectively perform dynamic storage of feature data based on its display frequency, step S101, after storing the feature data in the feature library, also includes a step of real-time statistics and dynamic storage of the feature data's display frequency, specifically:

[0043] The display frequency of the feature data is counted in real time based on the number of times the feature data is read by the data visualization processing request. When the feature data is read by the data visualization processing request once, the display frequency of the feature data is incremented by "1".

[0044] The display frequency of the feature data and the display threshold are compared in real time, and the feature data is dynamically stored according to the comparison result. If the display frequency is greater than or equal to the display threshold, the feature data is stored in the high-speed storage space, deleted from the low-speed storage space, and the data storage location of the feature data is updated. If the display frequency is less than the display threshold, the feature data is stored in the low-speed storage space, deleted from the high-speed storage space, and the data storage location of the feature data is updated.

[0045] The display threshold is set by the system administrator based on the needs of big data visualization.

[0046] S102, receive a data visualization processing request, parse the data visualization processing request according to the feature data structure, and determine the user name, data category and description data according to the parsing result.

[0047] In this embodiment, step S102 specifically includes:

[0048] The big data visualization data processing platform receives and parses the data visualization processing request to obtain the parsing result, which includes the user name, the feature library name, the data category, and the descriptive data.

[0049] in,

[0050] The data classification corresponds one-to-one with the data feature attributes, and the descriptive data includes the data source attributes and the event attributes.

[0051] S103, determine the data storage location based on the user name and the data category, and obtain visual feature data from the feature library based on the data storage location, the feature data structure, and the description data.

[0052] In this embodiment, step 103 specifically includes:

[0053] Determine the data feature attributes corresponding to the data visualization processing request based on the user name and the data category;

[0054] The feature data classification label of the feature data is determined based on the data feature attributes corresponding to the data visualization processing request, and the data storage location of the feature data is determined by the feature data classification label;

[0055] Based on the described data, determine the event name, event window length, event start time, and event end time corresponding to the data visualization processing request;

[0056] Based on the aforementioned feature data structure, and according to the data storage location of the feature data, as well as the event name, event window length, event start time, and event end time corresponding to the data visualization processing request, the visualization feature data is obtained from the feature library.

[0057] Optionally, the data visualization processing request corresponds to one or more of the user names, and each user name corresponds to one or more events. The event window length of each event is determined by the event start time and the event end time.

[0058] Event window length = Event end time - Event start time;

[0059] The event is invalid if the event window length is "0".

[0060] Alternatively, the visualization feature data consists of static visualization feature data and dynamic visualization feature data, wherein the dynamic visualization feature data corresponds to the dynamic feature data;

[0061] If, after processing the data visualization processing request, another data visualization processing request identical to the previous one is received, then based on the feature data structure, according to the data storage location of the feature data, and the event name, event window length, event start time, and event end time corresponding to the data visualization processing request, new dynamic visualization feature data is obtained from the dynamic feature data in the feature library, and the visualization feature data is updated based on the new dynamic visualization feature data.

[0062] S104, The visualized feature data is sent to the visualization interface for display via the interface.

[0063] By applying the above technical solutions, a feature library is constructed based on the feature data structure. Feature data is extracted from the source data, and feature data classification labels are added. The feature data is stored in the feature library, and the display frequency of the feature data is statistically analyzed in real time. A data visualization processing request is received, parsed according to the feature data structure, and the user name, data category, and description data are determined based on the parsing results. The data storage location is determined based on the user name and data category, and the visualization feature data is obtained from the feature library based on the data storage location, the feature data structure, and the description data. The visualization feature data is then sent to the visualization interface via an interface. This system enables the visualization and display of large-scale data by constructing a feature library based on the feature data structure and performing visualization processing and display based on the feature data structure and feature library. It completes the extraction of data features for large-scale data, classifies and stores large-scale data based on the extracted data features, counts the display frequency of feature data in real time, and responds quickly to data visualization display requests based on the display frequency and feature data structure. It solves the technical problems of complexity and low display efficiency in large-scale data visualization processing in existing technologies, meets users' requirements for continuously improving the real-time performance of data visualization display and the business needs of large-scale data visualization processing with rapidly growing data volume, thereby improving the performance, accuracy and efficiency of large-scale data visualization processing.

[0064] Corresponding to the data visualization processing method based on big data described in one embodiment of the present invention, the present invention also discloses a data visualization processing system based on big data, such as... Figure 2 As shown, the system includes a feature library construction module, a request processing module, a visual feature data acquisition module, and a visual feature data display module;

[0065] in,

[0066] The feature library construction module is used to construct a feature library based on the feature data structure, extract features from the source data to obtain feature data, add feature data classification labels, store the feature data in the feature library, and count the display frequency of the feature data in real time.

[0067] The request processing module is used to receive data visualization processing requests, parse the data visualization processing requests according to the feature data structure, and determine the user name, data category and description data according to the parsing results.

[0068] The visualization feature data acquisition module is used to determine the data storage location based on the user name and the data classification, and to acquire visualization feature data from the feature library based on the data storage location, the feature data structure, and the description data.

[0069] The visualization feature data display module is used to send the visualization feature data to the visualization interface for display via an interface;

[0070] The feature data structure includes: feature library name, user name, data source attribute, display frequency, display threshold, data storage location, data attribute, data feature attribute, feature data classification label, and event attribute;

[0071] The data source attributes include: data source name, data source type, port number, and IP address;

[0072] The data attributes include: data storage type and data block size;

[0073] The data feature attributes include: static features and dynamic features;

[0074] The event attributes include: event name, event window length, event start time, and event end time;

[0075] The feature data classification labels are determined based on data analysis of user habits and browsing history.

[0076] The various embodiments in this specification are described in a related manner. For the same or similar parts between the various embodiments, please refer to each other. Each embodiment focuses on describing the differences from other embodiments.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data visualization processing method based on big data, characterized in that, The method is applied in a big data visualization and data processing platform, and the method includes: S101, construct a feature library based on the feature data structure, extract features from the source data to obtain feature data, add feature data classification labels, store the feature data in the feature library, and count the display frequency of the feature data in real time. S102, receive a data visualization processing request, parse the data visualization processing request according to the feature data structure, and determine the user name, data category and description data according to the parsing result; S103, determine the data storage location based on the user name and the data category, and obtain visual feature data from the feature library based on the data storage location, the feature data structure, and the description data; S104, The visualized feature data is sent to the visualization interface for display via the interface; in, The feature data structure includes: feature library name, user name, data source attribute, display frequency, display threshold, data storage location, data attribute, data feature attribute, feature data classification label, and event attribute; The data source attributes include: data source name, data source type, port number, and IP address; The data attributes include: data storage type and data block size; The data feature attributes include: static features and dynamic features; The event attributes include: event name, event window length, event start time, and event end time; The feature data classification labels are determined based on user habits and browsing history data analysis.

2. The method as described in claim 1, characterized in that, In step S101, before storing the feature data in the feature database, the user data needs to be classified. The specific steps are as follows: Based on the data source attributes, obtain the data source name, data source type, port number, and IP address, and call the data source connector based on the data source type to connect to the data source through the port number and IP address; Based on the user name and the event attribute, obtain the source data from the data source; The source data is classified according to the data feature attributes, and the classified source data is extracted, feature data classification labels are added, and the feature data is obtained. Initialize the display frequency of the feature data by setting the initial value of the display frequency of the feature data to zero. The feature data includes static feature data and dynamic feature data.

3. The method as described in claim 1, characterized in that, In step S101, when storing the feature data in the feature library, it is also necessary to classify and store the feature data. The specific steps are as follows: Based on the data feature attributes, the feature library is classified to form a static feature library and a dynamic feature library, and high-speed storage space and low-speed storage space are initially allocated. Based on the feature data classification labels, the feature data are stored in the low-speed storage space of the corresponding feature library; in, Static feature data is stored in the low-speed storage space of the static feature library, and dynamic feature data is stored in the low-speed storage space of the dynamic feature library. The high-speed storage space and the low-speed storage space are allocated according to a preset ratio, which is set according to the amount of data of the feature data. The size of the low-speed storage space is 1.5 times the amount of data of the feature data.

4. The method as described in claim 1, characterized in that, Step S101, after storing the feature data in the feature library, further includes a step of real-time statistics and dynamic storage of the display frequency of the feature data, specifically: The display frequency of the feature data is calculated in real time based on the number of times the feature data is read by the data visualization processing request. When the feature data is read by the data visualization processing request once, the display frequency of the feature data is incremented by "1". The display frequency of the feature data and the display threshold are compared in real time, and the feature data is dynamically stored according to the comparison result. If the display frequency is greater than or equal to the display threshold, the feature data is stored in the high-speed storage space, deleted from the low-speed storage space, and the data storage location of the feature data is updated. If the display frequency is less than the display threshold, the feature data is stored in the low-speed storage space, deleted from the high-speed storage space, and the data storage location of the feature data is updated. The display threshold is set by the system administrator based on the needs of big data visualization.

5. The method as described in claim 1, characterized in that, Step S102 is as follows: The big data visualization data processing platform receives and parses the data visualization processing request to obtain the parsing result, which includes the user name, the feature library name, the data category, and the descriptive data. in, The data classification corresponds one-to-one with the data feature attributes, and the descriptive data includes the data source attributes and the event attributes.

6. The method as described in claim 1, characterized in that, Step S103 is as follows: Determine the data feature attributes corresponding to the data visualization processing request based on the user name and the data category; The feature data classification label of the feature data is determined based on the data feature attributes corresponding to the data visualization processing request, and the data storage location of the feature data is determined by the feature data classification label; Based on the described data, determine the event name, event window length, event start time, and event end time corresponding to the data visualization processing request; Based on the feature data structure, and according to the data storage location of the feature data, as well as the event name, event window length, event start time, and event end time corresponding to the data visualization processing request, the visualization feature data is obtained from the feature library.

7. The method as described in claim 6, characterized in that, The data visualization processing request corresponds to one or more of the user names, and each user name corresponds to one or more events. The event window length of each event is determined by the event start time and the event end time. Event window length = Event end time - Event start time; The event is invalid if the event window length is "0".

8. The method as described in claim 6, characterized in that, The visualization feature data consists of static visualization feature data and dynamic visualization feature data, and the dynamic visualization feature data corresponds to the dynamic feature data. If, after processing the data visualization processing request, another data visualization processing request identical to the previous one is received, then based on the feature data structure, according to the data storage location of the feature data, and the event name, event window length, event start time, and event end time corresponding to the data visualization processing request, new dynamic visualization feature data is obtained from the dynamic feature data in the feature library, and the visualization feature data is updated based on the new dynamic visualization feature data.

9. A system for implementing the data visualization processing method based on big data as described in claim 1, characterized in that, The system includes a feature library construction module, a request processing module, a visual feature data acquisition module, and a visual feature data display module; in, The feature library construction module is used to construct a feature library based on the feature data structure, extract features from the source data to obtain feature data, add feature data classification labels, store the feature data in the feature library, and count the display frequency of the feature data in real time. The request processing module is used to receive data visualization processing requests, parse the data visualization processing requests according to the feature data structure, and determine the user name, data category and description data according to the parsing results. The visualization feature data acquisition module is used to determine the data storage location based on the user name and the data classification, and to acquire visualization feature data from the feature library based on the data storage location, the feature data structure, and the description data. The visualization feature data display module is used to send the visualization feature data to the visualization interface for display via an interface; The feature data structure includes: feature library name, user name, data source attribute, display frequency, display threshold, data storage location, data attribute, data feature attribute, feature data classification label, and event attribute; The data source attributes include: data source name, data source type, port number, and IP address; The data attributes include: data storage type and data block size; The data feature attributes include: static features and dynamic features; The event attributes include: event name, event window length, event start time, and event end time; The feature data classification labels are determined based on user habits and browsing history data analysis.

Images