Multi-dimensional data analysis system based on big data

Abstract

The invention discloses a multidimensional data analysis system based on big data. The multidimensional data analysis system comprises a metadata unified management module, a data full-life-cycle management engine module, a multidimensional data analysis module, an intelligent recommendation module, a security permission module and a visual display module. According to the method, a complete and systematic solution is formed through a plurality of key modules including metadata management, data full-life-cycle management, multi-dimensional data analysis, intelligent recommendation, security authority management, visual display and the like, all links of big data analysis can be covered, the comprehensiveness and efficiency of data processing are improved, and the big data analysis efficiency is improved. Through the multi-dimensional data analysis module and the intelligent recommendation module, trends, modes and association rules in data can be deeply mined, more accurate and comprehensive support is provided for business decision making, and meanwhile, the system can cope with a rapidly changing data environment through a real-time data stream processing function.

Description

Technical Field

[0001] The present invention belongs to the technical field of multidimensional data analysis, and specifically refers to a multidimensional data analysis system based on big data. Background Art

[0002] With the rapid development of big data technology, the amount of data faced by enterprises is exploding. How to efficiently and accurately manage and utilize this data has become an urgent problem to be solved. Metadata, as the "data" of data, is crucial for the understanding, organization, retrieval, sharing and protection of data. However, traditional metadata management methods often have problems such as decentralized management, delayed updates, and difficulty in unified management across the life cycle, which limits the full mining of the value of big data.

[0003] In addition, the existing multidimensional data analysis still has certain defects. The existing multidimensional data analysis is composed of multiple independent tools and modules, lacking a unified platform to manage the entire life cycle of data, and there is no dedicated metadata management module, or the metadata management function is weak, which makes it difficult to meet complex business needs. The analysis algorithms and models are relatively simple and cannot deeply explore the complex trends and patterns in the data. Therefore, it is necessary to propose a multidimensional data analysis system based on big data. Summary of the invention

[0004] The purpose of the present invention is to provide a multidimensional data analysis system based on big data to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a multidimensional data analysis system based on big data, comprising a metadata unified management module, a data full life cycle management engine module, a multidimensional data analysis module, an intelligent recommendation module, a security authority module and a visual display module; The metadata unified management module is used to collect, store and manage all metadata information related to data, including business metadata, technical metadata and management metadata; The data lifecycle management engine module is used to manage the entire process of data from creation to destruction, including data collection, cleaning, conversion, storage, processing, analysis, presentation, use, and archiving stages; The multidimensional data analysis module receives data from the data lifecycle management engine module, and deeply mines the trends and patterns of the data from different angles to perform multidimensional data analysis; The intelligent recommendation module recommends associated data sets and reports based on metadata and analysis results; The security permission module is used to implement data access control, encryption, auditing and monitoring measures to protect data from unauthorized access, tampering or disclosure; The visualization display module is used to convert the results of data analysis into a visualization view.

[0006] Furthermore, the metadata unified management module designs metadata entities, attributes and relationships according to business needs and data characteristics, formulates metadata naming specifications, format standards and storage rules, uses ETL to extract metadata from different data sources, captures metadata generated during runtime through log analysis, and writes code according to system design documents to implement various functions of the metadata unified management module, including metadata collection, storage, retrieval, update and deletion.

[0007] Furthermore, the metadata unified management module integrates the collected metadata according to predetermined rules to form a unified metadata center, performs functional and performance tests on the metadata unified management module, optimizes and adjusts the system according to the test results, and records the history of each metadata change to support rollback to the previous version.

[0008] Furthermore, the data lifecycle management engine module collects data in the metadata unified management module through ELT, imports the data into the temporary storage area, performs a preliminary quality check on the collected data, identifies missing values, outliers and inconsistent data, and performs preprocessing operations; Convert the preprocessed data into a unified format and standard, integrate multiple data sets after format conversion to generate a more comprehensive data view, and store the data in the database.

[0009] Furthermore, the multidimensional data analysis module uses the data of the data lifecycle management engine module to deeply explore the trends and patterns of the data from different angles and perform multidimensional data analysis; Based on the cleaned, converted and stored data obtained from the metadata lifecycle management engine module, an adaptive multi-dimensional algorithm is performed. Assuming the evaluation function is F(D, Y), the implementation formula is: , In the formula, represents the best dimension combination selected, D represents the candidate dimension set, and Y represents the target variable; According to the query frequency analysis, based on the collection of historical query logs, the number of occurrences of each query Q at different time points t is recorded, and the implementation formula is: , In the formula, P(Q) represents the pre-calculated probability of query Q, T represents the time window length, represents the time weight, represents the frequency of query Q at time t.

[0010] Furthermore, the multidimensional data analysis module mines data associations based on the data sets in the statistical database. Suppose the support of items A and B is And conditional probability P(B|A), the implementation formula is: , In the formula, represents association rules, P(B|A) represents conditional probability, represents the support of A and B, Indicates support threshold; The multidimensional data analysis module performs real-time data stream processing in the following manner: , each data stream defines a corresponding processing function , apply a processing function to each data stream , get the processed results, summarize all the processed results, and get the final real-time analysis result R(t). The implementation formula is: , In the formula, R(t) represents the real-time analysis result, Represents the processing function, Represents a real-time data stream.

[0011] Furthermore, the intelligent recommendation module obtains detailed information of the data set from the metadata unified management module, obtains the latest analysis results from the multidimensional data analysis module, and extracts data features to extract the feature vector of user u. and the feature vector of data set i , the implementation formula is: , In the formula, represents the predicted score of user u for data set i, represents the feature vector of user t, Represents the feature vector of data set i.

[0012] Furthermore, the intelligent recommendation module includes: assuming that the original data is f(x), generating the normally distributed noise is , add noise to the original data to get the noise-added result M(x). The differential privacy protection implementation formula is: , In the formula, M(x) represents the result after adding noise, and f(x) represents the original data. represents the normally distributed noise, by adjusting The size controls the degree of privacy protection.

[0013] Furthermore, the security permission module formulates detailed security policies and specifications based on security goals, including user access rights, data encryption standards, and audit requirements, adopts a strong password policy to verify the authenticity of user identities, assigns appropriate access rights to each user or user group based on a role-based permission allocation model, encrypts data transmission through the SSL / TLS protocol, encrypts sensitive data stored in the database, generates and distributes encryption keys, and monitors abnormal patterns in real time.

[0014] Furthermore, the visualization display module converts and aggregates the data according to visualization requirements based on the data analyzed by the multidimensional data analysis module, selects a suitable chart type according to the data type and analysis target, and maps the processed data to a visualization view for viewing.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention forms a complete and systematic solution through multiple key modules, including metadata management, data life cycle management, multi-dimensional data analysis, intelligent recommendation, security authority management and visual display, which can cover all aspects of big data analysis and improve the comprehensiveness and efficiency of data processing; 2. The present invention uses a unified metadata management module. The system can flexibly design metadata models according to business needs and data characteristics, support the collection and integration of different data sources, and provide a solid foundation for subsequent data analysis and processing. At the same time, each module is relatively independent and easy to maintain and expand. 3. The present invention uses advanced algorithms and models through the multidimensional data analysis module and the intelligent recommendation module to deeply explore trends, patterns and association rules in the data, providing more accurate and comprehensive support for business decision-making. At the same time, the real-time data stream processing function enables the system to cope with the rapidly changing data environment; 4. The present invention converts complex data analysis results into intuitive charts and views through a visual display module, so that users with non-technical backgrounds can easily understand and apply the data analysis results, thereby improving the readability and usability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the architecture of the multidimensional data analysis system based on big data of the present invention; Figure 2 This is a flowchart for implementing the metadata unified management module of the multidimensional data analysis system based on big data of the present invention; Figure 3 This is a flow chart for implementing a data lifecycle management engine module of a multidimensional data analysis system based on big data of the present invention; Figure 4This is a flow chart for implementing the multidimensional data analysis module of the multidimensional data analysis system based on big data of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1-Figure 4 ,The present invention provides a technical solution: including a metadata unified management module, a data full life cycle management engine module, a multidimensional data analysis module, an intelligent recommendation module, a security permission module and a visual display module; The metadata unified management module is used to collect, store and manage all metadata information related to data, including business metadata, technical metadata and management metadata; The data lifecycle management engine module is used to manage the entire process of data from creation to destruction, including data collection, cleaning, conversion, storage, processing, analysis, presentation, use, and archiving stages; The multidimensional data analysis module obtains data from the data lifecycle management engine module, deeply mines data trends and patterns from different angles, and performs multidimensional data analysis; The intelligent recommendation module recommends associated data sets and reports based on metadata and analysis results; The security permission module is used to implement strict data access control, encryption, auditing and monitoring measures to protect data from unauthorized access, tampering or leakage; The visualization display module is used to convert the results of data analysis into a visualization view.

[0019] Among them, the metadata unified management module designs a suitable metadata model according to business needs and data characteristics, including metadata entities, attributes, and relationships, formulates metadata naming specifications, format standards, and storage rules, extracts metadata from different data sources through ETL, captures metadata generated during runtime through log analysis and other methods, and writes code according to system design documents to implement various functions of the metadata unified management module, including metadata collection, storage, retrieval, update, and deletion.

[0020] Among them, according to the identification and determination of the sources of metadata that need to be collected, including internal systems and external data sources, a collection strategy is formulated according to the characteristics of the data source and metadata requirements, the collected metadata is integrated according to predetermined rules to form a unified metadata center, and the function and performance of the unified metadata management module are tested. According to the test results, the system is optimized and adjusted, and the history of each metadata change is recorded to support rollback to the previous version.

[0021] The data lifecycle management engine module is used for the management of the entire process from data creation to destruction, including data collection, cleaning, conversion, storage, processing, analysis, presentation, use, and archiving. It collects data through ELT according to the data of the metadata unified management module, imports the data into the temporary storage area, performs preliminary quality checks on the collected data, identifies missing values, outliers, and inconsistent data, and performs preprocessing operations. The preprocessed data is converted into a unified format and standard, multiple data sets after format conversion are integrated together to generate a more comprehensive data view, and the data is stored in the database.

[0022] The multidimensional data analysis module uses the data from the data lifecycle management engine module to deeply explore the trends and patterns of data from different angles and perform multidimensional data analysis; Based on the cleaned, converted and stored data obtained from the metadata lifecycle management engine module, an adaptive multi-dimensional algorithm is performed. Assuming the evaluation function is F(D, Y), the implementation formula is: , In the formula, represents the best dimension combination selected, D represents the candidate dimension set, and Y represents the target variable; According to the query frequency analysis, based on the collection of historical query logs, the number of occurrences of each query Q at different time points t is recorded, and the implementation formula is: , In the formula, P(Q) represents the pre-calculated probability of query Q, T represents the time window length, represents the time weight, represents the frequency of query Q at time t.

[0023] Among them, according to the data set in the statistical database, the data association is mined, and the support of items A and B is And conditional probability P(B|A), the implementation formula is: , In the formula, represents association rules, P(B|A) represents conditional probability, represents the support of A and B, Indicates support threshold; According to the data and real-time data stream processing, set up multiple data streams , each data stream defines a corresponding processing function , apply a processing function to each data stream , get the processed results, summarize all the processed results, and get the final real-time analysis result R(t). The implementation formula is: , In the formula, R(t) represents the real-time analysis result, Represents the processing function, Represents a real-time data stream.

[0024] The intelligent recommendation module recommends related data sets and reports based on metadata and analysis results, obtains detailed information of data sets from the metadata unified management module, obtains the latest analysis results from the multidimensional data analysis module, and extracts data features to extract the feature vector of user u. and the feature vector of data set i , the implementation formula is: , In the formula, represents the predicted score of user u for data set i, represents the feature vector of user t, Represents the feature vector of data set i.

[0025] The intelligent recommendation module assumes that the original data is f(x) and generates a normally distributed noise as , add noise to the original data to get the noise-added result M(x). The differential privacy protection implementation formula is: , In the formula, M(x) represents the result after adding noise, and f(x) represents the original data. represents the normally distributed noise, by adjusting The size controls the degree of privacy protection.

[0026] Among them, the security permission module is used to implement strict data access control, encryption, auditing and monitoring measures to protect data from unauthorized access, tampering or leakage; according to security goals, detailed security policies and specifications are formulated, including user access rights, data encryption standards, and audit requirements. Strong password policies are used to verify the authenticity of user identities. Based on the role-based permission allocation model, appropriate access rights are assigned to each user or user group. Data transmission is encrypted through the SSL / TLS protocol, sensitive data stored in the database is encrypted, encryption keys are generated and distributed, and abnormal patterns are monitored in real time.

[0027] Among them, the visualization display module is used to convert the results of data analysis into a visual view. According to the data analyzed by the multidimensional data analysis module, the necessary conversion and aggregation of the data are performed according to the visualization requirements. According to the data type and analysis target, the appropriate chart type is selected, and the processed data is mapped to the visual view for viewing.

[0028] In this example, specifically: the multidimensional data analysis module uses the data of the data lifecycle management engine module to deeply explore the trends and patterns of the data from different angles and perform multidimensional data analysis; Based on the cleaned, converted and stored data obtained from the metadata lifecycle management engine module, an adaptive multi-dimensional algorithm is performed. Assuming the evaluation function is F(D, Y), the implementation formula is: , In the formula, represents the best dimension combination selected, D represents the candidate dimension set, and Y represents the target variable; According to the query frequency analysis, based on the collection of historical query logs, the number of occurrences of each query Q at different time points t is recorded, and the implementation formula is: , In the formula, P(Q) represents the pre-calculated probability of query Q, T represents the time window length, represents the time weight, represents the frequency of query Q at time t.

[0029] Among them, according to the data set in the statistical database, the data association is mined, and the support of items A and B is And conditional probability P(B|A), the implementation formula is: , In the formula, represents association rules, P(B|A) represents conditional probability, represents the support of A and B, Indicates support threshold; According to the data and real-time data stream processing, set up multiple data streams , each data stream defines a corresponding processing function , apply a processing function to each data stream , get the processed results, summarize all the processed results, and get the final real-time analysis result R(t). The implementation formula is: , In the formula, R(t) represents the real-time analysis result, Represents the processing function, Represents a real-time data stream.

[0030] In this example, specifically: the intelligent recommendation module recommends related data sets and reports based on metadata and analysis results, obtains detailed information of the data set from the metadata unified management module, obtains the latest analysis results from the multidimensional data analysis module, and performs data feature extraction to extract the feature vector of user u. and the feature vector of data set i , the implementation formula is: , In the formula, represents the predicted score of user u for data set i, represents the feature vector of user t, Represents the feature vector of data set i.

[0031] The intelligent recommendation module assumes that the original data is f(x) and generates a normally distributed noise as , add noise to the original data to get the noise-added result M(x). The differential privacy protection implementation formula is: , In the formula, M(x) represents the result after adding noise, and f(x) represents the original data. represents the normally distributed noise, by adjusting The size controls the degree of privacy protection.

[0032] The working principle of the present invention is as follows: through the unified metadata management module, all metadata information related to data is collected, stored and managed, including business metadata, technical metadata and management metadata, and metadata models are designed, including entities, attributes, relationships, etc., and naming specifications, format standards and storage rules are formulated. Metadata is extracted from different data sources through the ETL process, and metadata generated during runtime is captured through log analysis and other methods. The functions of metadata collection, storage, retrieval, update and deletion are also realized; The present invention manages the entire process of data from creation to destruction through a data lifecycle management engine module, including data collection, cleaning, conversion, storage, processing, analysis, presentation, use and archiving stages. Data is collected through ELT and a preliminary quality check is performed. The data is converted into a unified format and standard, multiple data sets are integrated to generate a comprehensive data view, and finally stored in a database. The best dimension combination is selected for data analysis through an adaptive multi-dimensional algorithm. At the same time, statistical methods and real-time data stream processing technology are used to perform association mining and real-time analysis on the data. The evaluation function is used to select the best dimension combination, while query frequency analysis helps optimize query performance. Through data feature extraction, the predicted score between the user and the data set is calculated to achieve personalized recommendation. At the same time, a differential privacy protection mechanism is introduced to add noise to the original data to protect user privacy. The security permission module formulates detailed security policies and specifications, including user access rights, data encryption standards and audit requirements, adopts a strong password policy to verify user identity, and allocates access rights based on a role-based permission allocation model. Data transmission is encrypted through the SSL / TLS protocol, sensitive data is encrypted and stored, and abnormal patterns are monitored in real time. According to the results of the multidimensional data analysis module, the appropriate chart type is selected, and after necessary transformation and aggregation of the data, it is mapped to a visual view for users to view.

[0033] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0034] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A multidimensional data analysis system based on big data, characterized in that: It includes metadata unified management module, data full life cycle management engine module, multi-dimensional data analysis module, intelligent recommendation module, security permission module and visual display module; The metadata unified management module is used to collect, store and manage all metadata information related to data, including business metadata, technical metadata and management metadata; The data lifecycle management engine module is used to manage the entire process of data from creation to destruction, including data collection, cleaning, conversion, storage, processing, analysis, presentation, use, and archiving stages; The multidimensional data analysis module receives data from the data lifecycle management engine module, and deeply mines the trends and patterns of the data from different angles to perform multidimensional data analysis; The intelligent recommendation module recommends associated data sets and reports based on metadata and analysis results; The security permission module is used to implement data access control, encryption, auditing and monitoring measures to protect data from unauthorized access, tampering or disclosure; The visualization display module is used to convert the results of data analysis into a visualization view.

2. The multidimensional data analysis system based on big data according to claim 1, characterized in that: The metadata unified management module designs metadata entities, attributes and relationships according to business needs and data characteristics, formulates metadata naming specifications, format standards and storage rules, extracts metadata from different data sources using ETL, captures metadata generated during runtime through log analysis, and writes code according to system design documents to implement various functions of the metadata unified management module, including metadata collection, storage, retrieval, update and deletion.

3. The multidimensional data analysis system based on big data according to claim 2, characterized in that: The metadata unified management module integrates the collected metadata according to predetermined rules to form a unified metadata center, performs function and performance tests on the metadata unified management module, optimizes and adjusts the system according to the test results, and records the history of each metadata change to support rollback to the previous version.

4. The multidimensional data analysis system based on big data according to claim 1, characterized in that: The data lifecycle management engine module collects data in the metadata unified management module through ELT, imports the data into the temporary storage area, performs preliminary quality checks on the collected data, identifies missing values, outliers and inconsistent data, and performs preprocessing operations; Convert the preprocessed data into a unified format and standard, integrate multiple data sets after format conversion, generate a comprehensive data view, and store the data in a database.

5. The multidimensional data analysis system based on big data according to claim 1, characterized in that: The multidimensional data analysis module performs adaptive multidimensional calculations based on the cleaned, converted and stored data obtained from the metadata lifecycle management engine module. Assuming the evaluation function is F(D, Y), the implementation formula is: ， represents the best dimension combination selected, D represents the candidate dimension set, and Y represents the target variable; According to the query frequency analysis, based on the collection of historical query logs, the number of occurrences of each query Q at different time points t is recorded, and the implementation formula is: ， P(Q) represents the pre-calculated probability of query Q, T represents the time window length, represents the time weight, represents the frequency of query Q at time t.

6. The multidimensional data analysis system based on big data according to claim 5, characterized in that: The multidimensional data analysis module mines data associations based on the data sets in the statistical database. Suppose the support of items A and B is And conditional probability P(B|A), the implementation formula is: ， represents association rules, P(B|A) represents conditional probability, represents the support of A and B, Indicates support threshold; The multidimensional data analysis module performs real-time data stream processing in the following manner: , each data stream defines a corresponding processing function , apply a processing function to each data stream , get the processed results, summarize all the processed results, and get the final real-time analysis result R(t). The implementation formula is: ， R(t) represents the real-time analysis result, Represents the processing function, Represents a real-time data stream.

7. The multidimensional data analysis system based on big data according to claim 1, characterized in that: The intelligent recommendation module obtains detailed information of the data set from the metadata unified management module, obtains the latest analysis results from the multidimensional data analysis module, and performs data feature extraction to extract the characteristic feature vector of user u. and the feature vector of data set i , the implementation formula is: ， In the formula, represents the predicted score of user u for data set i, represents the feature vector of user t, Represents the feature vector of data set i.

8. The multidimensional data analysis system based on big data according to claim 7, characterized in that: The intelligent recommendation module includes: assuming that the original data is f(x), generating the normally distributed noise is , add noise to the original data to get the noise-added result M(x). The differential privacy protection implementation formula is: ， M(x) represents the result after adding noise, f(x) represents the original data, represents the normally distributed noise, by adjusting The size controls the degree of privacy protection.

9. The multidimensional data analysis system based on big data according to claim 1, characterized in that: The security permission module formulates detailed security policies and specifications based on security objectives, including user access rights, data encryption standards, and audit requirements, adopts strong password policies to verify the authenticity of user identities, assigns appropriate access rights to each user or user group based on the role-based permission allocation model, encrypts data transmission through the SSL / TLS protocol, encrypts sensitive data stored in the database, generates and distributes encryption keys, and monitors abnormal patterns in real time.

10. The multidimensional data analysis system based on big data according to claim 1, characterized in that: The visualization display module converts and aggregates the data according to the visualization requirements based on the data analyzed by the multidimensional data analysis module, selects the appropriate chart type according to the data type and analysis target, and maps the processed data to the visualization view for viewing.

Images