Data mining analysis system based on OLAM

Abstract

The invention discloses an OLAM-based data mining analysis system, which belongs to the technical field of data mining analysis, and comprises a data integration preprocessing module, a multi-source data fusion module, an OLAM multi-dimensional analysis module, a data mining integration module, a visual analysis platform module and a visual portal push module, the OLAM multi-dimensional analysis module is used for a user to slice, block, rotate, roll up and drill down data from multiple angles and multiple dimensions, the OLAP tool is used for creating a multi-dimensional data set and defining the dimensions and measurement, the flexibility and depth of data analysis are improved, the data mining integration module is used for providing multiple programming languages and algorithm libraries, and the data mining integration module is used for providing the multi-dimensional data set and the multi-dimensional data set and the multi-dimensional data set and the multi-dimensional data set and the multi-dimensional data set and the multi-dimensional data set and the multi-dimensional data set. The method supports the user to carry out customized data analysis, reads the data through an API interface, imports the data into a user development environment, achieves the flexible processing and deep mining of the data, provides an interactive data exploration capability through a visual graphical interface, and reduces the data analysis threshold.

Description

Technical Field

[0001] The present invention belongs to the technical field of data mining and analysis, and specifically refers to a data mining and analysis system based on OLAM (On-line Analytical Mining). Background Art

[0002] With the advent of the big data era, enterprises have an increasing demand for data value mining. Although traditional BI (business intelligence) tools can provide basic drill-down analysis functions, they are difficult to meet the complex and ever-changing data analysis needs. At the same time, although many algorithms and tools have been developed in the field of data mining, there is a lack of a unified platform to integrate these resources and achieve efficient and intuitive data exploration and insight.

[0003] However, existing data mining and analysis still have certain defects. Existing data mining and analysis lack an effective incremental loading mechanism. It needs to extract all data each time, resulting in a heavy processing burden, especially on frequently updated data sources. This not only wastes resources, but also increases the time for data processing. In addition, the existing OLAP (Online Analytical Processing, multidimensional analysis) only provides basic multidimensional analysis functions, but lacks more advanced operations and the ability to customize dimensions and metrics, which limits the user's ability to deeply analyze data from multiple angles. It only supports limited programming languages ​​and algorithm libraries, which limits the user's ability to perform customized data analysis. Therefore, a data mining and analysis system based on OLAM is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a data mining and analysis system based on OLAM to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: a data mining and analysis system based on OLAM, comprising a data integration preprocessing module, a multi-source data fusion module, an OLAM multidimensional analysis module, a data mining integration module, a visualization analysis platform module and a visualization portal push module; The data integration preprocessing module is used to collect data from different data sources and perform data cleaning, conversion and preprocessing; The multi-source data fusion module is used to obtain the data processed by the data integration preprocessing module and integrate the data from different data sources; The OLAM multidimensional analysis module is used to slice, dice, rotate, roll up and drill down data from multiple angles and dimensions to perform multidimensional data analysis; The data mining integration module is used to provide multiple programming languages ​​and algorithm libraries for customized data analysis; The visual analysis platform module provides interactive data exploration functions through an intuitive graphical interface to complete complex data processing tasks; The visualization portal push module is used to display the analysis results in the form of charts and reports, and provide real-time updated information push services.

[0006] Furthermore, the data integration preprocessing module includes online direct reporting, directory library, Internet crawler, ETL extracts data from various data sources, sets up incremental loading mechanism for frequently updated data sources, extracts only newly added or modified data, identifies and deletes duplicate data records through unique identifiers, and preprocesses the acquired data, including removing duplicate records, processing missing values, correcting erroneous data, and standardizing data.

[0007] Furthermore, the data integration preprocessing module converts, summarizes and aggregates the preprocessed data, generates new statistical indicators, verifies the consistency of different data sources and performs data quality assessment, stores the verified data, partitions large data sets, and creates indexes for query fields.

[0008] Furthermore, the multi-source data fusion module aligns the data containing time information, and the unified implementation formula according to the time zone is: , In the formula, Indicates the original timestamp, g indicates Convert to the specified time zone , Indicates the timestamp of unified time zone after conversion; The unified implementation formula is: , In the formula, h represents the timestamp Convert to the specified format , Indicates the final timestamp in a unified format; The field matching implementation formula is: , In the formula, Represents a field in a data source. Indicates the field mapping relationship; The unit transfer realization formula is: , In the formula, represents the original unit, and u represents Convert to the specified unit ut, Indicates the unit after conversion.

[0009] Furthermore, the multi-source data fusion module performs data fusion according to the processed data, executes data merging according to the selected merging strategy, checks whether there are duplicate records and inconsistent data in the merged data, sets priorities according to business needs, selects data from more reliable data sources, and sets automated rules to resolve conflicts.

[0010] Furthermore, the OLAM multidimensional analysis module creates a multidimensional data set through an OLAP tool for the data fused by the multi-source data fusion module, defines dimensions and measures in the multidimensional data set, pre-calculates common queries, and performs analysis based on the created multidimensional data, including slicing, dicing, rotation, rolling up and drilling down.

[0011] Furthermore, the OLAM multidimensional analysis module includes: slicing selects a member of a specific dimension to view a data subset, and the implementation formula is: , In the formula, D represents dimension, m represents a member in the dimension, and C represents a multidimensional dataset. Represents a data item; Slicing is to select a member combination of multiple dimensions at the same time to view the data subset. The implementation formula is: , In the formula, Represents multiple dimensions, Represents a member in a dimension, and C represents a multidimensional dataset; Pivot means changing the position of rows and columns in a pivot table to view data from different perspectives. Roll-up means viewing higher-level summary information by aggregating more fine-grained data. Drill-down means drilling down from the highest-level summary data to more fine-grained data.

[0012] Furthermore, the data mining integration module reads data from the data warehouse by providing an API, imports the data into the user's development environment, processes missing values, outliers and duplicate records through library functions, creates new features, selects algorithms for model training according to analysis objectives, evaluates model performance through evaluation indicators, and adjusts model parameters for optimization based on the evaluation results.

[0013] Furthermore, the visualization analysis platform module integrates the R language to put custom tools for big data analysis in the hands of data analysts, supports customized, interactive data exploration, thereby gaining data insights and discovering patterns and trends, and presenting data analysis results through a visualization library based on the data.

[0014] Furthermore, the visualization portal push module obtains processed data from the OLAM multidimensional analysis module and the data mining integration module, and pushes the analysis results and real-time information to the user according to the user subscription and real-time data update status.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention collects data from different data sources, cleans, converts and preprocesses them, sets an incremental loading mechanism, extracts only newly added or modified data, reduces the processing burden, and supports incremental loading for frequently updated data sources, extracts only newly added or modified data, and improves the efficiency of data processing; 2. The present invention integrates data from different data sources to achieve comprehensiveness and integrity of data, uniformly processes time, format, field and unit, ensures consistency and comparability of data, sets priorities according to business needs, resolves data conflicts and improves data quality; 3. The present invention uses the OLAM multidimensional analysis module to slice, dice, rotate, roll up and drill down data from multiple angles and dimensions, creates multidimensional data sets through OLAP tools, defines dimensions and measures, and improves the flexibility and depth of data analysis; 4. The present invention provides a variety of programming languages ​​and algorithm libraries through the data mining integration module, supports users to perform customized data analysis, reads data through the API interface, and imports it into the user development environment to achieve flexible data processing and deep mining, and provides interactive data exploration capabilities through an intuitive graphical interface, thereby lowering the threshold for data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural schematic diagram of the data mining and analysis system based on OLAM of the present invention; Figure 2 It is an operation flow chart of the data integration preprocessing module of the data mining and analysis system based on OLAM of the present invention; Figure 3 It is an operation flow chart of the multi-source data fusion module of the data mining and analysis system based on OLAM of the present invention; Figure 4 The present invention is a flowchart of the operation of the OLAM multidimensional analysis module of the OLAM-based data mining and analysis system. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1-Figure 4 ,The present invention provides a technical solution: including a data integration preprocessing module, a multi-source data fusion module, an OLAM multidimensional analysis module, a data mining integration module, a visualization analysis platform module and a visualization portal push module; The data integration preprocessing module is used to collect data from different data sources and perform necessary data cleaning, conversion and preprocessing; The multi-source data fusion module is used to obtain the data processed by the data integration preprocessing module and integrate the data from different data sources; The OLAM multidimensional analysis module is used by users to slice, dice, rotate, roll up and drill down data from multiple angles and dimensions to perform multidimensional data analysis; The data mining integration module is used for users to perform customized data analysis using multiple programming languages ​​and algorithm libraries; The visual analysis platform module provides interactive data exploration capabilities through an intuitive graphical interface, allowing users to complete complex data processing tasks; The visualization portal push module is used to present the analysis results to users in the form of charts and reports, and includes a real-time updated information push service.

[0019] Among them, the data integration preprocessing module is used to collect data from different data sources and perform necessary data cleaning, conversion and preprocessing, including online direct reporting, directory library, Internet crawler, ETL extracts data from various data sources, and sets up an incremental loading mechanism for frequently updated data sources to only extract newly added or modified data, identify and delete duplicate data records through unique identifiers, and preprocess the acquired data, including removing duplicate records, processing missing values, correcting erroneous data and standardizing data.

[0020] Among them, data conversion is performed based on the preprocessed data, data is summarized and aggregated, new statistical indicators are generated, the consistency of different data sources is verified and data quality is evaluated, the verified data is stored, large data sets are partitioned, and indexes are created for query fields.

[0021] The multi-source data fusion module is used to obtain the data processed by the data integration preprocessing module and integrate the data from different data sources together; assuming that the data contains time information, the formula for unification according to the time zone is as follows through alignment: , In the formula, Indicates the original timestamp, g indicates Convert to the specified time zone , Indicates the timestamp of unified time zone after conversion; The unified implementation formula is: , In the formula, h represents the timestamp Convert to the specified format , Indicates the final timestamp in a unified format; The field matching implementation formula is: , In the formula, Represents a field in a data source. Indicates the field mapping relationship; The unit transfer realization formula is: , In the formula, represents the original unit, and u represents Convert to the specified unit ut, Indicates the unit after conversion.

[0022] Among them, data fusion is performed based on the processed data, data merging is performed according to the selected merging strategy, the merged data is checked for duplicate records and inconsistent data, priorities are set according to business needs, data from more reliable data sources are selected, and automated rules are set to resolve conflicts.

[0023] Among them, the OLAM multidimensional analysis module is used for users to slice, dice, rotate, roll up and drill down on data from multiple angles and dimensions to perform multidimensional data analysis. Based on the data fused by the multi-source data fusion module, a multidimensional data set is created through the OLAP tool, dimensions and measures are defined in the multidimensional data set, common queries are pre-calculated, and analysis is performed based on the created multidimensional data, including slicing, dicing, rotating, rolling up and drilling down.

[0024] Among them, the OLAM multidimensional analysis module, slicing selects a member of a specific dimension to view the data subset, and the implementation formula is: , In the formula, D represents dimension, m represents a member in the dimension, and C represents a multidimensional dataset. Represents a data item; Slicing is to select a member combination of multiple dimensions at the same time to view the data subset. The implementation formula is: , In the formula, Represents multiple dimensions, Represents a member in a dimension, and C represents a multidimensional dataset; Pivot means changing the position of rows and columns in a pivot table to view data from different perspectives. Roll-up means viewing higher-level summary information by aggregating more fine-grained data. Drill-down means drilling down from the highest-level summary data to more fine-grained data.

[0025] Among them, the data mining integration module is used for users to perform customized data analysis using multiple programming languages ​​and algorithm libraries. By providing an API, users can read data from the data warehouse and import the data into the user's development environment. Library functions are used to process missing values, outliers and duplicate records, create new features, select algorithms for model training based on analysis objectives, evaluate model performance through evaluation indicators, and adjust model parameters for optimization based on the evaluation results.

[0026] Among them, the visualization analysis platform module provides interactive data exploration capabilities through an intuitive graphical interface, allowing users to complete complex data processing tasks; by integrating the R language, custom tools for big data analysis are placed in the hands of data analysts, supporting users with data analysis skills to conduct customized and interactive data exploration, thereby gaining data insights and discovering patterns and trends, and presenting data analysis results through a visualization library based on the data.

[0027] Among them, the visualization portal push module is used to display the analysis results to users in the form of charts and reports, and includes a real-time updated information push service, which obtains processed data from the OLAM multidimensional analysis module and the data mining integration module, and pushes the analysis results and real-time information to users based on user subscriptions and real-time data updates.

[0028] In this example, specifically: the data integration preprocessing module is used to collect data from different data sources and perform necessary data cleaning, conversion and preprocessing, including online direct reporting, directory libraries, and Internet crawlers. ETL extracts data from various data sources, and for frequently updated data sources, sets an incremental loading mechanism to extract only newly added or modified data, identifies and deletes duplicate data records through unique identifiers, and preprocesses the acquired data, including removing duplicate records, processing missing values, correcting erroneous data, and standardizing data.

[0029] Among them, data conversion is performed based on the preprocessed data, data is summarized and aggregated, new statistical indicators are generated, the consistency of different data sources is verified and data quality is evaluated, the verified data is stored, large data sets are partitioned, and indexes are created for query fields.

[0030] In this example, specifically: the multi-source data fusion module is used to obtain the data processed by the data integration preprocessing module, and integrate the data from different data sources together; assuming that the data contains time information, through alignment, the unified implementation formula according to the time zone is: , In the formula, Indicates the original timestamp, g indicates Convert to the specified time zone , Indicates the timestamp of unified time zone after conversion; The unified implementation formula is: , In the formula, h represents the timestamp Convert to the specified format , Indicates the final timestamp in a unified format; The field matching implementation formula is: , In the formula, Represents a field in a data source. Indicates the field mapping relationship; The unit transfer realization formula is: , In the formula, represents the original unit, and u represents Convert to the specified unit ut, Indicates the unit after conversion.

[0031] Among them, data fusion is performed based on the processed data, data merging is performed according to the selected merging strategy, the merged data is checked for duplicate records and inconsistent data, priorities are set according to business needs, data from more reliable data sources are selected, and automated rules are set to resolve conflicts.

[0032] In this example, specifically: the OLAM multidimensional analysis module is used by users to slice, dice, rotate, roll up and drill down on data from multiple angles and dimensions to perform multidimensional data analysis. Based on the data fused by the multi-source data fusion module, a multidimensional data set is created through the OLAP tool, dimensions and measures are defined in the multidimensional data set, common queries are pre-calculated, and analysis is performed based on the created multidimensional data, including slicing, dicing, rotating, rolling up and drilling down.

[0033] Among them, the OLAM multidimensional analysis module, slicing selects a member of a specific dimension to view the data subset, and the implementation formula is: , In the formula, D represents dimension, m represents a member of dimension, and C represents multidimensional data set. Represents a data item; Slicing is to select multiple members of a combination to view the data subset at the same time. The implementation formula is: , In the formula, Represents multiple dimensions, Represents a member in a dimension, and C represents a multidimensional dataset; Pivot means changing the position of rows and columns in a pivot table to view data from different perspectives. Roll-up means viewing higher-level summary information by aggregating more fine-grained data. Drill-down means drilling down from the highest-level summary data to more fine-grained data.

[0034] Working principle: Data is collected from different data sources through the data integration preprocessing module, and necessary data cleaning, conversion and preprocessing are performed. ETL technology is used to extract data from various data sources. An incremental loading mechanism is set for frequently updated data sources to extract only newly added or modified data. Duplicate data records are identified and deleted through unique identifiers. Data is preprocessed, including removing duplicate records, processing missing values, correcting erroneous data and standardizing data. The multi-source data fusion module integrates data from different data sources, and performs operations such as data alignment, format unification, field matching and unit transfer. It aligns time information, unifies timestamps according to time zones, formats timestamps, converts to specified formats, matches fields in different data sources, establishes field mapping relationships, converts data units, ensures data consistency, performs data merging according to the selected merging strategy, and resolves data conflicts. The OLAM multidimensional analysis module allows users to slice, dice, rotate, roll up and drill down data from multiple angles and dimensions, conduct multidimensional data analysis, create multidimensional data sets through OLAP tools, define dimensions and measures, and perform common queries. The data mining integration module reads data from the data warehouse through the API, imports data into the user's development environment, and uses library functions to process missing values, outliers, and duplicate records, creates new features, selects algorithms for model training according to the analysis objectives, uses evaluation indicators to evaluate model performance, and adjusts model parameters for optimization based on the evaluation results. The visual analysis platform module integrates data analysis tools such as R language, supports users to customize and interactively explore data, uses visualization libraries to present data analysis results, helps users discover patterns and trends in data, displays analysis results to users in the form of charts and reports, and provides real-time updated information push services.

[0035] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0036] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A data mining and analysis system based on OLAM, characterized by: It includes data integration preprocessing module, multi-source data fusion module, OLAM multi-dimensional analysis module, data mining integration module, visual analysis platform module and visual portal push module; The data integration preprocessing module is used to collect data from different data sources and perform data cleaning, conversion and preprocessing; The multi-source data fusion module is used to obtain the data processed by the data integration preprocessing module and integrate the data from different data sources; The OLAM multidimensional analysis module is used to slice, dice, rotate, roll up and drill down the data from multiple angles and dimensions to perform multidimensional data analysis; The data mining integration module is used to provide multiple programming languages ​​and algorithm libraries for customized data analysis; The visual analysis platform module provides interactive data exploration functions through an intuitive graphical interface to complete complex data processing tasks; The visualization portal push module is used to display the analysis results in the form of charts and reports, and provide real-time updated information push services.

2. The data mining and analysis system based on OLAM according to claim 1, characterized in that: The data integration preprocessing module includes online direct reporting, directory library, and Internet crawler. ETL extracts data from various data sources. For frequently updated data sources, an incremental loading mechanism is set up to extract only newly added or modified data, and duplicate data records are identified and deleted through unique identifiers. The acquired data is preprocessed, including removing duplicate records, processing missing values, correcting erroneous data, and standardizing data.

3. The data mining and analysis system based on OLAM according to claim 2 is characterized in that: The data integration preprocessing module converts, summarizes and aggregates the preprocessed data, generates new statistical indicators, verifies the consistency of different data sources and performs data quality assessment, stores the verified data, partitions large data sets, and creates indexes for query fields.

4. The data mining and analysis system based on OLAM according to claim 1, characterized in that: The multi-source data fusion module aligns the data containing time information, and the unified implementation formula according to the time zone is: ， In the formula, Indicates the original timestamp, g indicates Convert to the specified time zone , Indicates the timestamp of unified time zone after conversion; The unified implementation formula is: ， In the formula, h represents the timestamp Convert to the specified format , Indicates the final timestamp in a unified format; The field matching implementation formula is: ， In the formula, Represents a field in a data source. Indicates the field mapping relationship; The unit transfer realization formula is: ， In the formula, represents the original unit, and u represents Convert to the specified unit ut, Indicates the unit after conversion.

5. The OLAM-based data mining and analysis system according to claim 4, characterized in that: The multi-source data fusion module performs data fusion according to the processed data, executes data merging according to the selected merging strategy, checks whether there are duplicate records and inconsistent data in the merged data, sets priorities according to business needs, selects data from more reliable data sources, and sets automated rules to resolve conflicts.

6. The data mining and analysis system based on OLAM according to claim 1, characterized in that: The OLAM multidimensional analysis module creates a multidimensional data set through the OLAP tool for the data fused by the multi-source data fusion module, defines dimensions and measures in the multidimensional data set, pre-calculates common queries, and performs analysis based on the created multidimensional data, including slicing, dicing, rotation, roll-up and drill-down.

7. The OLAM-based data mining and analysis system according to claim 6, characterized in that: The OLAM multidimensional analysis module includes: slicing selects a member of a specific dimension to view a data subset, and the implementation formula is: ， In the formula, D represents dimension, m represents a member in the dimension, and C represents a multidimensional dataset. Represents a data item; Slicing is to select a member combination of multiple dimensions at the same time to view the data subset. The implementation formula is: ， In the formula, Represents multiple dimensions, Represents a member in a dimension, and C represents a multidimensional dataset; Pivot means changing the position of rows and columns in a pivot table to view data from different perspectives. Roll-up means viewing higher-level summary information by aggregating more fine-grained data. Drill-down means drilling down from the highest-level summary data to more fine-grained data.

8. The data mining and analysis system based on OLAM according to claim 1, characterized in that: The data mining integration module reads data from the data warehouse by providing an API, imports the data into the user's development environment, processes missing values, outliers and duplicate records through library functions, creates new features, selects algorithms for model training based on analysis objectives, evaluates model performance through evaluation indicators, and adjusts model parameters for optimization based on the evaluation results.

9. The data mining and analysis system based on OLAM according to claim 1, characterized in that: The visualization analysis platform module integrates the R language to put custom tools for big data analysis in the hands of data analysts, supports custom, interactive data exploration, thereby gaining data insights and discovering patterns and trends, and presenting data analysis results through a visualization library based on the data.

10. The data mining and analysis system based on OLAM according to claim 1, characterized in that: The visualization portal push module obtains processed data from the OLAM multidimensional analysis module and the data mining integration module, and pushes analysis results and real-time information to users based on user subscriptions and real-time data updates.

Images