An operation platform management system based on big data analysis

Abstract

The application discloses an operation platform management system based on big data analysis, which comprises a data integration module and a data mining module. The data integration module is used for special integration of heterogeneous data. The data mining module is used for realizing data parallel mining. The data integration module comprises a heterogeneous data screening module, a local feature extraction module, a global feature library creation module and a feature mapping module. The heterogeneous data screening module is used for screening heterogeneous data in the data integration process. The local feature extraction module is used for local feature extraction of the heterogeneous data to be processed. The global feature library creation module is used for mapping the extracted local features into global features and saving them into a library. The feature mapping module is used for creating a mapping rule of the local features to the global features. The application has the characteristics of high integration precision and high mining efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to an operation platform management system based on big data analysis. BACKGROUND

[0002] With the advent of the big data era, all walks of life have begun to attach importance to the mining of big data. Big data hides huge commercial value, and has developed in an explosive manner in recent years. Compared with the growth rate of data, the analysis ability of people for big data cannot meet the requirements, and a situation of great data stock but still information scarcity occurs. Traditional data integration methods cannot handle the data semantic conflicts caused by heterogeneous data, resulting in reduced integration accuracy, and with the increase of data volume, the semantic conflicts will continue to expand, seriously affecting the data integration efficiency. Traditional data mining is affected by the performance of a single machine, and the mining depth depends on the computing power of the server, so when processing massive data, the efficiency is low. Therefore, it is necessary to design an operation platform management system based on big data analysis with high integration accuracy and high mining efficiency. SUMMARY

[0003] The present application aims to provide an operation platform management system based on big data analysis to solve the problems in the background.

[0004] In order to solve the above technical problems, the present application provides the following technical scheme: an operation platform management system based on big data analysis, comprising a data integration module and a data mining module, the data integration module is used for special integration of heterogeneous data, and the data mining module is used for realizing data parallel mining.

[0005] According to the above technical scheme, the data integration module comprises a heterogeneous data screening module, a local feature extraction module, a global feature library creation module and a feature mapping module, the heterogeneous data screening module is used for screening heterogeneous data in the data integration process, the local feature extraction module is used for local feature extraction of the heterogeneous data to be processed, the global feature library creation module is used for mapping the extracted local features to global features and saving them to the library, the feature mapping module is used for creating a mapping rule of local features to global features, and the heterogeneous data screening module is connected in a closed loop with the local feature extraction module, the global feature library creation module and the feature mapping module.

[0006] According to the technical scheme, the data mining module comprises a classification training module, an association clustering module, a prediction rule making module, a result evaluation module and a visualization operation module, the classification training module is used for classification training according to different features of data, the association clustering module is used for forming data clusters by aggregation for features with association, the prediction rule making module makes prediction rules for unknown data sets by analyzing a training data set, the result evaluation module is used for evaluating the result of data mining and screening data with high importance, and the visualization operation module is used for realizing visualization display operation with a data owner. Different features of data are divided into different classes, the classes can be created according to a training data set, a dependent relationship between different data objects is found, an association rule is determined, and a functional feature of data in a class is extracted as a feature description by association clustering.

[0007] According to the technical scheme, the operation method of the data integration module comprises the following steps:

[0008] Step A: screening data sources to screen heterogeneous data with heterogeneous characteristics; since there are a large amount of heterogeneous data, screening is required to serve as a heterogeneous data source;

[0009] Step B: local feature extraction of heterogeneous data;

[0010] Step C: creating a global feature library according to the local features;

[0011] Step D: establishing a feature mapping rule to map the local features to the global feature library.

[0012] According to the technical scheme, in step A, the screening method of the heterogeneous data is:

[0013] Step S1: preliminary screening, and the specific screening method is to preliminarily screen according to surface structure differences of data; the surface structure differences include physical storage differences, operating system differences and data format differences of data;

[0014] Step S2: further screening, and the specific screening method is to trace data sources and judge logical models, and screen according to the logical models; data is stored and maintained in different business logics, and even the same data also has logical heterogeneity, and after tracing the data, the original application scene can be traced to perform logical heterogeneity screening;

[0015] Step S3: classifying according to the screening result, and the specific classification is to screen data with heterogeneous characteristics.

[0016] According to the technical scheme, in step B, the local feature extraction method of the heterogeneous data is:

[0017] establishing a feature tree, in which all data of the heterogeneous data source is traversed;

[0018] arranging data features occurring in the traversing process into the feature tree according to data size;

[0019] taking the data feature with the largest data size in the feature tree as a representative feature of a piece of data;

[0020] performing a simulation operation on the data from which the local feature is extracted, and recording the minimum edit number n i ; the operation process is to convert a piece of data into another piece of data with the same feature through character replacement, insertion or deletion, and record the minimum edit operation number;

[0021] obtaining an operation step, converting the minimum edit number into a distance value len(n i ) according to a distance conversion algorithm. The edit number cannot be involved in the calculation in the system, and conversion into a distance value is more convenient for calculation. The distance conversion algorithm can convert the edit number into a distance value according to the specific operation step, which is convenient for calculation and preparation for subsequent global mapping. The specific conversion rule is to specify the number of steps of each operation, which is related to the degree of character change caused by the operation and is in a proportional relationship.

[0022] According to the above technical scheme, in step D, the establishment method of the feature mapping rule is:

[0023] Step a: traverse all local features in the feature tree, and perform distance value calculation to establish a distance value sequence {len(n1), len(n2), …, len(n i )};

[0024] Step b: calculate the feature distance mean Ave(n) in the feature tree, wherein:

[0025]

[0026] wherein i is the number of local features in the feature tree, and i≥1;

[0027] Step c: taking the calculated feature distance mean Ave(n) as a mapping comparison amplitude; the feature distance mean Ave(n) reflects the average feature of the heterogeneous data cluster, and taking it as the mapping comparison amplitude can ensure that the upper and lower limits of the feature mapping process are controllable;

[0028] Step d: establishing a variance operation algorithm, and traversing all data in the feature tree again on the basis of the determined mapping comparison amplitude to generate a result; the result of the variance operation algorithm shows the difference between the data feature distance value and the average value, and the data storage is screened after the screening rule is determined;

[0029] Step e: the results generated by the opposite variance algorithm are subjected to numerical limitation, data below the mapping contrast amplitude is stored in the global feature library, and data above the mapping contrast amplitude is retained in the feature tree for integration in the next cycle. The data below the mapping contrast amplitude is data with feature distance values below the average value, and these data have lower feature complexity and a smaller probability of misjudgment, which can be directly filtered out and stored in the global feature library. The data above the mapping contrast amplitude has higher feature complexity, and continues to remain in the feature tree for a new round of integration and screening with the data of the next round.

[0030] According to the above technical scheme, in the data integration module, the feature distance value with the largest proportion after the data is mapped into the global feature library is taken as the main feature, and classification screening is performed, wherein the classification screening basis is the data features extracted in the artificial preparation and big data integration process.

[0031] According to the above technical scheme, the data mining module includes a method for formulating a prediction rule, specifically:

[0032] The data with the same feature type are associated and clustered to generate a data cluster with a feature value, wherein the feature value is the average value of the feature distance of all data in the data cluster;

[0033] The data cluster with the feature value is compared with the feature tree generated in the data integration process, and data with the same feature value is screened out and marked. In the data integration process, the data stored on the feature tree and the generated data cluster with the feature value will conflict, and the same feature value will conflict between different semantics, which needs to be screened out to reduce the waste of computing power caused by redundant operations and semantic conflicts in the prediction process.

[0034] Compared with the prior art, the present application has the following beneficial effects: the present application is provided with a feature mapping module, in the data integration process, because the traditional data integration method is difficult to overcome the semantic defects of heterogeneous data, therefore, after extracting the local features of the heterogeneous data, the mapping rule from the local features to the global features is created, the global feature library is created, and then the feature comparison of the heterogeneous data is performed, the integration of the heterogeneous data is realized; the classification training module is provided, different features of the data are divided into different classes, these classes can be created according to the training data set, the dependency relationship between different data objects is found, the association rule is determined, and the functional features of the data in a class are extracted as feature descriptions through association clustering. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0036] Figure 1 Figure 1 is a schematic diagram of the system module composition of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] Please refer to Figure 1 The present application provides a technical solution: an operation platform management system based on big data analysis, comprising a data integration module and a data mining module. The data integration module is used for special integration of heterogeneous data, and the data mining module is used for realizing data parallel mining. Traditional data integration methods cannot handle semantic conflicts caused by heterogeneous data, and a large amount of redundant data caused by semantic conflicts cannot be integrated during data processing, and the data processing efficiency will be reduced with the increase of data volume. In the process of data mining, distributed computing is realized through parallelization of data mining algorithm and programming model, so as to improve the efficiency of data mining method in processing massive data.

[0039] The data integration module comprises a heterogeneous data screening module, a local feature extraction module, a global feature library creation module and a feature mapping module. The heterogeneous data screening module is used for screening heterogeneous data in the data integration process. The local feature extraction module is used for extracting local features of the heterogeneous data to be processed. The global feature library creation module is used for mapping the extracted local features to global features and saving them to the library. The feature mapping module is used for creating mapping rules of local features to global features. The heterogeneous data screening module is connected in a closed loop with the local feature extraction module, the global feature library creation module and the feature mapping module. In the data integration process, since the traditional data integration method cannot overcome the semantic defects caused by heterogeneous data, after the local features of the heterogeneous data are extracted, the mapping rules of the local features to the global features are created, the global feature library is created, and then the feature comparison of the heterogeneous data is performed, so as to realize the integration of the heterogeneous data.

[0040] The data mining module comprises a classification training module, an association clustering module, a prediction rule making module, a result evaluation module and a visualization operation module. The classification training module is used for classification training according to different features of data. The association clustering module is used for forming data clusters by aggregation for features having association. The prediction rule making module makes prediction rules for unknown data sets by analyzing training data sets. The result evaluation module is used for evaluating the results of data mining and screening out data with high importance. The visualization operation module is used for realizing visualization display operation with a data owner. Different features of data are divided into different classes. The classes can be created according to training data sets. Association rules are determined by discovering the dependency relationship between different data objects. Functional features of data in a class are extracted as feature descriptions by association clustering.

[0041] The running method of the data integration module comprises the following steps:

[0042] Step A: screening data sources to screen out heterogeneous data having heterogeneous features; since there are a large amount of heterogeneous data, screening is required to serve as a heterogeneous data source;

[0043] Step B: performing local feature extraction of the heterogeneous data;

[0044] Step C: creating a global feature library according to the local features;

[0045] Step D: establishing a feature mapping rule to map the local features to the global feature library.

[0046] In step A, the screening method of the heterogeneous data is as follows:

[0047] Step S1: performing preliminary screening. The specific screening method is to perform preliminary screening according to surface structure differences of data. The surface structure differences include physical storage differences, operating system differences and data format differences of data;

[0048] Step S2: performing further screening. The specific screening method is to trace data sources and judge logical models to perform screening according to the logical models. Data is stored and maintained in different business logics. Even the same data can have logical heterogeneity. After tracing the original application scene, logical heterogeneity screening can be performed;

[0049] Step S3: classifying according to the screening results. Specifically, data having heterogeneous features is screened.

[0050] In step B, the local feature extraction method of the heterogeneous data is as follows:

[0051] A feature tree is established. All data of the heterogeneous data source is traversed in the feature tree;

[0052] Arranging the data features appearing in the traversal process into the feature tree according to the data amount size;

[0053] Taking the feature with the largest data amount in the feature tree as the representative feature of a piece of data;

[0054] Carrying out simulation operation on the data with the extracted local features, and recording the minimum edit number n i of operation completion; the operation process is to convert a piece of data into another piece of data with the same feature through character replacement, insertion or deletion, and record the minimum edit operation number required;

[0055] Obtaining operation step, converting the minimum edit number into distance value len(n i ) according to distance conversion algorithm; the edit number cannot participate in calculation in the system, and conversion into distance value is more convenient for calculation; the distance conversion algorithm can convert the edit number into distance value according to the specific operation step, which is convenient for calculation and preparation for subsequent global mapping; the specific conversion rule is to define the step number of each operation, which is related to the character change degree caused by the operation and is in a proportional relationship.

[0056] In step D, the establishment method of the feature mapping rule is:

[0057] Step a: traversing all the local features in the feature tree, and carrying out distance value calculation to establish distance value sequence {len(n1), len(n2), …, len(n i )};

[0058] Step b: calculating the feature distance mean Ave(n) in the feature tree, wherein:

[0059]

[0060] Wherein i is the number of local features in the feature tree, and i≥1;

[0061] Step c: taking the calculated feature distance mean Ave(n) as the mapping comparison amplitude; the feature distance mean Ave(n) reflects the average feature of the heterogeneous data cluster, and taking it as the mapping comparison amplitude can ensure that the upper and lower limits of the feature mapping process are controllable;

[0062] Step d: establishing variance operation algorithm, and traversing all the data in the feature tree again on the basis of the determined mapping comparison amplitude to generate results; the results of the variance operation algorithm show the difference between the data feature distance value and the average value, and the data storage is screened after the screening rule is determined;

[0063] Step e: numerical limitation is performed on the results generated by the opposite variance algorithm, data below the mapping contrast amplitude is stored in the global feature library, and data above the mapping contrast amplitude is retained in the feature tree for integration in the next cycle. Data below the mapping contrast amplitude is data with feature distance values below the average value, which has lower feature complexity and a lower probability of misjudgment. These data can be directly filtered and stored in the global feature library. Data above the mapping contrast amplitude has higher feature complexity and continues to be retained in the feature tree for a new round of integration and selection with the next round of data.

[0064] In the data integration module, the feature distance value with the highest proportion is taken as the main feature after the data is mapped into the global feature library, and classification filtering is performed, wherein the classification filtering is based on the data features extracted during the artificial preparation and big data integration process.

[0065] According to the mining module, the method for formulating prediction rules is as follows:

[0066] The data with the same feature type are associated and clustered to generate data clusters with feature values, wherein the feature value is the average feature distance of all data in the data cluster;

[0067] The data clusters with feature values are compared with the feature tree generated in the data integration process, and data with the same feature value are selected and marked. In the data integration process, the data stored on the feature tree and the generated data clusters with feature values may conflict, and the same feature value may conflict between different semantics. Therefore, it is necessary to filter out the conflicts to reduce the waste of computing power caused by redundant operations and semantic conflicts in the prediction process.

[0068] It should be noted that, in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that there is any such actual relationship or order between these entities or actions. Moreover, the terms "comprises", "comprising", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or other elements inherent to such a process, method, article, or apparatus.

[0069] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A big data analysis-based operation platform management system, comprising a data integration module and a data mining module, characterized in that: The data integration module is used for special integration of heterogeneous data, and the data mining module is used for data parallel mining; The operation method of the data integration module comprises the following steps: Step A: screening the data source to screen out heterogeneous data with heterogeneous characteristics; Step B: local feature extraction of the heterogeneous data; Step C: creating a global feature library according to the local features; Step D: establishing a feature mapping rule to map the local features into the global feature library; In step B, the local feature extraction method of the heterogeneous data is: Establishing a feature tree, and traversing all data of the heterogeneous data source in the feature tree; Arranging the data features in the traversal process into the feature tree according to the data size; Taking the feature with the largest data size in the feature tree as the representative feature of a segment of data; Simulate operation on the data from which the local features are extracted, and record the minimum edit number n of operation completion i ; The acquisition operation step converts the minimum edit distance into a distance value len(n) according to the distance conversion algorithm i ); In step D, the establishment method of the feature mapping rule is: Step a: traverse all local features in the feature tree and perform distance value calculation to establish distance value series {len(n1), len(n2)…len(n i )}; Step b: calculating the feature distance average Ave(n) in the feature tree, wherein: Wherein i is the number of local features in the feature tree, and i≥1; Step c: taking the calculated feature distance average Ave(n) as the mapping comparison amplitude; Step d: establishing a variance operation algorithm, and traversing all data in the feature tree again on the basis of the mapping comparison amplitude to generate a result; Step e: limiting the value of the result generated by the variance algorithm, storing the data lower than the mapping comparison amplitude into the global feature library, and reserving the data higher than the mapping comparison amplitude in the feature tree to enter the integration of the next period.

2. The operation platform management system based on big data analysis according to claim 1, characterized in that: The data integration module comprises a heterogeneous data screening module, a local feature extraction module, a global feature library creation module and a feature mapping module, the heterogeneous data screening module is used for screening the heterogeneous data in the data integration process, the local feature extraction module is used for local feature extraction of the heterogeneous data to be processed, the global feature library creation module is used for mapping the extracted local features into global features and saving them into the library, and the feature mapping module is used for creating a mapping rule of the local features to the global features, and the heterogeneous data screening module, the local feature extraction module, the global feature library creation module and the feature mapping module are connected in a closed loop.

3. The big data analytics based operations platform management system of claim 2, wherein: The data mining module comprises a classification training module, an association clustering module, a prediction rule making module, a result evaluation module and a visualization operation module, the classification training module is used for classification training according to different features of the data, the association clustering module is used for forming data clusters by aggregation through features with association, the prediction rule making module makes prediction rules to predict the features of unknown data sets by analyzing the training data sets, the result evaluation module is used for evaluating the results of data mining to screen out data with high importance, and the visualization operation module is used for visual display operation with the data owner.

4. The operation platform management system based on big data analysis according to claim 3, characterized in that: In step A, the screening method of the heterogeneous data is: Step S1: preliminary screening, and the specific screening method is to preliminarily screen according to the surface structure difference of the data; Step S2: further screening, and the specific screening method is to trace the data source and judge the logical model, and screen according to the logical model; Step S3: classification according to the screening results, specifically, the data with isomerism characteristics are screened.

5. The big data analytics based operations platform management system of claim 4, wherein: In the data integration module, after the data is mapped into the global feature library, the feature distance value with the largest proportion is taken as the main feature, and classification screening is performed, wherein the classification screening basis is the data features extracted during the artificial preparation and big data integration process.

6. The big data analytics based operations platform management system of claim 5, wherein: In the data mining module, the method for formulating prediction rules is included, specifically: The data with the same feature type are associated and clustered to generate data clusters with feature values, wherein the feature value is the average feature distance of all data in the data cluster; The data clusters with feature values are compared with the feature tree generated during the data integration process, and the data with the same feature value is screened out and marked.

Images