Big data-based data intelligent monitoring management platform and intelligent monitoring management method

Abstract

The application relates to the field of data monitoring, in particular to a data intelligent monitoring management platform and method based on big data, which comprises a governance object database acquisition module, a data governance feature model construction module, a governance configuration priority analysis module, an identifiable feature analysis module, a matching group generation module and a data matching analysis module; the governance object database acquisition module is used for acquiring each governance object database of an application data governance center monitoring platform; the data governance feature model construction module is used for constructing a data governance feature model corresponding to each governance object database; the governance configuration priority analysis module is used for outputting a governance configuration priority of each governance object database; the identifiable feature analysis module is used for analyzing identifiable features of a database analysis set; and the matching group generation module is used for forming a matching group by combining the identifiable features with governance configuration priorities recorded in the corresponding database analysis set.

Description

Technical Field

[0001] This invention relates to the field of data monitoring technology, specifically to a data intelligent monitoring and management platform and intelligent monitoring and management method based on big data. Background Technology

[0002] With the development of big data, cloud computing, and algorithms, the wave of artificial intelligence has continued for several years and has been widely applied in various industries and fields, becoming a leading technology in the next technological revolution. Similarly, with the surge in data volume and sources, data governance has become an indispensable part of enterprises' efforts to fully explore and utilize data value, and has gradually developed into one of their core businesses. Data governance has undergone many changes from concept to technology, especially the effective integration of data governance technology and artificial intelligence technology, making intelligent data governance possible. However, in various databases using data governance platforms, the governance dimensions are often set by the platform itself, lacking adaptability to data dimensions. Furthermore, the matching degree of the governance dimension scheme established by the platform to the database needs to be considered when facing diverse data to be processed. Therefore, how to maximize the data governance effect on different databases based on data governance platforms is a direction that requires further research. Summary of the Invention

[0003] The purpose of this invention is to provide a data intelligent monitoring and management platform and intelligent monitoring and management method based on big data, so as to solve the problems mentioned in the background art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a data intelligent monitoring and management method based on big data, comprising the following analysis steps: Step S100: Obtain each governance object database of the application data governance center monitoring platform, and the governance events recorded by each governance object database in different governance dimensions within the same monitoring period. Construct a data governance feature model corresponding to each governance object database based on the governance events. Step S200: Based on the data governance feature model, output the governance configuration priority of each governance object database; Step S300: Extract the database of governance objects whose similarity to the data governance feature model is greater than or equal to the similarity threshold to form a database analysis set. Analyze the identifiable features of each database analysis set and form a matching group with the governance configuration priority recorded in the corresponding database analysis set. Step S400: Obtain the database of governance objects that have not applied the data governance center monitoring platform as the database to be analyzed, read the identifiable features that meet the requirements of the database to be analyzed as the target features, and output the governance configuration priority required for the database to be analyzed when applying the data governance center monitoring platform based on the matching group to which the target features belong.

[0005] Furthermore, step S100 includes the following analysis steps: Step S110: Governance event log data governance scheme and data governance process in the data governance workbench; different governance dimensions refer to the various dimensions in the data governance scheme; obtain the name of the j-th scheme recorded under the i-th governance dimension. The unit table A that constitutes the record of the i-th governance dimension i Each scheme name corresponds to a scheme description and optimization suggestions, which together constitute a scheme regulation. Step S120: Obtain the total amount of data M to be processed under m scheme rules recorded in n unit tables, and the number of data Q to be processed within the same time interval after implementing the optimization suggestions, where n represents the total number of unit tables. 'm' represents the number of scheme rules recorded in each unit table. ; Reprocessing refers to the secondary or multiple processing steps performed on the same data after governance and optimization; using the formula:

[0006] Calculate the i-th cell table Effective governance index of the j-th scheme regulation and output the cell table. Based on the average effective governance index under all schemes and regulations ;in Represents the table of the i-th unit The total amount of data to be processed under the j-th scheme regulation. Represents the table of the i-th unit The number of reprocessed data items under the j-th scheme regulation; A higher effective governance index indicates that the database performs better after governance and optimization of the corresponding scheme and regulations. Step S130: Establish a two-dimensional rectangular coordinate system, sort the average effective governance indexes corresponding to the n unit tables from smallest to largest and mark them on the vertical axis. With the unit table as the horizontal axis and the average effective governance index corresponding to the unit table as the vertical axis, draw a bar chart of the governance dimensions corresponding to the unit table as the data governance feature model.

[0007] Furthermore, the governance configuration priority for each governance object database is output, including the following steps: Step S210: Obtain the data governance feature model generated by the database of each governance object, extract the maximum value of the bar graph corresponding to each unit table in the model, sort each unit table from smallest to largest based on the maximum value of the bar graph, and mark the governance dimension corresponding to the sorting. Step S220: Based on the sorting order of the marked governance dimensions, output the governance configuration priority of the database corresponding to each governance object; the governance configuration priority refers to the number of configuration scheme rules for the governance dimension that is sorted first being greater than the number of historical governance configuration scheme rules and greater than the number of configuration scheme rules for the governance dimension that is sorted last.

[0008] Increasing the number of configuration scheme regulations corresponding to governance dimensions can maximize the comprehensiveness of anomaly monitoring in the governance object database during the monitoring process. Reasonable configuration from different governance dimensions makes data governance more intelligent and targeted, rather than a uniform execution standard.

[0009] Furthermore, step S300 includes the following steps: Step S310: Obtain the average effective governance index in the data governance feature model. The first index is obtained by summing all governance dimensions in the governance object database. , ; A similarity greater than or equal to the similarity threshold means that the difference in the first index between any two governance object databases is less than or equal to the difference threshold and the governance configuration priority is the same. Step S320: Mark the governance dimension with the highest priority corresponding to the governance configuration of each governance object database in the database analysis set as the target dimension, and obtain the total number of issues generated by all governance projects under the target dimension. Data on issues to be processed and the number of problems to be assigned Calculate the data characteristic value T for each governance project. The governance project corresponding to the largest extracted data feature value is the feature to be analyzed and identified under the target dimension. Step S330: Verify the features to be analyzed and identified. The verification process is as follows: Extract the governance dimensions other than the target dimension from the governance object database as verification governance dimensions, calculate the data feature values ​​corresponding to the verification governance dimensions, and the verification identification features determined based on the data feature values. When the feature to be analyzed and identified is different from the verification and identification features of all verification and governance dimensions, the feature to be analyzed and identified is output as the valid identification feature of the governance object database; when the feature to be analyzed and identified is the same as the verification and identification feature of any verification and governance dimension, the data feature values ​​are sorted, and the governance items that satisfy the uniqueness of the feature to be analyzed and identified are identified as the valid identification features. Step S340: When the effective identification features corresponding to each governance object database in the database analysis set are the same, the effective identification features are output as identifiable features; when the effective identification features corresponding to each governance object database in the database analysis set are different, the effective identification features are filtered and analyzed.

[0010] Further screening analysis includes the following steps: Step S341: Obtain all valid identification features contained in the database analysis set; Calculate the effective governance probability of the target dimension of the database analysis set records. ,

[0011] This represents the average effective governance index corresponding to the target dimension; Calculate the record probability of valid identification features. ,

[0012] in This indicates the number of times a governance project corresponding to a valid identification feature is recorded in the governance object database. This indicates the number of records for all governance items in the governance object database; Step S342: Using the formula: Calculate the recording probability of effective recognition features in the target dimension; where This represents the ratio of the effective governance index when the target dimension contains effective identification features to the average effective governance index corresponding to the target dimension. Step S343: Output , This represents the probability that the governance dimension of the governance object database is the target dimension when a valid identification feature appears in the governance object database; Step S344: Extract data from the database analysis collection. The effective identifiable feature corresponding to the maximum value is the identifiable feature.

[0013] Further, step S400 includes the following: calculating the data feature value of each governance project in the database to be analyzed as the feature value to be analyzed; meeting the requirement means having the smallest difference with the feature value to be analyzed; outputting the identifiable feature corresponding to the data feature value with the smallest difference with the feature value to be analyzed as the target feature.

[0014] The big data-based intelligent monitoring and management platform includes a governance object database acquisition module, a data governance feature model construction module, a governance configuration priority analysis module, an identifiable feature analysis module, a matching group generation module, and a data matching analysis module. The governance object database acquisition module is used to acquire the database of each governance object from the application data governance center monitoring platform; The data governance feature model building module is used to build a data governance feature model corresponding to the database of each governance object based on governance events; The governance configuration priority analysis module is used to output the governance configuration priority of each governance object database; The identifiable feature analysis module is used to analyze the identifiable features of each database analysis set; The matching group generation module is used to form matching groups by matching identifiable features with the governance configuration priorities of the corresponding database analysis set records; The data matching and analysis module is used to read identifiable features that meet the requirements of the database to be analyzed as target features, and outputs the governance configuration priority required by the data governance center monitoring platform for the database to be analyzed based on the matching group to which the target features belong.

[0015] Furthermore, the data governance feature model construction module includes a unit table construction unit, an effective governance index calculation unit, and a data governance feature model output unit; The unit table construction unit is used to retrieve the solution name, solution description and optimization suggestion recorded under the governance dimension to form the unit table of the i-th governance dimension record; The effective governance index calculation unit is used to obtain the total amount of data to be processed and the number of data to be processed again within the same time interval after implementing optimization suggestions, and to calculate the effective governance index. The output unit of the data governance feature model is used to sort the average effective governance index corresponding to the unit table from small to large and mark it on the vertical axis. With the unit table as the horizontal axis and the average effective governance index corresponding to the unit table as the vertical axis, a bar chart of the governance dimension of the generated unit table is drawn as the data governance feature model.

[0016] Furthermore, the identifiable feature analysis module includes a first index calculation unit, a target dimension determination unit, a feature differentiation unit to be analyzed and identified unit, and a verification feature screening unit; The first index calculation unit is used to sum all governance dimensions in the governance object database to obtain the first index; The target dimension determination unit is used to mark the highest priority governance dimension of each governance object database in the database analysis set as the target dimension. The feature differentiation unit is used to extract the governance project corresponding to the target dimension as the feature to be analyzed and identified when the data feature value is the largest. The verification feature filtering unit is used to verify and filter the features to be analyzed and identified.

[0017] Furthermore, the verification feature selection unit includes a probability analysis unit and a recognizable feature output unit; The probability analysis unit is used to calculate the probability that the governance dimension of the governance object database is the target dimension when a valid identification feature appears in the governance object database; The identifiable feature output unit is used to extract the effective identifiable feature corresponding to the maximum data result of the probability analysis unit.

[0018] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention extracts governance data recorded by the data governance center monitoring platform in each governance object database to analyze the governance dimension characteristics and corresponding governance configuration priorities of the governance center in different governance object databases; thereby achieving efficient and personalized governance based on diverse data foundations and improving the effectiveness of data governance; in addition, this invention increases the number of configuration scheme regulations corresponding to governance dimensions, which can maximize the comprehensiveness of anomaly monitoring in the governance object database during the monitoring process. Reasonable configuration from different governance dimension directions makes data governance more intelligent and targeted, no longer a uniform execution scheme, so that the data governance center can adaptively adjust the configuration scheme of each database with the data itself as the analysis subject during the data governance process. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the data intelligence monitoring and management platform based on big data according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 This invention provides a technical solution: a data intelligent monitoring and management method based on big data, comprising the following analysis steps: Step S100: Obtain each governance object database of the application data governance center monitoring platform, and the governance events recorded by each governance object database in different governance dimensions within the same monitoring period. Construct a data governance feature model corresponding to each governance object database based on the governance events. Step S200: Based on the data governance feature model, output the governance configuration priority of each governance object database; Step S300: Extract the database of governance objects whose similarity to the data governance feature model is greater than or equal to the similarity threshold to form a database analysis set. Analyze the identifiable features of each database analysis set and form a matching group with the governance configuration priority recorded in the corresponding database analysis set. Step S400: Obtain the database of governance objects that have not applied the data governance center monitoring platform as the database to be analyzed, read the identifiable features that meet the requirements of the database to be analyzed as the target features, and output the governance configuration priority required for the database to be analyzed when applying the data governance center monitoring platform based on the matching group to which the target features belong.

[0022] Step S100 includes the following analysis steps: Step S110: Governance event log data governance scheme and data governance process in the data governance workbench; different governance dimensions refer to the various dimensions in the data governance scheme; obtain the name of the j-th scheme recorded under the i-th governance dimension. The unit table A that constitutes the record of the i-th governance dimension i Each scheme name corresponds to a scheme description and optimization suggestions, which together constitute a scheme regulation. Different governance dimensions refer to the five dimensions of computation, storage, quality, standardization, and value in the data governance solution; and the governance solutions implemented by the data governance center monitoring platform for different governance object databases are the same in the historical records. Step S120: Obtain the total amount of data M to be processed under m scheme rules recorded in n unit tables, and the number of data Q to be processed within the same time interval after implementing the optimization suggestions, where n represents the total number of unit tables. 'm' represents the number of scheme rules recorded in each unit table. ; Reprocessing refers to the secondary or multiple processing steps performed on the same data after governance and optimization; using the formula:

[0023] Calculate the i-th cell table Effective governance index of the j-th scheme regulation and output the cell table. Based on the average effective governance index under all schemes and regulations ;in Represents the table of the i-th unit The total amount of data to be processed under the j-th scheme regulation. Represents the table of the i-th unit The number of reprocessed data items under the j-th scheme regulation; A higher effective governance index indicates that the database performs better after governance and optimization of the corresponding scheme and regulations. For example, in the analysis and calculation dimension, based on the tasks of each project, we can judge excessively long tasks and abnormal situations. There is a solution named "Excessively Long Tasks". The corresponding solution description is "There are task instances with a runtime of more than 10 hours in the past 7 days". The optimization suggestion is "If the excessively long execution is caused by the large amount of data, it is recommended to add it to the whitelist". If, after optimizing the computational dimensions of Task 1, there are still solutions that satisfy the "ultra-long task" requirement, then Task 1 will be considered as reprocessed data.

[0024] Step S130: Establish a two-dimensional rectangular coordinate system, sort the average effective governance indexes corresponding to the n unit tables from smallest to largest and mark them on the vertical axis. With the unit table as the horizontal axis and the average effective governance index corresponding to the unit table as the vertical axis, draw a bar chart of the governance dimensions corresponding to the unit table as the data governance feature model.

[0025] Output the governance configuration priority for each governance object database, including the following steps: Step S210: Obtain the data governance feature model generated by the database of each governance object, extract the maximum value of the bar graph corresponding to each unit table in the model, sort each unit table from smallest to largest based on the maximum value of the bar graph, and mark the governance dimension corresponding to the sorting. Step S220: Based on the sorting order of the marked governance dimensions, output the governance configuration priority of the database corresponding to each governance object; the governance configuration priority refers to the number of configuration scheme rules for the governance dimension that is sorted first being greater than the number of historical governance configuration scheme rules and greater than the number of configuration scheme rules for the governance dimension that is sorted last.

[0026] Increasing the number of configuration scheme regulations corresponding to governance dimensions can maximize the comprehensiveness of anomaly monitoring in the governance object database during the monitoring process. Reasonable configuration from different governance dimensions makes data governance more intelligent and targeted, rather than a uniform execution standard.

[0027] By analyzing the data governance characteristic model, we can obtain the governance effect of the same governance object database in five governance dimensions. Then, based on the difference in effect, we can prioritize the configuration of the less effective database and the normal configuration of the more effective database.

[0028] Step S300 includes the following steps: Step S310: Obtain the average effective governance index in the data governance feature model. The first index is obtained by summing all governance dimensions in the governance object database. , ; A similarity greater than or equal to the similarity threshold means that the difference in the first index between any two governance object databases is less than or equal to the difference threshold and the governance configuration priority is the same. Step S320: Mark the governance dimension with the highest priority corresponding to the governance configuration of each governance object database in the database analysis set as the target dimension, and obtain the total number of issues generated by all governance projects under the target dimension. Data on issues to be processed and the number of problems to be assigned Calculate the data characteristic value T for each governance project. The governance project corresponding to the largest extracted data feature value is the feature to be analyzed and identified under the target dimension. Step S330: Verify the features to be analyzed and identified. The verification process is as follows: Extract the other governance dimensions from the governance object database except for the target dimension as the verification governance dimensions. Calculate the data feature values ​​corresponding to the verification governance dimensions and the verification identification features determined based on the data feature values. The verification identification features refer to the governance items corresponding to the largest data feature values ​​of the verification governance dimensions. When the feature to be analyzed and identified is different from the verification and identification features of all verification and governance dimensions, the feature to be analyzed and identified is output as the valid identification feature of the governance object database; when the feature to be analyzed and identified is the same as the verification and identification feature of any verification and governance dimension, the data feature values ​​are sorted, and the governance items that satisfy the uniqueness of the feature to be analyzed and identified are identified as the valid identification features. Step S340: When the effective identification features corresponding to each governance object database in the database analysis set are the same, the effective identification features are output as identifiable features; when the effective identification features corresponding to each governance object database in the database analysis set are different, the effective identification features are filtered and analyzed.

[0029] As shown in the example: There exists a database analysis set C, which includes governance object database 1 and governance object database 2; If the governance dimension with the highest priority in the governance configuration corresponding to database analysis set C is: computation dimension; The governance items included in the computation dimension are Governance Item 1 and Governance Item 2; Calculate the data characteristic value of governance object database 1 in governance project 1. The data characteristic value of governance project 1 is The data characteristic value of governance object database 2 in governance project 1 is... The data characteristic value of governance project 1 is ; The governance projects corresponding to the maximum values ​​are selected as the features to be analyzed and identified under the target dimension. If the feature to be analyzed and identified in the computational dimension of governance object database 1 is governance project 1; and the feature to be analyzed and identified in the computational dimension of governance object database 2 is governance project 1; Verification is performed by calculating the corresponding features to be analyzed and identified in other dimensions such as storage and quality for the two databases. If the features to be analyzed and identified in other governance dimensions of the two databases are not governance item 1, then governance item 1 can be used as a valid identification feature. Furthermore, since the effective identifiable feature in both databases is governance item 1, governance item 1 is an identifiable feature of the database analysis set C.

[0030] The screening analysis includes the following steps: Step S341: Obtain all valid identification features contained in the database analysis set; Calculate the effective governance probability of the target dimension of the database analysis set records. ,

[0031] This represents the average effective governance index corresponding to the target dimension; the data environment for analyzing different effective identification features is the governance object database where the effective identification features are located. Calculate the record probability of valid identification features. ,

[0032] in This indicates the number of times a governance project corresponding to a valid identification feature is recorded in the governance object database. This indicates the number of records for all governance items in the governance object database; Step S342: Using the formula: Calculate the recording probability of effective recognition features in the target dimension; where It represents the ratio of the effective governance index when the target dimension contains effective identification features to the average effective governance index corresponding to the target dimension. The effective governance index when the target dimension contains effective identification features is obtained based on the formula in step S120, which limits the total amount of data to be processed M and the number of data to be processed Q within the same interval after the optimization suggestions are implemented, with the governance items corresponding to the effective identification features as the premise. Step S343: Output , This represents the probability that the governance dimension of the governance object database is the target dimension when a valid identification feature appears in the governance object database; Step S344: Extract data from the database analysis collection. The effective identifiable feature corresponding to the maximum value is the identifiable feature.

[0033] Analyzing conditional probability is to identify the probability of each feature under the same evaluation criteria. The higher the probability, the more representative the corresponding feature is of the target dimension.

[0034] Step S400 includes the following: calculating the data feature value of each governance project in the database to be analyzed as the feature value to be analyzed; meeting the requirement means having the smallest difference with the feature value to be analyzed; outputting the identifiable feature corresponding to the data feature value with the smallest difference with the feature value to be analyzed as the target feature.

[0035] As shown in the example: The feature to be analyzed is t, and the data feature with the smallest difference from the feature to be analyzed is T1. The identifiable feature corresponding to T1 is governance project 1. Match the target dimension of the record corresponding to governance project 1. If it is a calculation dimension, then extract the governance configuration priority when the calculation dimension is ranked first in the governance configuration priority, which is the basis for the configuration of each dimension of the database.

[0036] The big data-based intelligent monitoring and management platform includes a governance object database acquisition module, a data governance feature model construction module, a governance configuration priority analysis module, an identifiable feature analysis module, a matching group generation module, and a data matching analysis module. The governance object database acquisition module is used to acquire the database of each governance object from the application data governance center monitoring platform; The data governance feature model building module is used to build a data governance feature model corresponding to the database of each governance object based on governance events; The governance configuration priority analysis module is used to output the governance configuration priority of each governance object database; The identifiable feature analysis module is used to analyze the identifiable features of each database analysis set; The matching group generation module is used to form matching groups by matching identifiable features with the governance configuration priorities of the corresponding database analysis set records; The data matching and analysis module is used to read identifiable features that meet the requirements of the database to be analyzed as target features, and outputs the governance configuration priority required by the data governance center monitoring platform for the database to be analyzed based on the matching group to which the target features belong.

[0037] The data governance feature model construction module includes a unit table construction unit, an effective governance index calculation unit, and a data governance feature model output unit; The unit table construction unit is used to retrieve the solution name, solution description and optimization suggestion recorded under the governance dimension to form the unit table of the i-th governance dimension record; The effective governance index calculation unit is used to obtain the total amount of data to be processed and the number of data to be processed again within the same time interval after implementing optimization suggestions, and to calculate the effective governance index. The output unit of the data governance feature model is used to sort the average effective governance index corresponding to the unit table from small to large and mark it on the vertical axis. With the unit table as the horizontal axis and the average effective governance index corresponding to the unit table as the vertical axis, a bar chart of the governance dimension of the generated unit table is drawn as the data governance feature model.

[0038] The identifiable feature analysis module includes a first index calculation unit, a target dimension determination unit, a feature differentiation unit to be analyzed and identified unit, and a verification feature screening unit; The first index calculation unit is used to sum all governance dimensions in the governance object database to obtain the first index; The target dimension determination unit is used to mark the highest priority governance dimension of each governance object database in the database analysis set as the target dimension. The feature differentiation unit is used to extract the governance project corresponding to the target dimension as the feature to be analyzed and identified when the data feature value is the largest. The verification feature filtering unit is used to verify and filter the features to be analyzed and identified.

[0039] The verification feature selection unit includes a probability analysis unit and a recognizable feature output unit; The probability analysis unit is used to calculate the probability that the governance dimension of the governance object database is the target dimension when a valid identification feature appears in the governance object database; The identifiable feature output unit is used to extract the effective identifiable feature corresponding to the maximum data result of the probability analysis unit.

[0040] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0041] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data-intelligent monitoring and management method based on big data, characterized in that: The analysis includes the following steps: Step S100: Obtain each governance object database of the application data governance center monitoring platform, and the governance events recorded by each governance object database in different governance dimensions within the same monitoring period. Construct a data governance feature model corresponding to each governance object database based on the governance events. Step S100 includes the following analysis steps: Step S110: The governance event record data governance scheme and the data governance process in the data governance workbench; the different governance dimensions refer to the various dimensions in the data governance scheme; obtain the name of the j-th scheme recorded under the i-th governance dimension. Solution Description and optimization suggestions The unit table A that constitutes the record of the i-th governance dimension i Each scheme name corresponds to a scheme description and optimization suggestions, which together constitute a scheme regulation. Step S120: Obtain the total amount of data M to be processed under the m scheme rules recorded in n unit tables, and the number of data Q to be processed within the same time interval after implementing the optimization suggestions, where n represents the total number of unit tables. 'm' represents the number of scheme rules recorded in each unit table. ; The reprocessing refers to the secondary or multiple processing steps performed on the same data to be processed after optimization; using the formula: Calculate the i-th cell table Effective governance index of the j-th scheme regulation and output the cell table. Based on the average effective governance index under all schemes and regulations ;in Represents the table of the i-th unit The total amount of data to be processed under the j-th scheme regulation. Represents the table of the i-th unit The number of reprocessed data items under the j-th scheme regulation; Step S130: Establish a two-dimensional rectangular coordinate system, sort the average effective governance index corresponding to the n unit tables from small to large and mark it on the vertical axis, and draw a bar chart of the governance dimension corresponding to the unit tables as the data governance feature model with the unit tables as the horizontal axis and the average effective governance index corresponding to the unit tables as the vertical axis. Step S200: Based on the data governance feature model, output the governance configuration priority of each governance object database; Step S300: Extract the database of governance objects whose similarity to the data governance feature model is greater than or equal to the similarity threshold to form a database analysis set. Analyze the identifiable features of each database analysis set and form a matching group with the governance configuration priority recorded in the corresponding database analysis set. Step S400: Obtain the database of governance objects that have not applied the data governance center monitoring platform as the database to be analyzed, read the identifiable features that meet the requirements of the database to be analyzed as the target features, and output the governance configuration priority required for the database to be analyzed when applying the data governance center monitoring platform based on the matching group to which the target features belong.

2. The data intelligent monitoring and management method based on big data according to claim 1, characterized in that: The process of outputting the governance configuration priority for each governance object database includes the following steps: Step S210: Obtain the data governance feature model generated by the database of each governance object, extract the maximum value of the bar graph corresponding to each unit table in the model, sort each unit table from small to large based on the maximum value of the bar graph, and mark the governance dimension corresponding to the sorting. Step S220: Based on the sorting order of the marked governance dimensions, output the governance configuration priority corresponding to each governance object database; the governance configuration priority refers to the number of configuration scheme regulations for the governance dimension that is sorted first being greater than the number of historical governance configuration scheme regulations and greater than the number of configuration scheme regulations for the governance dimension that is sorted last.

3. The data intelligent monitoring and management method based on big data according to claim 2, characterized in that: Step S300 includes the following steps: Step S310: Obtain the average effective governance index in the data governance feature model. The first index is obtained by summing all governance dimensions in the governance object database. , ; The similarity greater than or equal to the similarity threshold means that the difference in the first index between any two governance object databases is less than or equal to the difference threshold and the governance configuration priority is the same; Step S320: Mark the governance dimension with the highest priority corresponding to the governance configuration of each governance object database in the database analysis set as the target dimension, and obtain the total number of issues generated by all governance projects under the target dimension. Data on issues to be processed and the number of problems to be assigned Calculate the data characteristic value T for each governance project. The governance project corresponding to the largest extracted data feature value is the feature to be analyzed and identified under the target dimension. Step S330: Verify the features to be analyzed and identified. The verification process is as follows: Extract the other governance dimensions from the governance object database except for the target dimension as the verification governance dimensions, calculate the data feature values ​​corresponding to the verification governance dimensions and the verification identification features determined based on the data feature values; When the feature to be analyzed and identified is different from the verification and identification features of all verification and governance dimensions, the feature to be analyzed and identified is output as the valid identification feature of the governance object database; when the feature to be analyzed and identified is the same as the verification and identification feature of any verification and governance dimension, the data feature values ​​are sorted, and the governance items that satisfy the uniqueness of the feature to be analyzed and identified are identified as the valid identification features. Step S340: When the effective identification features corresponding to each governance object database in the database analysis set are the same, the effective identification features are output as identifiable features; when the effective identification features corresponding to each governance object database in the database analysis set are different, the effective identification features are filtered and analyzed.

4. The data intelligent monitoring and management method based on big data according to claim 3, characterized in that: The screening analysis includes the following steps: Step S341: Obtain all valid identification features contained in the database analysis set; Calculate the effective governance probability of the target dimension of the database analysis set records. , This represents the average effective governance index corresponding to the target dimension; Calculate the record probability of valid identification features , in This indicates the number of times a governance project corresponding to a valid identification feature is recorded in the governance object database. This indicates the number of records for all governance items in the governance object database; Step S342: Using the formula: Calculate the recording probability of effective recognition features in the target dimension; where This represents the ratio of the effective governance index when the target dimension contains effective identification features to the average effective governance index corresponding to the target dimension. Step S343: Output , This represents the probability that the governance dimension of the governance object database is the target dimension when a valid identification feature appears in the governance object database; Step S344: Extract data from the database analysis collection. The effective identifiable feature corresponding to the maximum value is the identifiable feature.

5. The data intelligent monitoring and management method based on big data according to claim 4, characterized in that: Step S400 includes the following: The data feature value of each governance project in the database to be analyzed is calculated as the feature value to be analyzed; the requirement is that the difference between the data feature value and the feature value to be analyzed is the smallest; the identifiable feature corresponding to the data feature value with the smallest difference from the feature value to be analyzed is output as the target feature.

6. A data intelligence monitoring and management platform based on big data, employing the data intelligence monitoring and management method based on big data as described in any one of claims 1-4, characterized in that, It includes a governance object database acquisition module, a data governance feature model construction module, a governance configuration priority analysis module, an identifiable feature analysis module, a matching group generation module, and a data matching analysis module; The governance object database acquisition module is used to acquire the database of each governance object in the application data governance center monitoring platform. The data governance feature model construction module is used to construct a data governance feature model corresponding to the database of each governance object based on governance events. The governance configuration priority analysis module is used to output the governance configuration priority of each governance object database; The identifiable feature analysis module is used to analyze the identifiable features of each database analysis set; The matching group generation module is used to form a matching group by matching identifiable features with the governance configuration priority recorded in the corresponding database analysis set; The data matching and analysis module is used to read identifiable features that meet the requirements of the database to be analyzed as target features, and output the governance configuration priority of the database to be analyzed when applying the data governance center monitoring platform based on the matching group to which the target features belong.

7. The data intelligent monitoring and management platform based on big data according to claim 6, characterized in that: The data governance feature model construction module includes a unit table construction unit, an effective governance index calculation unit, and a data governance feature model output unit. The unit table construction unit is used to retrieve the scheme name, scheme description and optimization suggestions recorded under the governance dimension to form the unit table of the i-th governance dimension record; The effective governance index calculation unit is used to obtain the total amount of data to be processed and the number of data to be processed again within the same time interval after implementing optimization suggestions, and to calculate the effective governance index. The output unit of the data governance feature model is used to sort the average effective governance index corresponding to the unit table from small to large and mark it on the vertical axis. With the unit table as the horizontal axis and the average effective governance index corresponding to the unit table as the vertical axis, a bar chart of the governance dimension of the unit table is generated as the data governance feature model.

8. The data intelligent monitoring and management platform based on big data according to claim 7, characterized in that: The identifiable feature analysis module includes a first index calculation unit, a target dimension determination unit, a feature differentiation unit to be analyzed and identified unit, and a verification feature screening unit. The first index calculation unit is used to sum all governance dimensions in the governance object database to obtain the first index; The target dimension determination unit is used to mark the highest priority governance dimension of each governance object database in the database analysis set as the target dimension. The feature differentiation unit is used to extract the governance project corresponding to the maximum data feature value as the feature to be analyzed and identified under the target dimension. The verification feature filtering unit is used to verify and filter the features to be analyzed and identified.

9. The data intelligent monitoring and management platform based on big data according to claim 8, characterized in that: The verification feature screening unit includes a probability analysis unit and a recognizable feature output unit; The probability analysis unit is used to calculate the probability that the governance dimension of the governance object database is the target dimension when a valid identification feature appears in the governance object database; The identifiable feature output unit is used to extract the effective identifiable feature corresponding to the maximum data result of the probability analysis unit as the identifiable feature.

Images