Comprehensive evaluation data analysis method and system based on big data

Abstract

The invention provides a comprehensive evaluation data analysis method and system based on big data, and relates to the technical field of data analysis, first category information, second category information and third category information are obtained, category information is updated and matched, preset analysis demand data is collected through a big data method, and a big data analysis result is obtained. Performing data division and classification on the analysis and acquisition data through a data evaluation dimension to obtain a data analysis library, and obtaining a comprehensive evaluation index; and calculating a third category index update value and a second category index update value, judging whether the second category index and the first category index are updated or not, and obtaining an updated comprehensive evaluation update index, so that the data processing efficiency and the evaluation accuracy can be greatly improved.

Description

Technical Field

[0001] The present invention proposes a comprehensive evaluation data analysis method and system based on big data, which relates to the technical field of data analysis, and specifically to the technical field of comprehensive evaluation data analysis. Background Art

[0002] As an important data analysis method, comprehensive evaluation data analysis aims to conduct a comprehensive and objective evaluation of complex systems by constructing a reasonable evaluation dimension and indicator system. However, traditional comprehensive evaluation data analysis methods often have problems such as low data processing efficiency, single evaluation dimension, and strong subjectivity in determining indicator weights, which makes it difficult to meet the requirements of accuracy and timeliness of data analysis and evaluation in the big data era. Summary of the invention

[0003] The present invention provides a comprehensive evaluation data analysis method and system based on big data to solve the above problems:

[0004] The present invention proposes a comprehensive evaluation data analysis method and system based on big data, the method comprising:

[0005] S1. Obtain preset category information and preset type information, and then obtain first category information, second category information, and third category information, construct a data evaluation dimension, update and match the category information, and obtain dimension update information;

[0006] S2. Collecting preset analysis demand data through a big data method to obtain analysis collection data, and classifying the analysis collection data through a data evaluation dimension to obtain a data analysis library;

[0007] S3. Determine the weight data of each indicator of the third category by the global entropy method, and use the comprehensive evaluation method to perform weighted summation on each indicator to obtain a comprehensive evaluation index;

[0008] S4. Obtain data change information of indicators of the third category and the second category, calculate update values ​​of the indicators of the third category and the second category, determine whether to update the indicators of the second category and the first category, and obtain an updated comprehensive evaluation update index.

[0009] Furthermore, the S1 includes:

[0010] Obtaining preset first category information;

[0011] Classify the first category information according to preset second category information to obtain second category information;

[0012] The second category information is classified according to the preset third category information to obtain the third category information

[0013] The first category of information includes the second category of information, and the second category of information includes the third category of information;

[0014] The first category information, the second category information and the third category information are combined to obtain a data evaluation dimension;

[0015] Acquire category update information, and match the category update information with the first category information, the second category information, and the third category information respectively to obtain a matching category level corresponding to the category update information;

[0016] Performing a preset category update on the matching category level through the category update information to obtain category update information of the corresponding matching category level;

[0017] Performing a preset category update on the lower category information of the matching category level to obtain category update information of the lower category information;

[0018] After the data is updated, the first category information, the second category information, and the third category information are obtained and combined to obtain dimension update information.

[0019] Further, the S2 includes:

[0020] Acquire preset analysis requirement information, collect the preset analysis requirement information through a big data method, and obtain analysis collection data;

[0021] Preprocessing the analysis and collection data to obtain preprocessed analysis and collection data;

[0022] Classify the analyzed collected data according to the first category information, the second category information and the third category information of the data evaluation dimension to obtain classified first category data, second category data and third category data;

[0023] The classified first category data, second category data and third category data are combined to obtain a data analysis library.

[0024] Further, the S3 includes:

[0025] Establish a three-dimensional time series data table based on the data analysis library and form an initial global evaluation matrix;

[0026] The extreme value method is used to standardize each indicator, and the standardized data is distributed between 0.1-0.9. The specific standardization formula includes positive indicators and negative indicators;

[0027] Normalize each index to obtain index normalization;

[0028] Calculate the entropy and redundancy of each indicator;

[0029] Determine the weight of each indicator;

[0030] Perform comprehensive weighting to obtain a comprehensive evaluation index.

[0031] Further, the S4 includes:

[0032] Acquire data change information of the third category indicator in real time, and calculate the third category indicator update value according to the data change information of the third category indicator;

[0033] Compare the third category indicator update value with the preset three categories of update thresholds to obtain a three-level comparison result;

[0034] updating the second category indicator according to the three-level comparison result, obtaining data change information of the second category indicator, and calculating the second category indicator update value according to the data change information of the second category indicator;

[0035] Compare the second category indicator update value with the preset second category update threshold to obtain a secondary comparison result;

[0036] updating the first category of indicators according to the secondary comparison result, thereby obtaining an indicator update database;

[0037] Obtain a comprehensive evaluation update index of the indicator update database.

[0038] Furthermore, the system comprises:

[0039] A category dimension construction module is used to obtain preset category information and preset type information, and then obtain first category information, second category information and third category information, construct a data evaluation dimension, update and match category information, and obtain dimension update information;

[0040] A data collection and classification module is used to collect preset analysis demand data through a big data method to obtain analysis and collection data, and to divide and classify the analysis and collection data through a data evaluation dimension to obtain a data analysis library;

[0041] The comprehensive evaluation analysis module is used to determine the weight data of each indicator of the third category by the global entropy method, and to perform weighted summation of each indicator by the comprehensive evaluation method to obtain a comprehensive evaluation index;

[0042] The indicator update evaluation module is used to obtain data change information of indicators of the third category and the second category, calculate the update value of the third category indicator and the update value of the second category indicator, determine whether to update the indicators of the second category and the first category, and obtain the updated comprehensive evaluation update index.

[0043] Furthermore, the category dimension building module includes:

[0044] A preset category classification module, used to obtain preset first category information;

[0045] Classify the first category information according to preset second category information to obtain second category information;

[0046] The second category information is classified according to the preset third category information to obtain the third category information

[0047] The first category of information includes the second category of information, and the second category of information includes the third category of information;

[0048] The first category information, the second category information and the third category information are combined to obtain a data evaluation dimension;

[0049] A preset category update module is used to obtain category update information, match the category update information with the first category information, the second category information and the third category information respectively, and obtain a matching category level corresponding to the category update information;

[0050] Performing a preset category update on the matching category level through the category update information to obtain category update information of the corresponding matching category level;

[0051] Performing a preset category update on the lower category information of the matching category level to obtain category update information of the lower category information;

[0052] After the data is updated, the first category information, the second category information, and the third category information are obtained and combined to obtain dimension update information.

[0053] Furthermore, the data collection and classification module includes:

[0054] A data collection module is used to obtain preset analysis requirement information, collect the preset analysis requirement information through a big data method, and obtain analysis collection data;

[0055] Preprocessing the analysis and collection data to obtain preprocessed analysis and collection data;

[0056] A collection data classification module is used to classify the analysis collection data according to the first category information, the second category information and the third category information of the data evaluation dimension to obtain the classified first category data, second category data and third category data;

[0057] The analysis library establishment module is used to combine the classified first category data, second category data and third category data to obtain a data analysis library.

[0058] Furthermore, the comprehensive evaluation and analysis module includes:

[0059] A standardization processing module is used to establish a three-dimensional time series data table based on the data analysis library and form an initial global evaluation matrix;

[0060] The extreme value method is used to standardize each indicator, and the standardized data is distributed between 0.1-0.9. The specific standardization formula includes positive indicators and negative indicators;

[0061] The indicator data processing module is used to normalize various indicators and obtain indicator normalization;

[0062] Calculate the entropy and redundancy of each indicator;

[0063] Determine the weight of each indicator;

[0064] The comprehensive evaluation calculation module is used to perform comprehensive weighting to obtain a comprehensive evaluation index.

[0065] Furthermore, the indicator update evaluation module includes:

[0066] A third category update analysis module, used to obtain data change information of the third category indicator in real time, and calculate the third category indicator update value according to the data change information of the third category indicator;

[0067] Compare the third category indicator update value with the preset three categories of update thresholds to obtain a three-level comparison result;

[0068] A second-category update analysis module, used to update the second-category indicator according to the third-level comparison result, obtain data change information of the second-category indicator, and calculate the second-category indicator update value according to the data change information of the second-category indicator;

[0069] Compare the second category indicator update value with the preset second category update threshold to obtain a secondary comparison result;

[0070] updating the first category of indicators according to the secondary comparison result, thereby obtaining an indicator update database;

[0071] The comprehensive update evaluation module is used to obtain the comprehensive evaluation update index of the indicator update database.

[0072] Beneficial effects of the present invention: By constructing data evaluation dimensions and using the global entropy method to determine weights, the system can more accurately reflect the actual status and value of the data and avoid interference from human factors. The system can continuously monitor data change information, and update indicators and comprehensive evaluations as needed to ensure the timeliness and accuracy of the evaluation. By constructing category information and data evaluation dimensions at different levels, the system can support multi-level and multi-angle data analysis to meet the needs and business scenarios of different users. The use of big data methods and automated processes for data collection, classification, and evaluation calculations greatly improves the efficiency and degree of automation of data processing and reduces labor costs. The comprehensive evaluation index and update index provided by the system can provide a scientific basis and data support for decision-making, which can improve the scientificity and accuracy of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of a comprehensive evaluation data analysis method based on big data. DETAILED DESCRIPTION

[0074] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0075] In one embodiment of the present invention, a comprehensive evaluation data analysis method and system based on big data is proposed by the present invention, and the method comprises:

[0076] S1. Obtain preset category information and preset type information, and then obtain first category information, second category information, and third category information, construct a data evaluation dimension, update and match the category information, and obtain dimension update information;

[0077] S2. Collecting preset analysis demand data through a big data method to obtain analysis collection data, and classifying the analysis collection data through a data evaluation dimension to obtain a data analysis library;

[0078] S3. Determine the weight data of each indicator of the third category by the global entropy method, and use the comprehensive evaluation method to perform weighted summation on each indicator to obtain a comprehensive evaluation index;

[0079] S4, obtain the data change information of the third category and the second category indicators, calculate the updated value of the third category indicators and the updated value of the second category indicators, determine whether to update the second category and the first category indicators, and obtain the updated comprehensive evaluation update index, such as Figure 1 shown.

[0080] The working principle of the above technical solution is as follows: In the initial stage, the system obtains preset category information and type information. Based on these preset information, the system further refines and constructs the first category information, the second category information and the third category information, which represent different data levels or analysis angles. The system constructs the data evaluation dimension and continues to update and match the preset category information to ensure the timeliness and accuracy of the preset categories of the data evaluation dimension. The preset analysis requirement data is collected through big data methods, and these data may come from a variety of data sources, such as databases, log files, sensors, etc. The collected data is divided and classified according to the previously constructed data evaluation dimension to form a structured data analysis library. This step ensures the orderliness and analyzability of the data. The system uses the global entropy method to determine the weight data of each indicator in the third category. The global entropy method is a method based on information entropy that can objectively reflect the amount of information and importance of indicator data. After determining the weight, the system uses the comprehensive evaluation method to perform weighted summation on each indicator and calculate the comprehensive evaluation index. This index reflects the overall performance or status of the third category data. The system continuously monitors the data change information of the third category and the second category indicators in order to promptly detect anomalies or trend changes in the data. Based on these change information, the system calculates the updated values ​​of the third category indicators and the second category indicators, and determines whether it is necessary to update the second category and the first category indicators. If an update is required, the system will recalculate the comprehensive evaluation update index to reflect the latest data status and evaluation results.

[0081] The technical effects of the above technical solution are as follows: by constructing data evaluation dimensions and using the global entropy method to determine weights, the system can more accurately reflect the actual status and value of the data and avoid interference from human factors. The system can continuously monitor data change information and update indicators and comprehensive evaluations as needed to ensure the timeliness and accuracy of the evaluation. By constructing category information and data evaluation dimensions at different levels, the system can support multi-level and multi-angle data analysis to meet the needs and business scenarios of different users. The use of big data methods and automated processes for data collection, classification, and evaluation calculations greatly improves the efficiency and automation of data processing and reduces labor costs. The comprehensive evaluation index and update index provided by the system can provide scientific basis and data support for decision-making, which can improve the scientificity and accuracy of decision-making.

[0082] In one embodiment of the present invention, the S1 includes:

[0083] Obtaining preset first category information;

[0084] Classify the first category information according to preset second category information to obtain second category information;

[0085] The second category information is classified according to the preset third category information to obtain the third category information

[0086] The first category of information includes the second category of information, and the second category of information includes the third category of information;

[0087] The first category information, the second category information and the third category information are combined to obtain a data evaluation dimension;

[0088] Obtain category update information, match the category update information with the first category information, the second category information and the third category information respectively, and obtain a matching category level corresponding to the category update information; the matching category level includes the first category information, the second category information and the third category information; when the category update information matches the first category information, the matching category level corresponding to the category update information is the category level of the first category information, the matching is performed by calculating information similarity, comparing the calculation result with a preset matching threshold, matching the category matching information according to the comparison result, and the similarity calculation and threshold settings are settings that can be performed by those skilled in the art based on the prior art.

[0089] Performing a preset category update on the matching category level through the category update information to obtain category update information of the corresponding matching category level;

[0090] Perform preset category update on the lower category information of the matching category level to obtain category update information of the lower category information; for example, after updating the first category information, the second and third category information need to be adaptively updated.

[0091] After the data is updated, the first category information, the second category information, and the third category information are obtained and combined to obtain dimension update information.

[0092] The working principle of the above technical solution is as follows: obtain the preset first category information, which constitutes the basic framework of data analysis. Classify the first category information according to the preset second category information to form the second category information. This step is to further refine and organize the first category information. Then classify the second category information according to the preset third category information to obtain the third category information. So far, the data has been divided into three levels, each of which represents a different data perspective or analysis dimension. Combine the first category information, the second category information and the third category information to form a data evaluation dimension. Obtain category update information, which comes from an external data source or internal data update. Match these category update information with the first category information, the second category information and the third category information to determine the category level to which the update information belongs. The matching process involves the calculation of information similarity. By comparing the calculation result with the preset matching threshold, the system can determine which category level the update information best matches. Once the match is successful, the system will perform a preset category update on the matching category level. At the same time, in order to ensure the consistency and integrity of the data, the system will also perform adaptive updates on the lower category information of the matching category level. For example, if the first category information is updated, the system will adjust the second and third category information accordingly. After the category update is completed, the system re-acquires the first category information, the second category information, and the third category information, and combines them to obtain the latest dimension update information. This information reflects the latest status of the data evaluation dimension.

[0093] The technical effect of the above technical solution is: by dividing the data into different levels of category information, the system can organize and manage the data more flexibly. At the same time, this hierarchical structure also makes the system easy to expand and adapt to new data needs. By matching the category update information with the preset category hierarchy, the update information can be accurately applied to the correct data level. At the same time, the adaptive update of the lower-level category information ensures the consistency and integrity of the data and improves the efficiency of data update. The constructed data evaluation dimension provides a basis for multi-level and multi-angle data analysis. The system can conduct in-depth analysis and evaluation of data from different dimensions and perspectives according to user needs or business scenarios. By introducing intelligent methods such as information similarity calculation and matching threshold setting, the system can automatically complete the matching and processing of category update information, reduce the degree of manual intervention, and improve the intelligent level of data processing.

[0094] In one embodiment of the present invention, S2 includes:

[0095] Acquire preset analysis requirement information, collect the preset analysis requirement information through a big data method, and obtain analysis collection data;

[0096] Preprocessing the analysis and collection data to obtain preprocessed analysis and collection data;

[0097] Classify the analyzed collected data according to the first category information, the second category information and the third category information of the data evaluation dimension to obtain classified first category data, second category data and third category data;

[0098] The classified first category data, second category data and third category data are combined to obtain a data analysis library.

[0099] The working principle of the above technical solution is as follows: the starting point of the whole process involves determining the specific content or target to be analyzed. The preset analysis requirement information comes from business needs, market analysis, data research and other aspects. According to the preset analysis requirements, big data methods (such as distributed storage, parallel computing, data mining, etc.) are used to collect relevant data from various data sources (such as databases, log files, social media, IoT devices, etc.). The purpose of this step is to collect information related to the analysis requirements as comprehensively as possible. The data collection method adopted is the data collection method that already exists in the prior art. The collected raw data often has problems such as noise, missing values, and outliers, so preprocessing is required. The preprocessing steps may include data cleaning (removing invalid or erroneous data), data transformation (such as normalization, standardization), data integration (merging data from multiple data sources), etc., to obtain clean, consistent, and usable analysis collection data. The preprocessed data is classified according to the first category information, second category information, and third category information of the data evaluation dimension. These category information may be defined based on the nature, source, importance or other relevant characteristics of the data. The purpose of classification is to organize the data into a format that is easier to analyze and understand. The classified data (first category data, second category data, and third category data) are combined to form a comprehensive data analysis library.

[0100] The technical effect of the above technical solution is: through big data collection and preprocessing methods, the quality of the data used for analysis can be ensured, and the analysis errors caused by data problems can be reduced. Data classification and combination make the data more orderly and accessible, thereby improving the efficiency of data analysis. Classification according to different data evaluation dimensions enables the data analysis library to support diverse analysis needs. Whether it is trend analysis, correlation analysis or predictive analysis, suitable data sets can be found from the classified data. A high-quality and structured data analysis library provides a solid foundation for decision-making. The analysis results based on these data can guide business decisions more accurately and reliably. By establishing a data analysis library, enterprises can better manage and utilize their data assets and improve the level of data governance. This can ensure the compliance, security and accessibility of data, laying the foundation for the long-term development of the enterprise.

[0101] In one embodiment of the present invention, S3 includes:

[0102] Establish a three-dimensional time series data table based on the data analysis library and form an initial global evaluation matrix;

[0103] The extreme value method is used to standardize each indicator, and the standardized data is distributed between 0.1-0.9. The specific standardization formula includes positive indicators and negative indicators;

[0104] Normalize each index to obtain index normalization;

[0105] Calculate the entropy and redundancy of each indicator;

[0106] Determine the weight of each indicator;

[0107] Perform comprehensive weighting to obtain a comprehensive evaluation index.

[0108] The working principle of the above technical solution is: the comprehensive evaluation module includes:

[0109] A matrix construction module is used to establish a three-dimensional time series data table and form an initial global evaluation matrix;

[0110] The initial global evaluation matrix is:

[0111]

[0112] in, is the jth indicator of the ith county in the tth year, m is the number of counties evaluated, n is the number of evaluation indicators, and T is the number of evaluation years;

[0113] The standardization module is used to standardize various indicators and distribute the standardized data between 0.1-0.9. The specific standardization formula includes positive indicators and negative indicators;

[0114] The calculation formula of the positive indicator is:

[0115]

[0116] The calculation formula of the negative indicator is:

[0117]

[0118] in, yes The standardized value of and is the maximum and minimum value of the jth indicator in all counties;

[0119] A normalization module is used to normalize various indicators and obtain indicator normalization;

[0120] The normalized calculation formula of the indicator is:

[0121]

[0122] in, for The normalized value of

[0123] The entropy value calculation module is used to calculate the entropy value of each indicator. The calculation formula of the entropy value is:

[0124]

[0125]

[0126] in, is the entropy value of the jth indicator;

[0127] The redundancy calculation module is used to calculate the redundancy of each indicator. The redundancy calculation formula is:

[0128]

[0129] in, is the redundancy of the j-th indicator;

[0130] The weight calculation module is used to calculate the weight of each indicator. The calculation formula for the weight of each indicator is:

[0131]

[0132] Among them, W j is the weight of each indicator;

[0133] The comprehensive evaluation module is used to perform comprehensive weighting on various indicators. The calculation formula of the comprehensive weighting is:

[0134]

[0135] in, It is a comprehensive evaluation index.

[0136] Check the completeness, accuracy and consistency of the data, and handle missing values, outliers, etc.

[0137] Use Python's pandas library or other data processing tools to build a three-dimensional data table containing time, county, and indicators based on the collected data.

[0138] Expand the three-dimensional data table into a two-dimensional matrix, where the rows represent the indicator values ​​of different counties in different years and the columns represent different indicators.

[0139] Use the formula to normalize the positive indicators.

[0140] Use the formula to normalize negative indicators.

[0141] Use numpy or pandas libraries for vectorized operations to improve computing efficiency.

[0142] The standardized data is normalized using the formula to obtain the normalized value.

[0143] Also use numpy or pandas library for vectorized operations.

[0144] The entropy value of each indicator is calculated using the formula.

[0145] The redundancy of each indicator is calculated.

[0146] Use the numpy library for mathematical operations, including logarithms and summations.

[0147] Use the formula to calculate the weight of each indicator.

[0148] The numpy library is used for vectorized operations to calculate the weights of various indicators.

[0149] Use the formula to calculate the comprehensive evaluation index of each county.

[0150] The comprehensive evaluation index is sorted or visualized by county and time for further analysis and comparison.

[0151] Use the pandas library for data processing and result output, and use matplotlib or seaborn library for visualization.

[0152] Through the implementation of the above technical means, each step in the above technical solution can be completed, thereby obtaining a comprehensive evaluation index for each county at a specific time.

[0153] The technical effects of the above technical scheme are as follows: through data preprocessing, standardization and normalization, the accuracy and consistency of the input data are ensured, and the distortion of the evaluation results caused by data errors or inconsistencies is avoided. By using efficient data processing libraries such as Python's pandas and numpy, the rapid cleaning, conversion and calculation of data are realized, which greatly improves the efficiency of data processing. Through the calculation of entropy value and redundancy, and the weight allocation based on redundancy, it is ensured that the objectivity and importance of each indicator in the evaluation process are fully considered, and the influence of subjective factors on the evaluation results is avoided. The comprehensive weighted evaluation method is adopted to combine the normalized value of each indicator with the weight to obtain the comprehensive evaluation index of each county in different years. By constructing a three-dimensional time series data table and an initial global evaluation matrix, the changes in indicators of each county in different years can be clearly seen, and the advantages and shortcomings of county development can be accurately located. Based on the comprehensive evaluation index, targeted development suggestions and policy directions can be provided for each county. Using visualization tools such as matplotlib or seaborn, the comprehensive evaluation index can be visualized. By building a complete county evaluation index system and data analysis model, we can promote the continuous improvement and optimization of the county governance system. With the help of data analysis results, we can improve the accuracy and scientificity of county governance and improve the efficiency and level of county governance.

[0154] In one embodiment of the present invention, the S4 includes:

[0155] Acquire data change information of the third category indicator in real time, and calculate the third category indicator update value according to the data change information of the third category indicator;

[0156] The calculation formula for the updated value of the third category indicator is:

[0157]

[0158] in, Update the value of the third category indicator for each second category, P is the number of third category indicators for the second category, is the data change of the ath third category indicator of the second category, is the initial data volume of the ath third category indicator of the second category.

[0159] Compare the third category indicator update value with the preset three categories of update thresholds to obtain a three-level comparison result;

[0160] updating the second category indicator according to the three-level comparison result, obtaining data change information of the second category indicator, and calculating the second category indicator update value according to the data change information of the second category indicator;

[0161] The calculation formula for the updated value of the second category indicator is:

[0162]

[0163] in, Update the value for each second category indicator of the first category, r is the number of second category indicators of the first category, is the data change of the e-th second category indicator of the first category, is the initial data volume of the e-th second category indicator of the first category.

[0164] Compare the second category indicator update value with the preset second category update threshold to obtain a secondary comparison result;

[0165] updating the first category of indicators according to the secondary comparison result, thereby obtaining an indicator update database;

[0166] Obtain a comprehensive evaluation update index of the indicator update database.

[0167] The working principle of the above technical solution is: collect data change information of the third category indicator in real time through specific data sources (such as sensors, databases, APIs, etc.). These indicators may involve multiple aspects such as business operations, production processes, and environmental monitoring. According to the acquired data change information, calculate the update value of the third category indicator. Compare the calculated update value of the third category indicator with the preset three-category update threshold. These thresholds are usually set by ordinary technicians according to business logic and actual needs to determine the magnitude and importance of indicator changes. According to the three-level comparison results, update the second category indicator. If the change of the third category indicator exceeds a certain threshold, it may trigger the adjustment or recalculation of the second category indicator. Similarly, calculate its update value according to the updated second category indicator data change information. Compare the second category indicator update value with the preset second category update threshold to further determine the magnitude and importance of its change. According to the second-level comparison results, update the first category indicator. Store the updated first, second, and third category indicators in the indicator update database. Use the data in the indicator update database to calculate the comprehensive evaluation update index. This index reflects the health, efficiency, or performance of the overall business or system.

[0168] The technical effect of the above technical solution is: by acquiring and updating the indicator data in real time, the system can quickly reflect the latest status of the business or system and improve the accuracy and timeliness of decision-making. By dividing the indicators into different categories and updating and comparing them in turn, the system can provide strong support for decision-making at different levels. The system is flexible in design and can adjust the category, threshold and update algorithm of the indicators according to actual needs. At the same time, the system is easy to expand and can accommodate more indicators and data sources. By calculating the comprehensive evaluation update index, the system can provide data support for decision-making and reduce the risk of subjective assumptions and blind decisions.

[0169] The system can automatically complete the process of acquiring, updating and comparing indicators, reducing manual intervention and improving work efficiency.

[0170] In one embodiment of the present invention, the system comprises:

[0171] A category dimension construction module is used to obtain preset category information and preset type information, and then obtain first category information, second category information and third category information, construct a data evaluation dimension, update and match category information, and obtain dimension update information;

[0172] A data collection and classification module is used to collect preset analysis demand data through a big data method to obtain analysis and collection data, and to divide and classify the analysis and collection data through a data evaluation dimension to obtain a data analysis library;

[0173] The comprehensive evaluation analysis module is used to determine the weight data of each indicator of the third category by the global entropy method, and to perform weighted summation of each indicator by the comprehensive evaluation method to obtain a comprehensive evaluation index;

[0174] The indicator update evaluation module is used to obtain data change information of indicators of the third category and the second category, calculate the update value of the third category indicator and the update value of the second category indicator, determine whether to update the indicators of the second category and the first category, and obtain the updated comprehensive evaluation update index.

[0175] The working principle of the above technical solution is as follows: In the initial stage, the system obtains preset category information and type information. Based on these preset information, the system further refines and constructs the first category information, the second category information and the third category information, which represent different data levels or analysis angles. The system constructs the data evaluation dimension and continues to update and match the preset category information to ensure the timeliness and accuracy of the preset categories of the data evaluation dimension. The preset analysis requirement data is collected through big data methods, and these data may come from a variety of data sources, such as databases, log files, sensors, etc. The collected data is divided and classified according to the previously constructed data evaluation dimension to form a structured data analysis library. This step ensures the orderliness and analyzability of the data. The system uses the global entropy method to determine the weight data of each indicator in the third category. The global entropy method is a method based on information entropy that can objectively reflect the amount of information and importance of indicator data. After determining the weight, the system uses the comprehensive evaluation method to perform weighted summation on each indicator and calculate the comprehensive evaluation index. This index reflects the overall performance or status of the third category data. The system continuously monitors the data change information of the third category and the second category indicators in order to promptly detect anomalies or trend changes in the data. Based on these change information, the system calculates the updated values ​​of the third category indicators and the second category indicators, and determines whether it is necessary to update the second category and the first category indicators. If an update is required, the system will recalculate the comprehensive evaluation update index to reflect the latest data status and evaluation results.

[0176] The technical effects of the above technical solution are as follows: by constructing data evaluation dimensions and using the global entropy method to determine weights, the system can more accurately reflect the actual status and value of the data and avoid interference from human factors. The system can continuously monitor data change information and update indicators and comprehensive evaluations as needed to ensure the timeliness and accuracy of the evaluation. By constructing category information and data evaluation dimensions at different levels, the system can support multi-level and multi-angle data analysis to meet the needs and business scenarios of different users. The use of big data methods and automated processes for data collection, classification, and evaluation calculations greatly improves the efficiency and automation of data processing and reduces labor costs. The comprehensive evaluation index and update index provided by the system can provide scientific basis and data support for decision-making, which can improve the scientificity and accuracy of decision-making.

[0177] In one embodiment of the present invention, the category dimension building module includes:

[0178] A preset category classification module, used to obtain preset first category information;

[0179] Classify the first category information according to preset second category information to obtain second category information;

[0180] The second category information is classified according to the preset third category information to obtain the third category information

[0181] The first category of information includes the second category of information, and the second category of information includes the third category of information;

[0182] The first category information, the second category information and the third category information are combined to obtain a data evaluation dimension;

[0183] A preset category update module is used to obtain category update information, match the category update information with the first category information, the second category information and the third category information respectively, and obtain the matching category level corresponding to the category update information; the matching category level includes the first category information, the second category information and the third category information; when the category update information matches the first category information, the matching category level corresponding to the category update information is the category level of the first category information, the matching is performed by calculating information similarity, comparing the calculation result with a preset matching threshold, and matching the category matching information according to the comparison result, and the similarity calculation and threshold setting are settings that can be performed by those skilled in the art based on the prior art.

[0184] Performing a preset category update on the matching category level through the category update information to obtain category update information of the corresponding matching category level;

[0185] Perform preset category update on the lower category information of the matching category level to obtain category update information of the lower category information; for example, after updating the first category information, the second and third category information need to be adaptively updated.

[0186] After the data is updated, the first category information, the second category information, and the third category information are obtained and combined to obtain dimension update information.

[0187] The working principle of the above technical solution is as follows: obtain the preset first category information, which constitutes the basic framework of data analysis. Classify the first category information according to the preset second category information to form the second category information. This step is to further refine and organize the first category information. Then classify the second category information according to the preset third category information to obtain the third category information. So far, the data has been divided into three levels, each of which represents a different data perspective or analysis dimension. Combine the first category information, the second category information and the third category information to form a data evaluation dimension. Obtain category update information, which comes from an external data source or internal data update. Match these category update information with the first category information, the second category information and the third category information to determine the category level to which the update information belongs. The matching process involves the calculation of information similarity. By comparing the calculation result with the preset matching threshold, the system can determine which category level the update information best matches. Once the match is successful, the system will perform a preset category update on the matching category level. At the same time, in order to ensure the consistency and integrity of the data, the system will also perform adaptive updates on the lower category information of the matching category level. For example, if the first category information is updated, the system will adjust the second and third category information accordingly. After the category update is completed, the system re-acquires the first category information, the second category information, and the third category information, and combines them to obtain the latest dimension update information. This information reflects the latest status of the data evaluation dimension.

[0188] The technical effect of the above technical solution is: by dividing the data into different levels of category information, the system can organize and manage the data more flexibly. At the same time, this hierarchical structure also makes the system easy to expand and adapt to new data needs. By matching the category update information with the preset category hierarchy, the update information can be accurately applied to the correct data level. At the same time, the adaptive update of the lower-level category information ensures the consistency and integrity of the data and improves the efficiency of data update. The constructed data evaluation dimension provides a basis for multi-level and multi-angle data analysis. The system can conduct in-depth analysis and evaluation of data from different dimensions and perspectives according to user needs or business scenarios. By introducing intelligent methods such as information similarity calculation and matching threshold setting, the system can automatically complete the matching and processing of category update information, reduce the degree of manual intervention, and improve the intelligent level of data processing.

[0189] In one embodiment of the present invention, the data collection and classification module includes:

[0190] A data collection module is used to obtain preset analysis requirement information, collect the preset analysis requirement information through a big data method, and obtain analysis collection data;

[0191] Preprocessing the analysis and collection data to obtain preprocessed analysis and collection data;

[0192] A collection data classification module is used to classify the analysis collection data according to the first category information, the second category information and the third category information of the data evaluation dimension to obtain the classified first category data, second category data and third category data;

[0193] The analysis library establishment module is used to combine the classified first category data, second category data and third category data to obtain a data analysis library.

[0194] The working principle of the above technical solution is as follows: the starting point of the whole process involves determining the specific content or target to be analyzed. The preset analysis requirement information comes from business requirements, market analysis, data research and other aspects. According to the preset analysis requirements, big data methods (such as distributed storage, parallel computing, data mining, etc.) are used to collect relevant data from various data sources (such as databases, log files, social media, IoT devices, etc.). The purpose of this step is to collect information related to the analysis requirements as comprehensively as possible. The data collection method adopted is the data collection method that already exists in the prior art. The collected raw data often has problems such as noise, missing values, and outliers, so preprocessing is required. The preprocessing steps may include data cleaning (removing invalid or erroneous data), data transformation (such as normalization, standardization), data integration (merging data from multiple data sources), etc., to obtain clean, consistent, and usable analysis collection data. The preprocessed data is classified according to the first category information, second category information, and third category information of the data evaluation dimension. These category information may be defined based on the nature, source, importance or other relevant characteristics of the data. The purpose of classification is to organize the data into a format that is easier to analyze and understand. The classified data (first category data, second category data, and third category data) are combined to form a comprehensive data analysis library.

[0195] The technical effect of the above technical solution is: through big data collection and preprocessing methods, the quality of the data used for analysis can be ensured, and the analysis errors caused by data problems can be reduced. Data classification and combination make the data more orderly and accessible, thereby improving the efficiency of data analysis. Classification according to different data evaluation dimensions enables the data analysis library to support diverse analysis needs. Whether it is trend analysis, correlation analysis or predictive analysis, suitable data sets can be found from the classified data. A high-quality and structured data analysis library provides a solid foundation for decision-making. The analysis results based on these data can guide business decisions more accurately and reliably. By establishing a data analysis library, enterprises can better manage and utilize their data assets and improve the level of data governance. This can ensure the compliance, security and accessibility of data, laying the foundation for the long-term development of the enterprise.

[0196] In one embodiment of the present invention, the comprehensive evaluation and analysis module includes:

[0197] A standardization processing module is used to establish a three-dimensional time series data table based on the data analysis library and form an initial global evaluation matrix;

[0198] The extreme value method is used to standardize each indicator, and the standardized data is distributed between 0.1-0.9. The specific standardization formula includes positive indicators and negative indicators;

[0199] The indicator data processing module is used to normalize various indicators and obtain indicator normalization;

[0200] Calculate the entropy and redundancy of each indicator;

[0201] Determine the weight of each indicator;

[0202] The comprehensive evaluation calculation module is used to perform comprehensive weighting to obtain a comprehensive evaluation index.

[0203] The working principle of the above technical solution is:

[0204] Check the completeness, accuracy and consistency of the data, and handle missing values, outliers, etc.

[0205] Use Python's pandas library or other data processing tools to build a three-dimensional data table containing time, county, and indicators based on the collected data.

[0206] Expand the three-dimensional data table into a two-dimensional matrix, where the rows represent the indicator values ​​of different counties in different years and the columns represent different indicators.

[0207] Use the formula to normalize the positive indicators.

[0208] Use the formula to normalize negative indicators.

[0209] Use numpy or pandas libraries for vectorized operations to improve computing efficiency.

[0210] The standardized data is normalized using the formula to obtain the normalized value.

[0211] Also use numpy or pandas library for vectorized operations.

[0212] The entropy value of each indicator is calculated using the formula.

[0213] The redundancy of each indicator is calculated.

[0214] Use the numpy library for mathematical operations, including logarithms and summations.

[0215] Use the formula to calculate the weight of each indicator.

[0216] The numpy library is used for vectorized operations to calculate the weights of various indicators.

[0217] Use the formula to calculate the comprehensive evaluation index of each county.

[0218] The comprehensive evaluation index is sorted or visualized by county and time for further analysis and comparison.

[0219] Use the pandas library for data processing and result output, and use matplotlib or seaborn library for visualization.

[0220] Through the implementation of the above technical means, each step in the above technical solution can be completed, thereby obtaining a comprehensive evaluation index for each county at a specific time.

[0221] The technical effects of the above technical scheme are as follows: through data preprocessing, standardization and normalization, the accuracy and consistency of the input data are ensured, and the distortion of the evaluation results caused by data errors or inconsistencies is avoided. By using efficient data processing libraries such as Python's pandas and numpy, the rapid cleaning, conversion and calculation of data are realized, which greatly improves the efficiency of data processing. Through the calculation of entropy value and redundancy, and the weight allocation based on redundancy, it is ensured that the objectivity and importance of each indicator in the evaluation process are fully considered, and the influence of subjective factors on the evaluation results is avoided. The comprehensive weighted evaluation method is adopted to combine the normalized value of each indicator with the weight to obtain the comprehensive evaluation index of each county in different years. By constructing a three-dimensional time series data table and an initial global evaluation matrix, the changes in indicators of each county in different years can be clearly seen, and the advantages and shortcomings of county development can be accurately located. Based on the comprehensive evaluation index, targeted development suggestions and policy directions can be provided for each county. Using visualization tools such as matplotlib or seaborn, the comprehensive evaluation index can be visualized. By building a complete county evaluation index system and data analysis model, we can promote the continuous improvement and optimization of the county governance system. With the help of data analysis results, we can improve the accuracy and scientificity of county governance and improve the efficiency and level of county governance.

[0222] In one embodiment of the present invention, the indicator update evaluation module includes:

[0223] A third category update analysis module, used to obtain data change information of the third category indicator in real time, and calculate the third category indicator update value according to the data change information of the third category indicator;

[0224] The calculation formula for the updated value of the third category indicator is:

[0225]

[0226] in, Update the value of the third category indicator for each second category, P is the number of third category indicators for the second category, is the data change of the ath third category indicator of the second category, is the initial data volume of the ath third category indicator of the second category.

[0227] Compare the third category indicator update value with the preset three categories of update thresholds to obtain a three-level comparison result;

[0228] A second-category update analysis module, used to update the second-category indicator according to the third-level comparison result, obtain data change information of the second-category indicator, and calculate the second-category indicator update value according to the data change information of the second-category indicator;

[0229] The calculation formula for the updated value of the second category indicator is:

[0230]

[0231] in, Update the value for each second category indicator of the first category, r is the number of second category indicators of the first category, is the data change of the e-th second category indicator of the first category, is the initial data volume of the e-th second category indicator of the first category.

[0232] Compare the second category indicator update value with the preset second category update threshold to obtain a secondary comparison result;

[0233] updating the first category of indicators according to the secondary comparison result, thereby obtaining an indicator update database;

[0234] The comprehensive update evaluation module is used to obtain the comprehensive evaluation update index of the indicator update database.

[0235] All thresholds in this application are set by ordinary technicians in this field based on historical knowledge and actual conditions.

[0236] The working principle of the above technical solution is: collect data change information of the third category indicator in real time through specific data sources (such as sensors, databases, APIs, etc.). These indicators may involve multiple aspects such as business operations, production processes, and environmental monitoring. According to the acquired data change information, calculate the update value of the third category indicator. Compare the calculated update value of the third category indicator with the preset three-category update threshold. These thresholds are usually set by ordinary technicians according to business logic and actual needs to determine the magnitude and importance of indicator changes. According to the three-level comparison results, update the second category indicator. If the change of the third category indicator exceeds a certain threshold, it may trigger the adjustment or recalculation of the second category indicator. Similarly, calculate its update value according to the updated second category indicator data change information. Compare the second category indicator update value with the preset second category update threshold to further determine the magnitude and importance of its change. According to the second-level comparison results, update the first category indicator. Store the updated first, second, and third category indicators in the indicator update database. Use the data in the indicator update database to calculate the comprehensive evaluation update index. This index reflects the health, efficiency, or performance of the overall business or system.

[0237] The technical effect of the above technical solution is: by acquiring and updating the indicator data in real time, the system can quickly reflect the latest status of the business or system and improve the accuracy and timeliness of decision-making. By dividing the indicators into different categories and updating and comparing them in turn, the system can provide strong support for decision-making at different levels. The system is flexible in design and can adjust the category, threshold and update algorithm of the indicators according to actual needs. At the same time, the system is easy to expand and can accommodate more indicators and data sources. By calculating the comprehensive evaluation update index, the system can provide data support for decision-making and reduce the risk of subjective assumptions and blind decisions.

[0238] The system can automatically complete the process of acquiring, updating and comparing indicators, reducing manual intervention and improving work efficiency.

[0239] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A comprehensive evaluation data analysis method based on big data, characterized in that: The method comprises: S1. Obtain preset category information and preset type information, and then obtain first category information, second category information, and third category information, construct a data evaluation dimension, update and match the category information, and obtain dimension update information; S2. Collecting preset analysis demand data through a big data method to obtain analysis collection data, and classifying the analysis collection data through a data evaluation dimension to obtain a data analysis library; S3. Determine the weight data of each indicator of the third category by the global entropy method, and use the comprehensive evaluation method to perform weighted summation on each indicator to obtain a comprehensive evaluation index; S4. Obtain data change information of indicators of the third category and the second category, calculate update values ​​of the indicators of the third category and the second category, determine whether to update the indicators of the second category and the first category, and obtain an updated comprehensive evaluation update index.

2. According to the comprehensive evaluation data analysis method based on big data as described in claim 1, it is characterized in that: The S1 includes: Obtaining preset first category information; Classify the first category information according to preset second category information to obtain second category information; The second category information is classified according to the preset third category information to obtain the third category information The first category information includes the second category information, and the second category information includes the third category information; The first category information, the second category information and the third category information are combined to obtain a data evaluation dimension; Acquire category update information, and match the category update information with the first category information, the second category information, and the third category information respectively to obtain a matching category level corresponding to the category update information; Performing a preset category update on the matching category level through the category update information to obtain category update information of the corresponding matching category level; Performing a preset category update on the lower category information of the matching category level to obtain category update information of the lower category information; After the data is updated, the first category information, the second category information and the third category information are obtained and combined to obtain dimension update information.

3. According to the comprehensive evaluation data analysis method based on big data as claimed in claim 1, it is characterized in that: The S2 includes: Acquire preset analysis requirement information, collect the preset analysis requirement information through a big data method, and obtain analysis collection data; Preprocessing the analysis and collection data to obtain preprocessed analysis and collection data; Classify the analyzed collected data according to the first category information, the second category information and the third category information of the data evaluation dimension to obtain classified first category data, second category data and third category data; The classified first category data, second category data and third category data are combined to obtain a data analysis library.

4. According to the comprehensive evaluation data analysis method based on big data as claimed in claim 1, it is characterized in that: The S3 includes: Establish a three-dimensional time series data table based on the data analysis library and form an initial global evaluation matrix; The extreme value method is used to standardize each indicator, and the standardized data is distributed between 0.1-0.

9. The specific standardization formula includes positive indicators and negative indicators; Normalize each index to obtain index normalization; Calculate the entropy and redundancy of each indicator; Determine the weight of each indicator; Perform comprehensive weighting to obtain a comprehensive evaluation index.

5. According to the comprehensive evaluation data analysis method based on big data as claimed in claim 1, it is characterized in that: The S4 includes: Acquire data change information of the third category indicator in real time, and calculate the third category indicator update value according to the data change information of the third category indicator; Compare the third category indicator update value with the preset three categories of update thresholds to obtain a three-level comparison result; updating the second category indicator according to the three-level comparison result, obtaining data change information of the second category indicator, and calculating the second category indicator update value according to the data change information of the second category indicator; Compare the second category indicator update value with the preset second category update threshold to obtain a secondary comparison result; updating the first category of indicators according to the secondary comparison result, thereby obtaining an indicator update database; Obtain a comprehensive evaluation update index of the indicator update database.

6. A comprehensive evaluation data analysis system based on big data, characterized in that: The system comprises: A category dimension construction module is used to obtain preset category information and preset type information, and then obtain first category information, second category information and third category information, construct a data evaluation dimension, update and match category information, and obtain dimension update information; A data collection and classification module is used to collect preset analysis demand data through a big data method to obtain analysis and collection data, and to divide and classify the analysis and collection data through a data evaluation dimension to obtain a data analysis library; The comprehensive evaluation analysis module is used to determine the weight data of each indicator of the third category by the global entropy method, and to perform weighted summation of each indicator by the comprehensive evaluation method to obtain a comprehensive evaluation index; The indicator update evaluation module is used to obtain data change information of indicators of the third category and the second category, calculate the update value of the third category indicators and the update value of the second category indicators, determine whether to update the indicators of the second category and the first category, and obtain the updated comprehensive evaluation update index.

7. A comprehensive evaluation data analysis system based on big data according to claim 6, characterized in that: The category dimension building module includes: A preset category classification module, used to obtain preset first category information; Classify the first category information according to preset second category information to obtain second category information; The second category information is classified according to the preset third category information to obtain the third category information The first category information includes the second category information, and the second category information includes the third category information; The first category information, the second category information and the third category information are combined to obtain a data evaluation dimension; A preset category update module is used to obtain category update information, match the category update information with the first category information, the second category information and the third category information respectively, and obtain a matching category level corresponding to the category update information; Performing a preset category update on the matching category level through the category update information to obtain category update information of the corresponding matching category level; Performing a preset category update on the lower category information of the matching category level to obtain category update information of the lower category information; After the data is updated, the first category information, the second category information and the third category information are obtained and combined to obtain dimension update information.

8. The comprehensive evaluation data analysis system based on big data according to claim 6 is characterized in that: The data collection and classification module comprises: A data collection module is used to obtain preset analysis requirement information, collect the preset analysis requirement information through a big data method, and obtain analysis collection data; Preprocessing the analysis and collection data to obtain preprocessed analysis and collection data; A collection data classification module is used to classify the analysis collection data according to the first category information, the second category information and the third category information of the data evaluation dimension to obtain the classified first category data, second category data and third category data; The analysis library establishment module is used to combine the classified first category data, second category data and third category data to obtain a data analysis library.

9. The comprehensive evaluation data analysis system based on big data according to claim 6 is characterized in that: The comprehensive evaluation and analysis module includes: A standardization processing module is used to establish a three-dimensional time series data table based on the data analysis library and form an initial global evaluation matrix; The extreme value method is used to standardize each indicator, and the standardized data is distributed between 0.1-0.

9. The specific standardization formula includes positive indicators and negative indicators; The indicator data processing module is used to normalize various indicators and obtain indicator normalization; Calculate the entropy and redundancy of each indicator; Determine the weight of each indicator; The comprehensive evaluation calculation module is used to perform comprehensive weighting to obtain a comprehensive evaluation index.

10. The comprehensive evaluation data analysis system based on big data according to claim 6, characterized in that: The indicator update evaluation module includes: A third category update analysis module, used for obtaining data change information of the third category indicator in real time, and calculating the third category indicator update value according to the data change information of the third category indicator; Compare the third category indicator update value with the preset three categories of update thresholds to obtain a three-level comparison result; A second-category update analysis module, used to update the second-category indicator according to the third-level comparison result, obtain data change information of the second-category indicator, and calculate the second-category indicator update value according to the data change information of the second-category indicator; Compare the second category indicator update value with the preset second category update threshold to obtain a secondary comparison result; updating the first category of indicators according to the secondary comparison result, thereby obtaining an indicator update database; The comprehensive update evaluation module is used to obtain the comprehensive evaluation update index of the indicator update database.

Images